LinkedIn course Download Fail

Checklist

[X ] I'm reporting a broken site support
[ X] I've verified that I'm running youtube-dl version 2019.11.28
[ X] I've checked that all provided URLs are alive and playable in a browser
[X ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X ] I've searched the bugtracker for similar issues including closed ones

Verbose log

PASTE VERBOSE LOG HERE

youtube-dl -u *** -p *** https://www.linkedin.com/learning/excel-2016-essential-training/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'https://www.linkedin.com/learning/excel-2016-essential-training/', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.11.28
[debug] Python version 2.7.16 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1, rtmpdump 2.4
[debug] Proxy map: {}
[linkedin:learning:course] Downloading login page
[linkedin:learning:course] Logging in
[linkedin:learning:course] Downloading JSON metadata
[download] Downloading playlist: Excel 2016 Essential Training
[linkedin:learning:course] playlist Excel 2016 Essential Training: Collected 85 video ids (downloading 85 of them)
[download] Downloading video 1 of 85
[linkedin:learning] Downloading login page
[linkedin:learning] Logging in
[linkedin:learning] welcome: Downloading 360p JSON metadata
[linkedin:learning] welcome: Downloading 540p JSON metadata
[linkedin:learning] welcome: Downloading 720p JSON metadata
[linkedin:learning] welcome: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://files3.lynda.com/secure/courses/376985/VBR_MP4h264_main_HD720/376985_00_01_WX30_intro.mp4?yo44ep24HaZRQXRYq55s8a3D9P4hFS-zM_ysj4kyUFehiP3PJ7arq5AfAqeWuUDACuv3W860zaCDDEgytZ5vKFXjwvn1q27cWZx6GV7bKfyEhglrGDSIjG6CtHMorOjpCYWVfbgY96XaM07KJh0pBoqFma87md0GGssnbpD7TgcZg3qylC0f'
[download] Destination: Welcome-431592.mp4
[download] 100% of 4.61MiB in 00:01
[download] Downloading video 2 of 85
[linkedin:learning] using-the-exercise-files: Downloading 360p JSON metadata
[linkedin:learning] using-the-exercise-files: Downloading 540p JSON metadata
[linkedin:learning] using-the-exercise-files: Downloading 720p JSON metadata
[linkedin:learning] using-the-exercise-files: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://files3.lynda.com/secure/courses/376985/VBR_MP4h264_main_HD720/376985_00_02_XR15_exfiles.mp4?Qu401sLx3isYUe1WnKJFl2CD-DhPbEStE-vvrESaxicYbuHEX4OW4c3ITk_GHZ-0WlzZaDBZ_mSBVmJ9GX5U0v9gbI7rd96xJD7oOK_hNuA4Opv_FLTy4S5yrlSE63ihg5w8yDrzJeN5sVciAUj1px5IQmyWhFgp53OUGEyRPLGFSWuvYkExapY'
[download] Destination: Using the exercise files-431593.mp4
[download] 100% of 1.50MiB in 00:00
[download] Downloading video 3 of 85
[linkedin:learning] what-is-excel-used-for: Downloading 360p JSON metadata
[linkedin:learning] what-is-excel-used-for: Downloading 540p JSON metadata
[linkedin:learning] what-is-excel-used-for: Downloading 720p JSON metadata
[linkedin:learning] what-is-excel-used-for: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://files3.lynda.com/secure/courses/376985/VBR_MP4h264_main_HD720/376985_01_01_XR15_exceluses.mp4?bEL4djkeQyrbVkarC_UNdW8UgOxMVUoXaxKhGqHK6wayoVyE0RytBA2VUokRMnZJs5LTRGCXaJD4NwLXDlMXJYTtkZjz7D7skm6DCktf8FE7-qdqwZg5x2mC59GOpBtaUO1rDNdxh0Brqkv79g3wT5rJZm6-cjPg02TGLLNoBQ9iPC9kSmydqMNxyg'
[download] Destination: What is Excel used for-431595.mp4
[download] 100% of 8.51MiB in 00:02
[download] Downloading video 4 of 85
[linkedin:learning] using-the-menu-system: Downloading 360p JSON metadata
[linkedin:learning] using-the-menu-system: Downloading 540p JSON metadata
[linkedin:learning] using-the-menu-system: Downloading 720p JSON metadata
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
ie_result = self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/linkedin.py", line 127, in _real_extract
self._sort_formats(formats, ('width', 'height', 'source_preference', 'tbr', 'abr'))
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1327, in _sort_formats
raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Description
Hi,
i try to download Linkedin Learning.
When i just use „youtube-dl https://www.linkedin.com/learning/excel-2016-essential-training/“ then its comes up with an Error.
When I use „youtube-dl -u xyz -p yxz https://www.linkedin.com/learning/excel-2016-essential-training/“ youtube-dl is downloading 3 Videos.
When i use my linked-in Account its still just downloads 3 Videos.
I only use Copy and Past for my Password and i can log in with Safari, Chrom and the iOS app.
My Account is a old Lynda Account with i got as a Student but my login process is normale. Its my normal University email address withe i got from my University.
I hope someone can help! :):)

try to use cookies.

Hi,
thanks for your tip but Cookies also do not work withe linked in Learning.
i tried my method on YouTube and it works but not on linked in Learning.
I Saved the Cookies with the Google Chrom Plugin „cookies.txt“
:(
youtube-dl --cookies /Users/ln/cookies.txt  https://www.linkedin.com/learning/excel-2016-essential-training/ -v [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'--cookies', u'/Users/ln/cookies.txt', u'https://www.linkedin.com/learning/excel-2016-essential-training/', u'-v'] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2019.11.28 [debug] Python version 2.7.16 (CPython) - Darwin-18.7.0-x86_64-i386-64bit [debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1, rtmpdump 2.4 [debug] Proxy map: {} Traceback (most recent call last):   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main     "__main__", fname, loader, pkg_name)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code     exec code in run_globals   File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>   File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 474, in main   File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main   File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2018, in download   File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info   File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract   File "/usr/local/bin/youtube-dl/youtube_dl/extractor/linkedin.py", line 158, in _real_extract   File "/usr/local/bin/youtube-dl/youtube_dl/extractor/linkedin.py", line 36, in _call_api   File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 2831, in _get_cookies   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 583, in __init__     if input: self.load(input)   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 643, in load     for k, v in rawdata.items(): AttributeError: 'unicode' object has no attribute 'items'

some of the code interacting with cookies is currently broken in python 2.x, you might want to try python 3 untill this is fixed.

Thank you but also not working.
I used "pip3 install youtube-dl" to run/get it in a different Python Version. That was new for me :)

youtube-dl --cookies /Users/ln/cookies.txt  https://www.linkedin.com/learning/excel-2016-essential-training/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', '/Users/ln/cookies.txt', 'https://www.linkedin.com/learning/excel-2016-essential-training/', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.11.28
[debug] Python version 3.7.4 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: An extractor error has occurred. (caused by KeyError('JSESSIONID')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 530, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/linkedin.py", line 158, in _real_extract
course_data = self._call_api(course_slug, 'chapters,description,title')
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/linkedin.py", line 36, in _call_api
'Csrf-Token': self._get_cookies(api_url)['JSESSIONID'].value,
KeyError: 'JSESSIONID'
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 530, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/linkedin.py", line 158, in _real_extract
course_data = self._call_api(course_slug, 'chapters,description,title')
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/linkedin.py", line 36, in _call_api
'Csrf-Token': self._get_cookies(api_url)['JSESSIONID'].value,
KeyError: 'JSESSIONID'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 543, in extract
raise ExtractorError('An extractor error has occurred.', cause=e)
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('JSESSIONID')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

the cookies you have exported are not complete.

WOW. THANK YOU! Now I used the FireFox Add on "Export Cookies" to Export the Cookies and it works! Thank you!
https://addons.mozilla.org/de/firefox/addon/export-cookies-txt/?src=search

For me also, not working with username and password for some videos. Then I tried cookies. The result is as follows.
> youtube-dl --cookies ./cookies.txt -f progressive-360p --playlist-start 1 --playlist-end 10 -o '%(playlist)s - %(playlist_index)s - %(title)s.%(ext)s'  --download-archive downloaded.txt --prefer-ffmpeg --external-downloader aria2c --external-downloader-args -x5 -ciw https://www.linkedin.com/learning/serverless-architecture --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', './cookies.txt', '-f', 'progressive-360p', '--playlist-start', '1', '--playlist-end', '10', '-o', '%(playlist)s - %(playlist_index)s - %(title)s.%(ext)s', '--download-archive', 'downloaded.txt', '--prefer-ffmpeg', '--external-downloader', 'aria2c', '--external-downloader-args', '-x5', '-ciw', 'https://www.linkedin.com/learning/serverless-architecture', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.06.06
[debug] Python version 3.6.9 (CPython) - Linux-5.3.0-53-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
ERROR: An extractor error has occurred. (caused by KeyError('JSESSIONID',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/linkedin.py", line 158, in _real_extract
    course_data = self._call_api(course_slug, 'chapters,description,title')
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/linkedin.py", line 36, in _call_api
    'Csrf-Token': self._get_cookies(api_url)['JSESSIONID'].value,
KeyError: 'JSESSIONID'
Traceback (most recent call last):
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/linkedin.py", line 158, in _real_extract
    course_data = self._call_api(course_slug, 'chapters,description,title')
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/linkedin.py", line 36, in _call_api
    'Csrf-Token': self._get_cookies(api_url)['JSESSIONID'].value,
KeyError: 'JSESSIONID'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/home/ramesh/default/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 543, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('JSESSIONID',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Any idea how to fix it..?

WOW. THANK YOU! Now I used the FireFox Add on "Export Cookies" to Export the Cookies and it works! Thank you!
https://addons.mozilla.org/de/firefox/addon/export-cookies-txt/?src=search

hello is it still working please inform me

Thank You!
still works great but right now I'm using YT-DLP


[WIP] Add dask operations to spectral-cube

This is an experiment in progress to see how and whether we can use dask under the hood to enable massively parallel applications of expensive operations w/o loading too much into memory and without handling it ourselves manually w/joblib.
See #566 for some of my notes prior to writing this.
TODO:

[x] verify that dasking works at all
[ ] come up with some specific test cases in which dask operations make sense (e.g., applying sigmaclip along an axis - very expensive)
[ ] develop benchmarks (and tests), run them on different systems...
[ ] compare performance w/joblib approach
[ ] search for further optimizations using mask (chunk size, for example)
[ ] determine whether we should be using dask masked arrays instead of loading a filled-masked-array, or what options there are for creating and utilizing the arrays

Coverage decreased (-0.6%) to 76.287% when pulling 39337bfba65b12f084a1eae3933cdf32dfeac08b on keflavich:dask_experiments into efb1f05aeb8065b9e79dc187dfc640a61d54a212 on radio-astro-tools:master.

dask
{1: 10.755578144000001, 2: 8.903129371999999, 3: 8.853326063999997, 4: 9.675582806999998}

memmap=False
{1: 10.592345619999996, 2: 9.260779098999997, 3: 9.415384600000003, 4: 9.408817252999995}

memmap=True
{1: 10.817149012999991, 2: 8.418423907000005, 3: 6.104963103999992, 4: 6.64484474599999, 0: 10.399119409000008}

Still unclear how to apply along each image.  Application across spectra appears to work really well?

In case anyone is following along, my last few commits are primarily notes-to-self and experiments.  Some of the tests will be useful but most (all?) of the redundant code needs to be removed when we solve the dask-like-joblib problem

So I've made a lot more negative progress:

dask.distributed clients cannot write to memory maps
This works:

import numpy as np
import dask.array
x = np.memmap('blah', mode='w+', shape=(10,10), dtype=int)
zz = dask.array.from_array(np.arange(100).reshape(10,10))
dask.array.store(zz, x, lock=False, compute=True)

This does not:
import dask.distributed
client = dask.distributed.Client()
import numpy as np
import dask.array
x = np.memmap('blah', mode='w+', shape=(10,10), dtype=int)
zz = dask.array.from_array(np.arange(100).reshape(10,10))
dask.array.store(zz, x, lock=False, compute=True)

The obvious workaround fails:

def get_x():
    return np.memmap('blah', mode='r+', shape=(10,10), dtype=int)
dask.array.store(zz, dask.delayed(get_x)(), lock=False, compute=True)

It works if zz is a single contiguous array that doesn't get passed around to different workers, but fails to write the whole array to the memory map if there are multiple workers (at least, I think it's broken down by workers/processes).  In brief, I get something like [data, data, 0, 0] or [0, data, 0, data] if I try this workaround w/4 slices.

The workaround for THAT fails trivially:

dask.array.store(zz, dask.delayed(get_x)(), lock=True, compute=True)
...
TypeError: can't pickle _thread.lock objects

Maybe this works?   To have it recognized inside a file reader or other code, enable it with 'u.add_enabled_units'. For details, see http://docs.astropy.org/en/latest/units/combining_and_defining.html

This error is not reproducible, but it does recur occasionally.  What does it mean?  It's... not true, right?  'm/s' does parse as unit...

Another interesting traceback, which is probably unrelated to dask, but is related to the tests:
Exception in thread QueueFeederThread:
Traceback (most recent call last):
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/queues.py", line 150, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/reduction.py", line 243, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/reduction.py", line 236, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 267, in dump
    return Pickler.dump(self, obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 887, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 400, in save_function
    self.save_function_tuple(obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 594, in save_function_tuple
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 510, in save
    rv = reduce(obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/_memmapping_reducer.py", line 304, in __call__
    return _reduce_memmap_backed(a, m)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/_memmapping_reducer.py", line 218, in _reduce_memmap_backed
    offset += m.offset
AttributeError: 'numpy.ndarray' object has no attribute 'offset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/adam/miniconda3/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/Users/adam/miniconda3/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/queues.py", line 175, in _feed
    onerror(e, obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 310, in _on_queue_feeder_error
    self.thread_wakeup.wakeup()
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 155, in wakeup
    self._writer.send_bytes(b"")
  File "/Users/adam/miniconda3/lib/python3.7/multiprocessing/connection.py", line 183, in send_bytes
    self._check_closed()
  File "/Users/adam/miniconda3/lib/python3.7/multiprocessing/connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/queues.py", line 150, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/reduction.py", line 243, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/loky/backend/reduction.py", line 236, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 267, in dump
    return Pickler.dump(self, obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 887, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 400, in save_function
    self.save_function_tuple(obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 594, in save_function_tuple
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/Users/adam/miniconda3/lib/python3.7/pickle.py", line 510, in save
    rv = reduce(obj)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/_memmapping_reducer.py", line 304, in __call__
    return _reduce_memmap_backed(a, m)
  File "/Users/adam/miniconda3/lib/python3.7/site-packages/joblib/_memmapping_reducer.py", line 218, in _reduce_memmap_backed
    offset += m.offset
AttributeError: 'numpy.ndarray' object has no attribute 'offset'
"""

The above exception was the direct cause of the following exception:

PicklingError                             Traceback (most recent call last)
<timed exec> in <module>()

~/repos/spectral-cube/spectral_cube/spectral_cube.py in wrapper(*args, **kwargs)
    121     @wraps(func)
    122     def wrapper(*args, **kwargs):
--> 123         return func(*args, **kwargs)
    124
    125     line1 = wrapper.__doc__.split("\n")[1]

~/repos/spectral-cube/spectral_cube/spectral_cube.py in spatial_smooth(self, kernel, convolve, **kwargs)
   3007
   3008         newcube = self.apply_function_parallel_spatial(_gsmooth_image,
-> 3009                                                        **kwargs)
   3010
   3011         return newcube

~/repos/spectral-cube/spectral_cube/spectral_cube.py in apply_function_parallel_spatial(self, function, num_cores, verbose, use_memmap, parallel, use_dask, **kwargs)
   3288                                                   use_memmap=use_memmap,
   3289                                                   use_dask=use_dask,
-> 3290                                                   **kwargs)
   3291
   3292     def apply_function_parallel_spectral(self,

~/repos/spectral-cube/spectral_cube/spectral_cube.py in _apply_function_parallel_base(self, iteration_data, function, applicator, num_cores, verbose, use_memmap, parallel, use_dask, memmap_dir, update_function, **kwargs)
   3193                                                               function,
   3194                                                               **kwargs)
-> 3195                                           for arg in iteration_data)
   3196             except ImportError:
   3197                 if num_cores is not None and num_cores > 1:

~/miniconda3/lib/python3.7/site-packages/joblib/parallel.py in __call__(self, iterable)
    932
    933             with self._backend.retrieval_context():
--> 934                 self.retrieve()
    935             # Make sure that we get a last message telling us we are done
    936             elapsed_time = time.time() - self._start_time

~/miniconda3/lib/python3.7/site-packages/joblib/parallel.py in retrieve(self)
    831             try:
    832                 if getattr(self._backend, 'supports_timeout', False):
--> 833                     self._output.extend(job.get(timeout=self.timeout))
    834                 else:
    835                     self._output.extend(job.get())

~/miniconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    519         AsyncResults.get from multiprocessing."""
    520         try:
--> 521             return future.result(timeout=timeout)
    522         except LokyTimeoutError:
    523             raise TimeoutError()

~/miniconda3/lib/python3.7/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~/miniconda3/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

PicklingError: Could not pickle the task to send it to the workers.

this does not affect the new tests I wrote, so my best guess is there's some failure related to FITS reading?

The E           _pickle.PicklingError: Could not pickle the task to send it to the workers. error comes from memory-mapped data + joblib; see 9798d1c

I've run some tests on cville machines, and they seem to just crash the session.   That could be a memory allocation failure or something else, it's not clear; all I know is it's a failure.

I'm now getting a whole bunch of un-reproducible errors in which py.test sessions fail, but running the same commands in independent sessions succeeds.  I suspect something related to client creation/teardown...

List of favorite error messages that have come up:

OSError: AF_UNIX path too long
error: [Errno 28] No space left on device
OSError: [Errno 24] Too many open files
E   ValueError: 'm/s' did not parse as unit: Syntax error parsing unit 'm/s' If this is meant to be a custom unit, define it with 'u.def_unit'. To have it recognized inside a file reader or other code, enable it with 'u.add_enabled_units'. For details, see http://docs.astropy.org/en/latest/units/combining_and_defining.html

After cleaning out the tempdirs on cv, I got the parallel tests to pass (!!).  However, another failure down the line related to memory allocation showed up:
spectral_cube/lower_dimensional_structures.py .                                                                                [  0%]
spectral_cube/spectral_axis.py .                                                                                [  0%]
spectral_cube/tests/test_analysis_functions.py .........                                                                                [  1%]
spectral_cube/tests/test_casafuncs.py ........                                                                                [  2%]
spectral_cube/tests/test_cube_utils.py ..                                                                                [  2%]
spectral_cube/tests/test_io.py .........                                                                                [  3%]
spectral_cube/tests/test_masks.py ..................................................................X..                                                                                [ 11%]
spectral_cube/tests/test_moments.py ..............................................................................................                                                                                [ 21%]
spectral_cube/tests/test_parallel_methods.py ................................x.x.............................................................x.x................................                                                 [ 36%]
spectral_cube/tests/test_performance.py ...ssF

...

        # writing the cube should not occupy any more memory
        snap3 = tracemalloc.take_snapshot()
        diff = snap3.compare_to(snap2, 'lineno')
>       assert sum([dd.size_diff for dd in diff])*u.B < 100*u.kB
E       assert (959615 * Unit("byte")) < (100 * Unit("kbyte"))
E        +  where 959615 = sum([250072, 140976, 137941, 126768, 120048, 42028, ...])
E        +  and   Unit("byte") = u.B
E        +  and   Unit("kbyte") = u.kB

I don't know if we can (or should) loosen the memory clearing constraints, but this still seems small enough that it's not an issue.

a dask approach that does work:
source = intake.open_fits_array('W51-E_B6_spw0_12M_lines.image.fits', ext=0, chunks=(1, 1920, 128, 128))
darr = source.to_dask()
sigclip_op = dask.array.apply_along_axis(arr=darr.rechunk((1,1920,90,90)),
                                             func1d=stats.sigma_clip, sigma=1.8, axis=1, shape=(1920,))
sigclip_op.to_zarr(url='W51-E_B3_spw3_12M_lines.image.sigmaclipped.zarr')

i.e., use intake to read the cube, then rechunk to have ~100 MB chunks and force shape=(1920,) (that's the key piece I had missed on earlier tries w/this same approach), then dump the result out to a zarr rather than try to keep it in memory.
Note that my attempt to chain sigclip_op w/mean failed

@keflavich - I have rebased this to include the latest CASA improvements. I have also implemented sigma_clip_spectrally_dask which seems to perform well. Here is an example script I used to profile this compared to joblib and to raw Numpy:
import time
import numpy as np
from astropy.stats import mad_std, sigma_clip
from astropy.io import fits
from spectral_cube import SpectralCube
import dask
dask.config.set(scheduler='single-threaded')

cube = SpectralCube.read('/home/tom/Data/ALMA-IMF/n2hp/G327.29_B3_spw0_12M_n2hp.contsub.image')[:, 495:500, 495:500]

print(cube)

time1 = time.time()
raw_result = sigma_clip(cube.unitless_filled_data[...], sigma=3, stdfunc=mad_std, axis=0)
hdu = fits.ImageHDU(raw_result)

with open("test_clipped0.fits", "wb+") as fobj:
    hdu.writeto(fobj)
time2 = time.time()

print('sigma_clip [raw]', time2-time1)

time1 = time.time()
mcube_dask = cube.sigma_clip_spectrally(3, num_cores=8, verbose=True)
hdu = fits.ImageHDU(mcube_dask._data)

with open("test_clipped1.fits", "wb+") as fobj:
    hdu.writeto(fobj)
time2 = time.time()

print('sigma_clip_spectrally', time2-time1)

time1 = time.time()
mcube_dask = cube.sigma_clip_spectrally_dask(3)
hdu = fits.ImageHDU(mcube_dask._data)

with open("test_clipped2.fits", "wb+") as fobj:
    hdu.writeto(fobj)
time2 = time.time()

print('sigma_clip_spectrally_dask', time2-time1)

Note that the [:, 495:500, 495:500] is because the joblib implementation is so slow that it would take around 6000 hours to complete the whole cube :scream:. Also to really test performance properly the above requires https://github.com/astropy/astropy/pull/9742 - by the way this is the reason for the slightly weird writing out of FITS. I'm also including the writing out of FITS in all cases so that we measure the same operations for all cases (for dask, the array is evaluated at I/O time).
Another caveat is that I think there is an unrelated bug with beam averaging with this cube so I replaced:
return self._handle_beam_areas_wrapper(origfunc)

with
return origfunc

in spectral_cube.py.
The output in the above case is:
VaryingResolutionSpectralCube with shape=(217, 5, 5) and unit=Jy / beam:
 n_x:      5  type_x: RA---SIN  unit_x: deg    range:   238.316655 deg:  238.316867 deg
 n_y:      5  type_y: DEC--SIN  unit_y: deg    range:   -54.630326 deg:  -54.630203 deg
 n_s:    217  type_s: FREQ      unit_s: Hz     range:<PHONE_NUMBER>4.272 Hz:93195455580.656 Hz
sigma_clip [raw] 0.6289448738098145

[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done  25 out of  25 | elapsed:   22.6s finished
sigma_clip_spectrally 22.601516723632812

sigma_clip_spectrally_dask 1.1982429027557373

Note that in some experiments I did the results of the joblib approach does not agree with the two other approaches - whereas the dask method agrees with the plain numpy one, so it may be there is a bug in the joblib approach.
Then if I ignore the joblib approach, to test performance of dask versus Numpy, and do the whole cube, I get:
VaryingResolutionSpectralCube with shape=(217, 1728, 2240) and unit=Jy / beam:
 n_x:   2240  type_x: RA---SIN  unit_x: deg    range:   238.224785 deg:  238.343018 deg
 n_y:   1728  type_y: DEC--SIN  unit_y: deg    range:   -54.645441 deg:  -54.592671 deg
 n_s:    217  type_s: FREQ      unit_s: Hz     range:<PHONE_NUMBER>4.272 Hz:93195455580.656 Hz

sigma_clip [raw] 89.39581990242004

sigma_clip_spectrally_dask<PHONE_NUMBER>8242493

For now this is just using single-threaded dask, but I'm trying to get the parallel version to work. For now the dask version is slightly slower but on the other hand uses less memory. The joblib version is ~200,000 times slower :thinking:
Here are some memory graphs - the most important here is the top left panel, since it does not count the cache/buffers which are used by the memory mapping.
Memory usage of Numpy version

Memory usage of dask version

So the Numpy version uses 25Gb of RAM and the dask version peaks at 0.8Gb.
The sigma_clip_spectrally_dask function is super simple, which makes me think that we can significantly simplify the current code base by using a similar approach for each different methods. I'm not even sure it's worth trying to be smart about sending all the different methods to a centralized generalized method since the code is so simple to start with and dask is basically the generalized framework.
@keflavich - let me know what you think, feel free to try it out (remember to use the correct branch of astropy as mentioned above), and let me know if you'd like me to extend this to other methods.

I like the approach.  We clearly need to simplify the write-to-disk interface.  I'll experiment with the new code.

First try resulted in:
In [9]: with open('W51-IRS2_B6_spw1_12M_spw1.image.pbcor.sigmaclip.fits', 'wb+') as fh:
   ...:     hdu.writeto(fh)
   ...:
Killed

I like the approach. We clearly need to simplify the write-to-disk interface. I'll experiment with the new code.

Yeah I need to adapt the FITS writer - the above was just temporary while we test out specifically the sigma clipping function.
Could you try writing the dask array from the cube into e.g. zarr to see if you also run into issues there?

Yes, I verified that I'm on that branch.
In [1]: import zarr

In [2]: from spectral_cube import SpectralCube

In [3]: from astropy.io import fits

In [4]: from astropy.stats import mad_std, sigma_clip

In [5]: cube = SpectralCube.read('W51-IRS2_B6_spw1_12M_spw1.image.pbcor', format='casa_image').mask_out_bad_beams(0.1)
WARNING: StokesWarning: Cube is a Stokes cube, returning spectral cube for I component [spectral_cube.io.core]

In [6]: mcube_dask = cube.sigma_clip_spectrally_dask(3, stdfunc=mad_std)

In [7]: mcube_dask._data.to_zarr('W51-IRS2_B6_spw1_12M_spw1.image.pbcor.sigmaclip.zarr')
Killed

Is there any way for me to increase the verbosity of the process so we see where it gets killed?  I know one can monitor dask tasks, but setting up that monitoring console is not trivial on this machine.

@keflavich - I'll investigate verbose options - in the mean time can you try and enforce the serial scheduler for dask to see if we are looking at parallelization issues? So do:
import dask
dask.config.set(scheduler='single-threaded')
at the start of your IPython session.

Just for the record:
import dask
dask.config.set(scheduler='single-threaded')

import zarr
from spectral_cube import SpectralCube
from astropy.io import fits
from astropy.stats import mad_std, sigma_clip

cube = SpectralCube.read('W51-IRS2_B6_spw0_12M_spw0.image.pbcor', format='casa_image').mask_out_bad_beams(0.1)

mcube_dask = cube.sigma_clip_spectrally_dask(3, stdfunc=mad_std)

mcube_dask._data.to_zarr('W51-IRS2_B6_spw0_12M_spw1.image.pbcor.sigmaclip.zarr')

runs in just under 6 minutes on my laptop.

superceded by #618


Added Contact Us Page

🛠 Fixes Issue : "Closes #42"

👨💻 Changes proposed:

Added Contact Us Page.
Also added redirection URL's to navbar
✔ Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.
[x] I am a participant of Hacktoberfest 2022.

📄 Note to reviewers

📷 Screenshots

Thank you for the wonderful contribution. Please upgrade your forked repository✨

I have upgraded my forked repository.


CMake: install target not working: fails: file INSTALL cannot find absl-config.cmake

I tried building and installing the source code from git (I tried both master and tags/v7.0-beta1 with the same error) on Mac OS, installing as dependencies only wget/cmake (using homebrew).
Configuring, building and even testing using the standard CMake workflow
mkdir build && cd build
cmake ..
make -j8

work fine (except for tons of warnings regarding the deprecated "register" keyword and some .cc-files without symbols).
However, running make install (or, equivalently, cmake --build . --target install) fails because the file build/abseil-cpp-src/CMake/absl-config.cmake is missing.
I tried to run
find build | grep absl-config

too see whether the file is present somewhere else, but there appears to be no such file in the build directory.
-- Installing: /usr/local/lib/cmake/protobuf/protobuf-module.cmake
-- Installing: /usr/local/lib/cmake/protobuf/protobuf-options.cmake
-- Installing: /usr/local/lib/cmake/protobuf/protobuf-config.cmake
-- Installing: /usr/local/lib/cmake/protobuf/protobuf-config-version.cmake
-- Installing: /usr/local/lib/cmake/absl/absl-targets.cmake
-- Installing: /usr/local/lib/cmake/absl/absl-targets-release.cmake
CMake Error at abseil-cpp-build/cmake_install.cmake:55 (file):
  file INSTALL cannot find
  "/Users/me/Source/or-tools/build/abseil-cpp-src/CMake/absl-config.cmake".
Call Stack (most recent call first):
  external/cmake_install.cmake:41 (include)
  cmake_install.cmake:103 (include)

If I manually add an empty file in its place, the install goes through without error. However, as expected, I cannot use ortools in my cmake project with an empty absl-config.cmake file installed:
find_package(ortools CONFIG REQUIRED)

yields the following error:
CMake Error at CMakeLists.txt:12 (find_package):
  Found package configuration file:

    /usr/local/lib/cmake/ortools/ortoolsConfig.cmake

  but it set ortools_FOUND to FALSE so package "ortools" is considered to be
  NOT FOUND.  Reason given by package:

  The following imported targets are referenced, but are missing:
  absl::absl_base absl::absl_container absl::absl_hash absl::absl_memory
  absl::absl_meta absl::absl_str_format absl::absl_strings
  absl::absl_synchronization absl::absl_types

I then proceeded to find absl-config.cmake in /patches (in the source directory).
This file seems to contain a syntax error at least in tags/v7.0-beta1 (missing closing parenthesis).
I guess something was supposed to copy that file to the build directory where install expects it; I did that manually, retried the install, and it worked.
I still had no luck consuming ortools cmake config files in my cmake project, still with the same error message as before.
Adding a
find_package(absl CONFIG REQUIRED)
before that fixed this issue, so that configuring, building and linking (in my own CMake project) against ortools::ortools now works.
Was I doing something wrong here, like missing a "patching" step when running cmake? Then that should maybe be included somewhere in the documentation.

Don't want to block the release v7.0 which BTW only use the Makefile-based build to generate all its binary artifacts.
So I will postpone it to 7.1 since we need a deep rework of all the CMake (experimental) stuff.


Test

@mohammadJBYaseen Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@mohammadJBYaseen Thank you for signing the Contributor License Agreement!


More issues with expired subkeys

See keybase/keybase-issues#1776

I worry this is going to be an issue with PGP key merging and non-expiring keys.

Any updates on this issue?

kourier's key (https://keybase.io/kourier) is supposed to be non-expiring. And indeed, when I import it by hand, that's what I see. But when the node client imports it, it comes out with a non-null expiration time. It turns out that the difference is the time_travel option. I don't yet know why. Any immediate theories @maxtaco?
Example code: https://gist.github.com/oconnor663/6ba322b1ad2a3fe8f257

let's go over it today

I addressed this in keybase/kbpgp#101.  In review now by @oconnor663

keybase/kbpgp#101 is merged and live.  Available in v2.0.47 of kbpgp.

node-client has been pushed


LoadFrom() now uses envy to load environment variables before establi…

I had an issue with my environment where the database connection details in my .env file were not being consumed by pop's config loading logic.  The simple solution was to use envy.Load() before the connections were made.  This allows the environment variables to replace the placeholder tokens in database.yml.
The only hang up I have with this solution is the dependency to the envy package.
Example database.yml:
development:
  dialect: "mysql"
  database: "mailedit_development"
  host: {{ env "DATABASE_HOST" }}
  port: {{ env "DATABASE_PORT" }}
  user: {{ env "DATABASE_USERNAME" }}
  password: {{ env "DATABASE_PASSWORD" }}

Thanks @paganotoni  for the fix suggestion

This looks good, but I would recommend not using envy.Load(), but instead changing the os.Getenv lines a few lines down. That will have the same affect, but it will allow for extra benefits of using envy.Get (like setting temp vars, etc...)

@markbates I hesitated making that change, but I agree with your suggestion.  It makes sense.
func LoadFrom(r io.Reader) error {
	tmpl := template.New("test")
	tmpl.Funcs(map[string]interface{}{
		"envOr": func(s1, s2 string) string {
			return envy.Get(s1, s2)
		},
		"env": func(s1 string) string {
			val, err := envy.MustGet(s1)
			if err != nil {
				return ""
			}
			return val
		},
	})
...

@markbates I'll make the changes to the Getenv calls a few lines below, but I believe envy.Load() is still required.  Get and MustGet in the envy package rely on a map generated by the Load() func.

Thanks, I already made the changes so I could get the next buffalo release out. A PR was made earlier today that makes sure to load .env files when the envy package init happens, so with that update and this one, everything should be good to go. Thanks a lot

Awesome


Do not show registration page when school registration is closed

Finishes the work of the previous PR to add a "close pilot for school" journey by ensuring that the registration page does not show when the school's registration is closed.
Also adds a test for the close pilot registration journey.

What happens if you try to submit a registration form (that you started while registration was still open) after registration is closed?

What happens if you try to submit a registration form (that you started while registration was still open) after registration is closed?

There are no guards against that, so it would go through.


Publish archives for the month of June 2024

Review and publish archives
For each of the following archives, find the run status in the Github archiver run. If validation tests pass, manually review the archive and publish. If no changes detected, delete the draft. If changes are detected, manually review the archive following the guidelines in step 3 of README.md, then publish the new version. Then check the box here to confirm publication status, adding a note on the status (e.g., "v1 published", "no changes detected, draft deleted"):
- [x] eia176 - No changes detected, draft deleted.
- [x] eia191 - v7 published
- [x] eia757a - No changes detected, draft deleted
- [x] eia860 - No changes detected, draft deleted
- [x] eia861 - No changes detected, draft deleted
- [x] eia923 - v17 published
- [x] eia930 - v5 published
- [x] eiawater - No changes detected, draft deleted
- [x] eia_bulk_elec - v9 published
- [x] epacamd_eia - No changes detected, draft deleted
- [x] ferc60 - v6 published
- [x] mshamines - v6 published.
- [x] phmsagas - No changes detected, draft deleted.
- [x] epacems - v10 published

Validation failures
For each run that failed because of validation test failures (seen in the GHA logs), add it to the tasklist. Download the run summary JSON by going into the "Upload run summaries" tab of the GHA run for each dataset, and follow the link. Investigate the validation failure.
If the validation failure is deemed ok after manual review (e.g., Q2 of 2024 data doubles the size of a file that only had Q1 data previously, but the new data looks as expected), go ahead and approve the archive and leave a note explaining your decision in the task list.
If the validation failure is blocking (e.g., file format incorrect, whole dataset changes size by 200%), make an issue to resolve it.
- [x] eia860m - "The following files have absolute changes in file size >|25%|: {'eia860m-2024.zip': 0.32988297082960333}" - Added new month of data, all looks ok. V21 published.
- [x] ferc714 - "The following files have absolute changes in file size >|25%|: {'ferc714-xbrl-2023.zip': 7.36364212353986}". Substantial increase in data available for 2023. V9 published.
- [ ] https://github.com/catalyst-cooperative/pudl-archiver/issues/355
- [ ] ferc2 - "The following files have absolute changes in file size >|25%|: {'ferc2-xbrl-2023.zip': 0.445777352478636}". INVESTIGATE - File shrinks in size by 44%.
- [ ] ferc6 - "The following files have absolute changes in file size >|25%|: {'ferc6-xbrl-2023.zip': 0.3979477881736035}". INVESTIGATE - File shrinks in size by 40%.

Other failures
For each run that failed because of another reason (e.g., underlying data changes, code failures), create an issue describing the failure and take necessary steps to resolve it.
- [ ] eiaaeo - "The record has been deleted". Emailed Zenodo support 6/11.
- [ ] nrelatb - "The record has been deleted". Emailed Zenodo support 6/11.

Relevant logs
Link to logs from GHA run

Superceded by #369, closing.


Failed to make

I tried to compile the library. However, I get the following error message. Could you please give me some clue on how to solve this or what is going wrong?
CC       core/ibv/ibprof_ibv.lo
In file included from core/ibv/ibprof_ibv.c:18:0:
/usr/include/infiniband/arch.h:37:2: warning: #warning "This header is obsolete." [-Wcpp]
#warning "This header is obsolete."
^~~~~~~
core/ibv/ibprof_ibv.c: In function ‘ibv_open_device_handler’:
core/ibv/ibprof_ibv.c:754:13: error: ‘struct ibv_context_ops’ has no member named ‘query_port’; did you mean ‘_compat_query_port’?
check_api(query_port);
^
core/ibv/ibprof_ibv.c:706:12: note: in definition of macro ‘check_api’
ret->ops.func = ibv_module_context.mean.ibv##_func;
^~~~~
In file included from core/ibv/ibprof_ibv.c:17:0:
core/ibv/ibprof_ibv.c: At top level:
core/ibv/ibprof_ibv.c:58:26: error: expected declaration specifiers or ‘...’ before ‘__builtin_constant_p’
struct ibv_mr* TYPE ## ibv_reg_mr(struct ibv_pd *pd, void *addr, size_t length, int access)
^
core/ibv/ibprof_ibv.c:621:9: note: in expansion of macro ‘HAVE_IBV_REG_MR_FUNC’
HAVE_IBV_REG_MR_FUNC(TYPE)
^~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:841:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_PROTOTYPED’
DECLARE_OPTION_FUNCTIONS_PROTOTYPED( )
^
In file included from core/ibv/ibprof_ibv.c:20:0:
core/ibv/ibprof_ibv.c: In function ‘NONEibv_query_port’:
core/ibv/ibprof_ibv.c:666:101: warning: passing argument 3 of ‘f’ from incompatible pointer type [-Wincompatible-pointer-types]
BV, ibv_query_port, _compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:847:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(NONE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:666:101: note: expected ‘struct compat_ibv_port_attr *’ but argument is of type ‘struct ibv_port_attr *’
BV, ibv_query_port, compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:847:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(NONE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c: At top level:
core/ibv/ibprof_ibv.c:846:24: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
#define PREFIX_NONE(x) NONE##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_NONE’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:847:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(NONE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:846:24: note: (near initialization for ‘ibv_NONE_funcs.ibv_query_port’)
#define PREFIX_NONE(x) NONE##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_NONE’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:847:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(NONE)
^~~~~~~~~~~~~~
In file included from core/ibv/ibprof_ibv.c:20:0:
core/ibv/ibprof_ibv.c: In function ‘PROFibv_query_port’:
core/ibv/ibprof_ibv.c:666:101: warning: passing argument 3 of ‘f’ from incompatible pointer type [-Wincompatible-pointer-types]
BV, ibv_query_port, _compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:850:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(PROF)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:666:101: note: expected ‘struct compat_ibv_port_attr *’ but argument is of type ‘struct ibv_port_attr *’
BV, ibv_query_port, compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:850:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(PROF)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c: At top level:
core/ibv/ibprof_ibv.c:849:24: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
#define PREFIX_PROF(x) PROF##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_PROF’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:850:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(PROF)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:849:24: note: (near initialization for ‘ibv_PROF_funcs.ibv_query_port’)
#define PREFIX_PROF(x) PROF##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_PROF’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:850:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(PROF)
^~~~~~~~~~~~~~
In file included from core/ibv/ibprof_ibv.c:20:0:
core/ibv/ibprof_ibv.c: In function ‘VERBOSEibv_query_port’:
core/ibv/ibprof_ibv.c:666:101: warning: passing argument 3 of ‘f’ from incompatible pointer type [-Wincompatible-pointer-types]
BV, ibv_query_port, _compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:853:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(VERBOSE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:666:101: note: expected ‘struct compat_ibv_port_attr *’ but argument is of type ‘struct ibv_port_attr *’
BV, ibv_query_port, compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:853:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(VERBOSE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c: At top level:
core/ibv/ibprof_ibv.c:852:27: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
#define PREFIX_VERBOSE(x) VERBOSE##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_VERBOSE’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:853:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(VERBOSE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:852:27: note: (near initialization for ‘ibv_VERBOSE_funcs.ibv_query_port’)
#define PREFIX_VERBOSE(x) VERBOSE##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_VERBOSE’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:853:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(VERBOSE)
^~~~~~~~~~~~~~
In file included from core/ibv/ibprof_ibv.c:20:0:
core/ibv/ibprof_ibv.c: In function ‘TRACEibv_query_port’:
core/ibv/ibprof_ibv.c:666:101: warning: passing argument 3 of ‘f’ from incompatible pointer type [-Wincompatible-pointer-types]
BV, ibv_query_port, _compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:856:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(TRACE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:666:101: note: expected ‘struct compat_ibv_port_attr *’ but argument is of type ‘struct ibv_port_attr *’
BV, ibv_query_port, compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:856:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(TRACE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c: At top level:
core/ibv/ibprof_ibv.c:855:25: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
#define PREFIX_TRACE(x) TRACE##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_TRACE’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:856:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(TRACE)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:855:25: note: (near initialization for ‘ibv_TRACE_funcs.ibv_query_port’)
#define PREFIX_TRACE(x) TRACE##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_TRACE’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:856:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(TRACE)
^~~~~~~~~~~~~~
In file included from core/ibv/ibprof_ibv.c:20:0:
core/ibv/ibprof_ibv.c: In function ‘ERRibv_query_port’:
core/ibv/ibprof_ibv.c:666:101: warning: passing argument 3 of ‘f’ from incompatible pointer type [-Wincompatible-pointer-types]
BV, ibv_query_port, _compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:859:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(ERR)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:666:101: note: expected ‘struct compat_ibv_port_attr *’ but argument is of type ‘struct ibv_port_attr *’
BV, ibv_query_port, compat_query_port, context, context, port_num, port_attr) };
^
core/ibv/ibprof_ibv.h:132:13: note: in definition of macro ‘FUNC_BODY_INT’
ret = f(VA_ARGS);
^~~~~~~~~~~
core/ibv/ibprof_ibv.c:835:1: note: in expansion of macro ‘DECLARE_OPTION_FUNCTIONS_INLINE’
DECLARE_OPTION_FUNCTIONS_INLINE(type)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:859:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(ERR)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c: At top level:
core/ibv/ibprof_ibv.c:858:23: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
#define PREFIX_ERR(x) ERR##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_ERR’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:859:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(ERR)
^~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:858:23: note: (near initialization for ‘ibv_ERR_funcs.ibv_query_port’)
#define PREFIX_ERR(x) ERR##x,
^
core/ibv/ibprof_ibv.c:481:2: note: in expansion of macro ‘PREFIX_ERR’
OP(ibv_query_port)
^~
core/ibv/ibprof_ibv.c:544:2: note: in expansion of macro ‘OP_ON_MEMBERS_LIST’
OP_ON_MEMBERS_LIST(PREFIX####TYPE)
^~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:836:1: note: in expansion of macro ‘DECLARE_OPTION_STRUCT’
DECLARE_OPTION_STRUCT(type)
^~~~~~~~~~~~~~~~~~~~~
core/ibv/ibprof_ibv.c:859:1: note: in expansion of macro ‘DECLARE_OPTION’
DECLARE_OPTION(ERR)
^~~~~~~~~~~~~~
Makefile:588: recipe for target 'core/ibv/ibprof_ibv.lo' failed
make[2]: *** [core/ibv/ibprof_ibv.lo] Error 1
make[2]: Leaving directory '/home/bernard/rdma_exp/libibprof/src'
Makefile:532: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/bernard/rdma_exp/libibprof'
Makefile:408: recipe for target 'all' failed
make: *** [all] Error 2

I got the same error, do you have any hint on how to resolve this?


Binaries are not signed

The reogrid binaries are not signed using the strong-name system https://docs.microsoft.com/en-us/dotnet/framework/app-domains/create-and-use-strong-named-assemblies
This has the side effect that it is not possible to create a signed assembly that has reogrid as a dependency.

Now the DLLs will be published as strong-named assemblies since v3.0.0.
Thanks for your suggestion!


add santacoder fine-tuning

Add the first version of santacoder fine-tuning, and will add more detailed soon.

Nice let's merge?

@Muennighoff Merged!


Powermock ignores the answer in the @Mock annotation

From<EMAIL_ADDRESS>on March 06, 2014 09:21:47
What steps will reproduce the problem? Write a test, where a field is annotated with @Mock(answer = Answers.CALLS_REAL_METHODS) What is the expected output? What do you see instead? In Mockito, this works perfectly. It simply calls the real methods of the mock.
In PowerMock, this does nothing. It's simply a mock. You have to mock the methods to get any result. What version of the product are you using? On what operating system? PowerMockito 1.5.4
Mockito 1.9.5
Mac OS X 10.9.2 Please provide any additional information below. Workaround for this problem: let's say that I have this mock present in my test:
@Mock(answer = Answers.CALLS_REAL_METHODS) private TimestampToStringConverter timestampToStringConverter;
In order to have it actually call the real methods of the mock, I have to do this in my setup method:
when(timestampToStringConverter.convert(any(Timestamp.class))).thenCallRealMethod();
It is infuriating, however, because you have to actually do this for EVERY method you want to use in your test. If you have a class which has 8 methods, which you all use, you're doing this for every single one of them. It leads to an ugly blob of code.
Original issue: http://code.google.com/p/powermock/issues/detail?id=486

From<EMAIL_ADDRESS>on August 04, 2014 00:08:43
Status: Fixed


[BugFix]Fix colocate state inaccuracy bug

What type of PR is this：

[x] bugfix
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

This PR #9354 added the check logic that if the tablet has been in the tablet scheduler queue, then do not check the health status. But the isGroupStable will remain true if all the tablets are in the tablet scheduler queue, which makes the status  inaccurate. Set isGroupStable to false if there is tablet in the tablet scheduler queue.
Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] I have added user document for my new feature or new function

[FE PR Coverage Check]
:disappointed: fail : 1 / 2 (50.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
1
2
50.00%
[264]

run starrocks_admit_test

run starrocks_admit_test

run starrocks_fe_unittest

run starrocks_fe_unittest

run starrocks_fe_unittest

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

@Mergify backport branch-2.4

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/clone/ColocateTableBalancer.java
2
2
100.00%
[]


Wrap sqlite3 OPEN_URI option

Add a boolean option to the Database constructor controlling the SQLITE_OPEN_URI flag. Obviously this isn't a superset of #735, but it implements its most requested feature.
I'm open to bikeshedding w.r.t. the option name.
Please note that I disabled the directory existance check if uriPath is passed. If the URI requests that a custom sqlite VFS implementation is used, the db path might not be accessible by the node fs APIs. Furthermore an extension can install itself as the default VFS, so it might make sense to completely remove that check. (Also note that it generally suffers from TOCTUO issues -- the directory can be deleted after fs.existsSync() succeeded but before sqlite3_open_v2() tries to open the file).
Closes #483

Nice! Can you add some tests please?

Closing due to lack of activity / interest.


panic: interface conversion

panic: interface conversion: interface {} is user.WasRegisteredWithEmail, not *user.WasRegisteredWithEmail
goroutine 65 [running]:
github.com/vardius/go-api-boilerplate/cmd/user/internal/application/eventhandler.WhenUserWasRegisteredWithEmail.func1({0xada780, 0x4000283470}, 0x4000200880)
/app/cmd/user/internal/application/eventhandler/when_user_was_registered_with_email.go:22 +0x374
github.com/vardius/go-api-boilerplate/pkg/eventbus/memory.(*eventBus).Subscribe.func1({0xada780, 0x4000283470}, 0x4000200880, 0x40002d6840)
/app/pkg/eventbus/memory/event_bus.go:107 +0xe4
reflect.Value.call({0x7a6480, 0x400045d200, 0x13}, {0x8836e1, 0x4}, {0x400011f3e0, 0x3, 0x4})
/usr/local/go/src/reflect/value.go:543 +0x590
reflect.Value.Call({0x7a6480, 0x400045d200, 0x13}, {0x400011f3e0, 0x3, 0x4})
/usr/local/go/src/reflect/value.go:339 +0x8c
github.com/vardius/message-bus.(*messageBus).Subscribe.func1(0x400045d220)
/go/pkg/mod/github.com/vardius/message-bus@v1.1.4/bus.go:60 +0x74
created by github.com/vardius/message-bus.(*messageBus).Subscribe
/go/pkg/mod/github.com/vardius/message-bus@v1.1.4/bus.go:58 +0x278

I fixed the issue https://github.com/vardius/go-api-boilerplate/commit/1dca296ffc7d92f891b4dc4b7734ebfd37770453
but it would be good as well to update interface casting as well
-		e := event.Payload.(*user.WasRegisteredWithEmail)
+		e, ok := event.Payload.(*user.WasRegisteredWithEmail) // handle !ok

Thanks. I cloned the revised version. But sill no luck -- just the same error. I'm a golang newbie, but I guess the marshalling/unmarshalling does not work as expected, althougth the object factory had changed.

This https://github.com/vardius/go-api-boilerplate/commit/1b87ca51e43c982963db378a4cc01582e314c16f should help, I missed it earlier

Up and running. Thank you very much.


Split Panes

This adds an initial implementation of split panes, as shown in the screenshot below.

Features

Split panes can be created by hitting Cmd+D and Cmd+Shift+D (which are also accessible from the Window menu).
Split panes can be arbitrarily nested, and support tabs like you'd expect.
The active pane can be changed with mouse clicks, or by cycling through them using Show Next Pane/Show Previous Pane.
Panes can be closed with CmdOrCtrl+W.

How
There's definitely lots of ways to implement something like this, however I've opted for a pretty Redux heavy direction. The whole split pane structure is stored in a new reducer, term-groups.js, which reacts to create/remove/focus change actions and adapts the structure as needed.
Reducer Structure
The term-groups reducer maintains a tree like structure of the different tabs.
The split structure in the screenshot would resemble something like this:
   +-----+Root+----+
   |               |
   v               v
++Left++       ++Right++
|      |       |   +   |
v      v       v   v   v
S      S       S   S   S

Each term group would have the following properties:

uid - group identifier
parentUid - the parent's uid
sessionUid - if the group is a leaf node, conforms to the actual session's uid
direction - what direction the split is created in (only for internal nodes)
children - an array of uids of the term group's children.

Additionally, the reducer holds a mapping between each root term group and their respective active sessionUid, so that you can change tabs without losing the focus of the session you'd selected.
Caveats
To be able to create new term groups the SESSION_ADD action is hijacked, and the split direction (if it's a split that caused the new session to be created) is added here. This feels a bit dirty, and I'd love suggestions on how this could be improved (and suggestions on how to improve the data structure in general!).
Rendering
The actual split panes are rendered using react-split-pane at the moment, but this probably needs to be changed before this is ready for primetime. A few issues:

It doesn't support more than two split panes out of the box without nesting split panes (workaround).
The styling is a slightly awkward. This could be fixed with an upstream pull request to allow custom class names for the pane divider (resizer), so we could use it with Aphrodite.
Mouse resizing also seems a bit glitchy, but this might be an issue on our end as well - haven't spent too much time on it yet.

We could either look into fixing these issues upstream, or just implement our own flexbox heavy approach.
TODOs
✅  Preserving already created hterm instances
At the moment we're creating hterm instances at the bottom of the React hierarchy, in the Term component. This doesn't work very well with split panes, since Terms will be removed and recreated when the split pane structure is changed. That means the same happens to their respective hterm instances, which means that sessions lose their output when splitting (i.e. needs to be fixed before splits are usable).
One of the ways of solving this would be to create hterm instances at a higher level, and then pass them down to the Term component. This could either be in a React component such as Terms, or even in a Redux reducer. The latter may sound a bit crazy, but I think it might make sense together with the current reducer structure. I'm interested in more ideas around this though.
✅  Hotkeys
This definitely needs some bikeshedding. Even if we need a way of configuring hotkeys in the future, sane defaults will always be important. As mentioned splits are currently created with CtrlOrCmd+D and CtrlOrCmd+Shift+D. Cmd+D is definitely okay, but Ctrl+D obviously isn't 😙. Maybe we can change it based on the user's OS. What would be a decent hotkey set for Linux?
There also isn't a hotkey for rotating back and forth between your open split panes yet. iTerm defaults this to Cmd+[/Cmd+], however this doesn't work on a lot of European keyboards. Personally I'd prefer something like Ctrl+Tab/Ctrl+Shift+Tab, which we've currently set to changing tabs (afaik it doesn't work though, the combination is eaten by hterm).
EDIT: Cycling between split panes is currently bound to Ctrl+Alt+Tab and Ctrl+Shift+Alt+Tab.
In the future we should definitely also support changing panes based on direction (Cmd+Alt+Arrow in iTerm).
✅  Resizing
Drag and drop resizing of split panes sort of works at the moment, but it's really glitchy.
✅  Styling
Needs to support custom styling in the same way as the rest of HyperTerm. We could probably add some sort of indicator to show what pane is in focus (i.e. an overlay over the inactive panes, like iTerm) as well.
Bugs

[x] Clicking to change focus doesn't work when applications such as vtop/htop are open.
[x] Splitting panes will make git status output look funky (and probably other resizing issues) - screenshot.
[x] A tab will sometimes crash after a pane is closed, needs more investigation.
[x] Closing the last tab won't exit HyperTerm (regression).
[x] Creating multiple horizontal splits is buggy.
[x] Webviews don't respect focus changes as they should.

...and probably a lot more!
Binary
I really recommend pulling down the branch, however I've made a binary that can be downloaded here: https://github.com/ekmartin/hyperterm/releases/download/0.8.0-splits.5/mac.zip (updated 2016-10-02)

THAT'S FREAKING AWESOME

Great work !
I found a little bug when you reduce the height of a horizontally splitted tab, the bottom one keeps its original size.
But it seems to be a react-split-pane bug : if I add display: flexto the second Pane div, it fixes its size.
Maybe related to https://github.com/tomkp/react-split-pane/issues/106
How can we help you with this PR ?

Thanks!
@chabou I think the main steps in moving forward will be deciding on what the correct direction to go regarding actions/reducers is, and what to do with the hterm instance issue I mentioned above.
I'm going to be looking into what to do with react-split-pane, probably by seeing if reimplementing some of the functionality directly in HyperTerm might be better than using it.
If you're interested in helping out with some of the bugs that need to be squashed before it's ready that would be helpful. Quick summary:

Clicking to change focus doesn't work when applications such as vtop/htop are open.
Normally this is done by attaching a focus listener here, however this is never fired when you for example click on a term that has htop open.

A tab will sometimes crash after a pane is closed.
Try opening a few splits in different tabs, then closing them. Usually it works fine, but I've seen it crash a few times.

Hotkeys for cycling back and forth between split panes.

This feature (plus some minor bug corrections) is what stops me (aka the average coding Joe ^^) from switching from iTerm to hyperTerm
So I just drop this message as a support to the team! :)

seems like all todos are implemented and all bugs are resolved

@ekmartin @rauchg thank you for all the work done in this issue.
Can you share some news/plans on this pr?

@iamstarkov: We found a few bugs with multiple horizontal (Y-directional) split panes, we're looking into that now :)

It's crazy seeing a vital feature like this in progress in real time right now.  I'M SO STOKED

I just realized I'm testing zet:add/splits instead of ekmartin:splits. Which branch should we be testing?

@flybayer Doesn't Cmd/Ctrl + - conflict with zoom out?
We should just make the hotkeys configurable :smile:

@maxdeviant oops yes :laughing: I guess I'm actually using Meta + - and Meta + \ and forgot the difference. Is Meta + - and Meta + \ a viable default for this?

@flybayer: the exit bug should be fixed with b2ca957 :)

@flybayer: I added the split hotkeys Ctrl+Shift+O (horizontal) and Ctrl+Shift+E (vertical) - stolen from Gnome Terminator. If there's better options we can open an issue and do some bikeshedding after this has been merged :)

🎉 yay 🎉

any plans to release?

This would be really nice to get into mainline.

I think it is included in 0.8.1 release https://github.com/zeit/hyper/releases/tag/0.8.1

@naim @chabou thats true

@naim though you need to download hyper. before it was hyperterm and has been renamed

In Progress and Review Needed labels seem confusing 😄


Can not get docker to start

Hello!
I can not get the docker image to run. I'm getting this error in the log:
/usr/local/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory
Is there something I'm doing wrong?

Could I get some information on the hardware/software stack of the host?
You can try the -buster images. They might be better behaved with older/more obscure versions of Docker.

I'm running unRAID 6.11.5 and Docker 20.10.21.
I tried the lastest and stable versions along with the buster variant and
got the same error.
Let me know if you need any additional information!
On Fri, Jan 13, 2023, 10:23 PM Ryan Young @.***> wrote:

Could I get some information on the hardware/software stack of the host?
You can try the -buster images. They might be better behaved with
older/more obscure versions of Docker.
—
Reply to this email directly, view it on GitHub
https://github.com/YoRyan/mailrise/issues/50#issuecomment-1382657462,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHBRC7ZZPW2BACPDFHMS7FDWSIS3TANCNFSM6AAAAAAT3BWTCY
.
You are receiving this because you authored the thread.Message ID:
@.***>

Are you running this via docker run or compose? Can you share the command/compose you're using?
Is the error inside the container or the host system?

I've only tried with docker run so far. unRAID uses a form of templating for docker run so there are certain envs it adds. I haven't mapped any ports yet, but of course, those will be needed once I get the container to run.
docker run
  -d
  --name='Mailrise'
  --net='bridge'
  -e TZ="America/Chicago"
  -e HOST_OS="Unraid"
  -e HOST_HOSTNAME="Tower"
  -e HOST_CONTAINERNAME="Mailrise"
  -l net.unraid.docker.managed=dockerman
  -v '/mnt/user/appdata/mailrise/':'/etc':'rw' 'yoryan/mailrise:stable'

The error is inside the container. It stops running right after starting it.
I plan to try docker-compose later today when I get some time.

I'm by no means a Docker expert, but I do see you're mounting a folder to
/etc inside the container; mailrise works for me with
/path/to/mailrise.conf:/etc/mailrise.conf
On Sat, Jan 14, 2023 at 1:22 PM photomatix18 @.***>
wrote:

I've only tried with docker run so far. unRAID uses a form of templating
for docker run so there are certain envs it adds. I haven't mapped any
ports yet, but of course, those will be needed once I get the container to
run.
docker run
-d
--name='Mailrise'
--net='bridge'
-e TZ="America/Chicago"
-e HOST_OS="Unraid"
-e HOST_HOSTNAME="Tower"
-e HOST_CONTAINERNAME="Mailrise"
-l net.unraid.docker.managed=dockerman
-v '/mnt/user/appdata/mailrise/':'/etc':'rw' 'yoryan/mailrise:stable'
The error is inside the container. It stops running right after starting
it.
I plan to try docker-compose later today when I get some time.
—
Reply to this email directly, view it on GitHub
https://github.com/YoRyan/mailrise/issues/50#issuecomment-1382921372,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQSC7UVE7XRIBHO22B2SFP3WSMKHTANCNFSM6AAAAAAT3BWTCY
.
You are receiving this because you commented.Message ID:
@.***>

Nor am I! 😬I wondered about that though. I may have to use compose because I'm not aware of a way to map just a file in unRAID's docker UI. I tried but the UI changed it to a directory.

Try just removing it and see if the container starts, at least that'll tell
you if that's the issue. I have a feeling it is because /etc contains a lot
of stuff.
On Sat, Jan 14, 2023 at 2:29 PM photomatix18 @.***>
wrote:

Nor am I! 😬I wondered about that though. I may have to use compose
because I'm not aware of a way to map just a file in unRAID's docker UI. I
tried but the UI changed it to a directory.
—
Reply to this email directly, view it on GitHub
https://github.com/YoRyan/mailrise/issues/50#issuecomment-1382949482,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQSC7UR5IJD2QRC3IIHMFU3WSMSDPANCNFSM6AAAAAAT3BWTCY
.
You are receiving this because you commented.Message ID:
@.***>

I tried that yesterday and got
mailrise: error: argument CONFIG: can't open '/etc/mailrise.conf': [Errno 2] No such file or directory: '/etc/mailrise.conf'

But you might be on to something because that error looks like it's coming from the mailrise service, which means it's running.
@YoRyan would it be possible for the dockerfile to direct the config file to a dedicated directory (/config rather than /etc) so the whole directory can be passed through? Rather than passing just a file through. This seems to be the norm for most containers that I use. I can't recall using any that want individual files passed from the host although I'm sure there are others.

With compose I'm getting
CRITICAL:mailrise.skeleton:Error loading configuration file: 'configs' node not a mapping
This is pretty bizarre. I'm definitely not as familiar with compose so I very well could have missed something simple. Here's my compose file:
version: '2.1'

services:
  mailrise:
    container_name: mailrise
#    restart: unless-stopped
    image: yoryan/mailrise:stable
    ports:
      - 8025:8025
    environment:
      - PUID=1000
      - PGID=1000
    volumes:
      - /mnt/user/appdata/mailrise/mailrise.conf:/etc/mailrise.conf

Your docker compose yaml looks right, can you post your mailrise.conf?
(redact what you need to)
On Sat, Jan 14, 2023 at 3:52 PM photomatix18 @.***>
wrote:

With compose I'm getting
CRITICAL:mailrise.skeleton:Error loading configuration file: 'configs'
node not a mapping
This is pretty bizarre. I'm definitely not as familiar with compose so I
very well could have missed something simple. Here's my compose file:
version: '2.1'
services:
mailrise:
container_name: mailrise
restart: unless-stopped
image: yoryan/mailrise:stable
ports:
  - 8025:8025
environment:
  - PUID=1000
  - PGID=1000
volumes:
  - /mnt/user/appdata/mailrise/mailrise.conf:/etc/mailrise.conf

—
Reply to this email directly, view it on GitHub
https://github.com/YoRyan/mailrise/issues/50#issuecomment-1382967028,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQSC7UQC7WGPWLXZU4CKNNDWSM32LANCNFSM6AAAAAAT3BWTCY
.
You are receiving this because you commented.Message ID:
@.***>

I just threw in a sample config. I haven't really wrapped my head around how the config works.
#
# Minimal Configuration Example
#

# Define your URLs
urls:
  # Either on-line each entry like this:
  - json://localhost
  - xml://localhost

  # Or add a colon to the end of the URL where you can optionally provide
  # over-ride entries.  One of the most likely entry to be used here
  # is the tag entry.  This gets extended to the global tag (if defined)
  # above
  - windows://:
   # 'tag' is a special keyword that allows you to associate tags with your
   # services:
    - tag: desktop

You suppose that's the issue?

You need to start the mailrise.conf with configs:, like below
configs:

  pve:
    urls:
      - discord://webhook_stuff

smtp:
  auth:
    basic:
      username: password

I think you also need a name above urls: in your config.
In my config, if I send an email to pve.mailrise.xyz, it will then forward it to my discord webhook and ultimately my discord server. If I wanted to send it to another address like other_server.mailrise.xyz, I'd have to configure another section along with whatever service I wanted to send it to.
  other_server:
    urls:
      - discord://other_webhook_stuff

@YoRyan would it be possible for the dockerfile to direct the config file to a dedicated directory (/config rather than /etc) so the whole directory can be passed through? Rather than passing just a file through. This seems to be the norm for most containers that I use. I can't recall using any that want individual files passed from the host although I'm sure there are others.

I did not know it was impossible to pass through a file in Unraid. To change the configuration file path, you need only add a custom argument to the container (the Dockerfile default is /etc/mailrise.conf). With the CLI, any arguments at the end of docker run after the yoryan/mailrise argument are passed to the container. Not sure how this is done in Unraid's UI?
docker run
  -d
  --name='Mailrise'
  -v '/mnt/user/appdata/mailrise/':'/etc/mailrise':'ro' 'yoryan/mailrise:stable'
  /etc/mailrise/myconfig.yml

Another user encountered your error, also on Unraid, probably because of this very limitation (#15). This may be worth adding to the Readme.

Poking around Unraid videos (I don't see it anywhere in their documentation) it looks like you could just add '/etc/mailrise/myconfig.yml' to the Post Arguments field in Unraid's advanced view.

Those both seem like good solutions. I'll give both a try sometime this week and let you know!

That did it!

docker run
  -d
  --name='mailrise'
  --net='bridge'
  -e TZ="America/Chicago"
  -e HOST_OS="Unraid"
  -e HOST_HOSTNAME="Tower"
  -e HOST_CONTAINERNAME="mailrise"
  -l net.unraid.docker.managed=dockerman
  -p '8025:8025/tcp'
  -v '/mnt/user/appdata/mailrise/':'/etc/mailrise':'rw' 'yoryan/mailrise:stable' /etc/mailrise/config.yml

Awesome! Glad it worked out for you.


weak docs and tutorials and custom sound !

Desktop Notifier version:3.4.0
Python version:3.10
Operating System:Linux Debian distros , Arch , MS Win 8.1 & 10 & 11,  Mac OS 10 & 12

Description
Hi, thanks for your hard workings.
in my office we use this library to create 2 necessary softwares one: to alert us  extracting  price list has done and the other: urgent sell has been issued.
for our developers, lack of docs and tutorials are pain. To be honest ,apologize my French!, a saw in our asses! not gonna lie!!
and for for sounds:
with this in mind this library play a vital role in my department having a same sound for 2 different software is not good.better to be able to use custome sounds
p.s. : all OS above are using in my office!

Hi @matin1099, glad to hear that you find this project useful.
Regarding the docs, which parts in particular do you find confusing or lacking? The docs for the public API, or rather platform-specific support of features? If you find particular parts could use some improvement, any contributions are more then welcome!
Regarding supporting custom sounds, I'll need to look into platform support first and see if I can find a reasonable abstraction that works for all platforms.

Hi again.
we have 5 python developers:1 senior ,2 junior ,2 intern.
The only person find how this project works was the senior! after 12-13 hour of reading codes!!
to be honest, this project is unique and cool; I'm  sure  a minimal docs will be great!
unfortunately our we can't help contribute in this project as DTWDG. but an individual , I will introduce this proj to someones with open times and interests!
with loves and peace for everyone

custom sounds are now available with the release of version 5.0. @samschott maybe it's time to close this issue?

Indeed, thanks for flagging!
Feel free to open a separate issue for specific documentation improvements you'd like to see.


new educational institution

Official name of the applied university: ICT Academie Apeldoorn.
We offer a four-year, accredited Bachelor of Science degree in Computer Science.
Official website URL: https://www.ictaa.nl
Long-term (>1 year) IT related course is offered by the university: https://www.ictaa.nl/hbo-software-engineer
Domain which you are submitting as an official email domain for the enrolled student: https://www.ictaa.nl/3rdpartysoftware

@ICT-Academie-Apeldoorn First stage of verification passed, official domain for ICT Academie Apeldoorn is ictaa.nl as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

I have removed the file ictaa/student.txt.
Could you review the ictaa.txt file again please?
Thank you.


Feature/email form send

use Hash router (res: react-router-dom)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ed1fbe6) 100.00% compared to head (c03936c) 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@              Coverage Diff              @@
##           development       #45   +/-   ##
=============================================
  Coverage       100.00%   100.00%
=============================================
  Files                2         2
  Lines                5         5
=============================================
  Hits                 5         5

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.