Refactoring ocean temperature - plus other new ocean diagnostics and preprocessors.

Duplicating the pull request #436  https://github.com/ESMValGroup/ESMValTool/pull/436#issuecomment-398441199
due to the pul request coming from a personal fork.
After several excellent contributions from various members of the community, an initial draft of the ocean diagnostic section is now available for review.
This pull request contains the following new features:

A fix for latitudes values greater than 90 degrees in HadGEM2-ES and HadGEM2-CC, described in issue #378
A new folder called ocean in the esmvaltool/diag_scripts. This folder contains four new diagnostics and one toolkit. The four new diagnostics are:

diagnostic_timeseries.py: Produces a simple time series plot
diagnostic_maps.py: Produces a 2D map of the region
diagnostic_transects.py: Produces an image of the transects
diagnostic_profiles.py: Produces a profile plot

The toolkit is diagnostic_tools.py, and it contains several useful functions used by the four ocean diagnostics.

Two new name list files are also included in this pull request:

esmvaltool/namelists/namelist_OceanBGC.yml:
esmvaltool/namelists/namelist_OceanPhysics.yml
These two name lists are not meant to be fully functioning nameliests to evaluate all facets of ocean models.  Instead, they should serve as an example of how to produce various kinds of figure of the ocean. They also make use of several preprocessors. Note that these diagnostics are not a direct replacement for the previous marine metrics (namelist_GlobalOcean.yml), described in issue #350.

In addition, several new preprocessors were needed in order to evaluate of CMIP5 models marine components.

average_volume
time_average
extract_volume
extract_trajectory
extract_slice
depth_integration

Finally, two new vertical interpolation schemes were added:

nearest_extrap
linear_extrap

Please merge the version2_development branch into this branch to fix the tests.

Please use the PEP8 conventions for naming functions and variables:
https://www.python.org/dev/peps/pep-0008/#prescriptive-naming-conventions
Specifically (see the Codacy issues, click the Details link above behind Codacy)

Function names should be lowercase, with words separated by underscores as necessary to improve readability.
Variable names follow the same convention as function name

Please use the numpy docstring style, at the very least for the functions that are part of the preprocessor:
http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html

It would also be nice if you could add unit tests for the new preprocessor functions.

@jvegasbsc @ledm

I have one general comment about the pull request: your diagnostics can be useful for more than just the ocean. After all, plotting a map, a section or a timeseries is done for variables in all the modelling realms. Also, I don see any assumption in the implementation that could make them fail for atmos variables, for example
It does not really make sense to hide them inside an 'ocean' folder. I think they belong to a 'general' or 'basic' folder so more people can benefit from your work

Generally useful stuff should go into esmvaltool/diag_scripts/shared, if it is plotting related in the subdirectory plot.

@jvegasbsc @bouweandela

I have one general comment about the pull request: your diagnostics can be useful for more than just the ocean. After all, plotting a map, a section or a timeseries is done for variables in all the modelling realms. Also, I don see any assumption in the implementation that could make them fail for atmos variables, for example
It does not really make sense to hide them inside an 'ocean' folder. I think they belong to a 'general' or 'basic' folder so more people can benefit from your work

Generally useful stuff should go into esmvaltool/diag_scripts/shared, if it is plotting related in the subdirectory plot.

I'm in favour of putting these tools as shared/general tools, however I feel that the current range of ocean diagnostics are still fairly basic in their capability. In the future, it is entirely possible that there will be much more ocean specific parts needed, and it would be good to keep that option available.

sorry - I meant only the reversal of changes from _time_area.py

I've split the time_area.py preprocessor into three separate files, and updated preprocessor/init.py accordingly.
The parts that I wrote and added seem to have passed their codacy tests, but a recent merge with version2_development has caused the CircleCI tests to fail the  test_no_namelist and the test_avoid_casing_collisions tests.

Yes, that test is there to warn you that you still have the namelists folder with namelists in it that you need to rename to recipe:
https://github.com/ESMValGroup/ESMValTool/tree/REFACTORING_OceanTemperature/esmvaltool/namelists

Ok. I've just asked because I've seen there are some changes in _multimodel.py to account for different dimensionality in input and I was thinking this was due to some area/time operations performed in advance.

hey @ledm great work in getting all the codacy/lint-changes done and cheers for devving this whole thing! I will take a careful look at the PR myself today but I don't think I can come up with anything new that @mattiarighi @bouweandela or @jvegasbsc haven't suggested yet. In the meantime, could you pls point me to a receipe that I can run after I have installed the branch just to test the functionality? Cheers! ...and as a general remark, let's try with smaller and more incremental PR's next time, this one's a monster, but a good monster :grin:

@mattiarighi I think that the multi-model operations always occur towards the end of the preproccessor, after any time or area operations. (rightly so, too)
The changes were needed in _multimodel.py because a zero-dimensional scenario (time dimensional 1D arrays) had not yet been considered.
Thanks @valeriupredoi, The recipe_OceanPhysics.yml recipe should be a good starting point. Let me know if there's anything else I can do.

Well, not really: the order of operations was chosen to minimize the loss of information through the different steps. Performing multi-model stats after the area/time operations will return something different, and in general less accurate, than applying that before. The same goes for regridding and masking of fill values.
In the end this is of course a choice of the diagnostic developer, but since we are supporting only one order of operation (so far), I believe we should choose it in a way that gives the best accuracy (even if this is not the most efficient in terms of cpu/memory).
This approach has also the advantage that, being more general, allows to apply the same preprocessor set to multiple diagnostics.

@ledm cheers, I grabbed the recipe and installed the branch: two mentions -- one not so important:
line # Takahashi, T. et al., Mar. Chem., 164, 95–125, in the namelist contains a special character at position 1040 that will make yaml crash, have a look
and another, rather important: your plotting will NOT work with iris 1.13 because of switches within nc_time_axis to cf_units (this a nested problem starting from iris.quickplot and I have the same issue with my autoassess plotting; the solution is either to not use quickplot or to return an empty plot if iris 1.13 is used). See stack:
Traceback (most recent call last):
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/ESMValTool-2.0a0-py3.6.egg/esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py", line 267, in <module>
    main(config)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/ESMValTool-2.0a0-py3.6.egg/esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py", line 248, in main
    metadatas,
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/ESMValTool-2.0a0-py3.6.egg/esmvaltool/diag_scripts/ocean/diagnostic_timeseries.py", line 188, in multi_model_time_series
    lw=2.,
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/iris/quickplot.py", line 268, in plot
    result = iplt.plot(*args, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/iris/plot.py", line 1113, in plot
    return _draw_1d_from_points('plot', _plot_args, *args, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/iris/plot.py", line 607, in _draw_1d_from_points
    result = draw_method(u, v, *args, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/pyplot.py", line 3358, in plot
    ret = ax.plot(*args, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/__init__.py", line 1855, in inner
    return func(ax, *args, **kwargs)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 1528, in plot
    self.add_line(line)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 1932, in add_line
    self._update_line_limits(line)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 1954, in _update_line_limits
    path = line.get_path()
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/lines.py", line 951, in get_path
    self.recache()
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/lines.py", line 651, in recache
    xconv = self.convert_xunits(self._xorig)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/artist.py", line 191, in convert_xunits
    return ax.xaxis.convert_units(x)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/matplotlib/axis.py", line 1526, in convert_units
    ret = self.converter.convert(x, self.units, self)
  File "/home/users/valeriu/anaconda3/envs/esmvaltool_v2_dev/lib/python3.6/site-packages/nc_time_axis/__init__.py", line 277, in convert
    raise ValueError('The datetime attribute of the CalendarDateTime '
ValueError: The datetime attribute of the CalendarDateTime object must be of type `cftime.datetime`.

@valeriupredoi, I've been doing all my development with iris.__version__ = 1.13.0, matplotlib.__version__ = 2.2.2 and haven't seen this error before.
I have replaced the Em dash ( –) in my recipes, but I've found that there are several other ones around the repository. I believe that it shouldn't matter in a yaml file if it's hidden in a comment.

@ledm that's because you are working with obsolete versions of iris=1.13 with nc_time_axis, this problem has started to creep up only in the last two-three weeks, after you've started working on this PR; if you uninstall iris and nc_time_axis, remove the iris=1.13* and nc_time_axis* files from your anaconda/pkgs and download+reinstall both packages via conda you will see it happening. The preprocessor bit of that receipe works fine so I am happy on that side but you need to fix this issue before the PR gets merged, unfortunately

to give you the repair solution, in /iris/plot.py line 425 should look like:
            r = [nc_time_axis.CalendarDateTime(
                 cftime.datetime(*date), coord.units.calendar)
                 for date in dates]

instead in iris 1.13, that same line looks like:
            r = [nc_time_axis.CalendarDateTime(
                 netcdftime.datetime(*date), coord.units.calendar)
                 for date in dates]

because nc_time_axis used to use netcdftime but not no more. Since we can not change the iris source cde, I suggest a workaround the plotting call -- either drop the quickplot, replace it with a matplotlib equivalent or produce an empty plot when using iris=<1.13. Note that this is fully solved by the dependency tree only in iris=2.1, so even in 2.0 you will have the same problemo

Hi @valeriupredoi, it looks like the iris cftime issue is also linked with the time preproccessor, time_slice(), now in preprocessor/_time_pp.py, formally in preprocessor/_time_area.py, and also the  preprocessor/_multimodel.py function _datetime_to_int_days.

@ledm what makes you think it affects it? it doesn't use netcdftime that now has changed to cftime in iris=2.1

Hi,
I think that there are some subtle differences between netcdftime and cftime. The time contraint in time_slice fails with the following error message:
2018-06-26 09:51:49,886 UTC [9911] ERROR   Program terminated abnormally, see stack trace below for more information
Traceback (most recent call last):
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_main.py", line 204, in run
    conf = main(args)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_main.py", line 133, in main
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_main.py", line 183, in process_recipe
    recipe.run()
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_recipe.py", line 922, in run
    self.tasks, max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 473, in run_tasks
    _run_tasks_sequential(tasks)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 484, in _run_tasks_sequential
    task.run()
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 184, in run
    input_files.extend(task.run())
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 185, in run
    self.output_files = self._run(input_files)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/__init__.py", line 268, in _run
    input_files, self.settings, self.order, debug=self.debug)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/__init__.py", line 193, in preprocess_multi_model
    debug)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/__init__.py", line 230, in preprocess
    result.append(function(item, **args))
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/_time_pp.py", line 44, in time_slice
    cube_slice = mycube.extract(my_constraint)
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/iris/cube.py", line 2223, in extract
    return constraint.extract(self)
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/iris/_constraints.py", line 151, in extract
    resultant_CIM = self._CIM_extract(cube)
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/iris/_constraints.py", line 175, in _CIM_extract
    resultant_CIM = resultant_CIM & coord_constraint.extract(cube)
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/iris/_constraints.py", line 290, in extract
    r = np.array([call_func(cell) for cell in coord.cells()])
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/iris/_constraints.py", line 290, in <listcomp>
    r = np.array([call_func(cell) for cell in coord.cells()])
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/_time_pp.py", line 42, in <lambda>
    my_constraint = iris.Constraint(time=lambda t: (time1 < t.point < time2))
  File "cftime/_cftime.pyx", line 1606, in cftime._cftime.datetime.__richcmp__
TypeError: cannot compare cftime._cftime.DatetimeNoLeap(2001, 1, 16, 12, 0, 0, 0, 2, 16) and 18615.0

This is resolved by using the code reviously hidden behind a TODO statement:
        my_constraint = iris.Constraint(time=lambda t: (
        time1 < time_units.date2num(t.point) < time2))

Similarly, multimodel fails with:
Traceback (most recent call last):
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_main.py", line 204, in run
    conf = main(args)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_main.py", line 133, in main
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_main.py", line 183, in process_recipe
    recipe.run()
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_recipe.py", line 922, in run
    self.tasks, max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 473, in run_tasks
    _run_tasks_sequential(tasks)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 484, in _run_tasks_sequential
    task.run()
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 184, in run
    input_files.extend(task.run())
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/_task.py", line 185, in run
    self.output_files = self._run(input_files)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/__init__.py", line 268, in _run
    input_files, self.settings, self.order, debug=self.debug)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/__init__.py", line 201, in preprocess_multi_model
    debug)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/__init__.py", line 224, in preprocess
    result = [function(items, **args)]
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/_multimodel.py", line 334, in multi_model_statistics
    interval = _get_overlap(selection)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/_multimodel.py", line 200, in _get_overlap
    span = _datetime_to_int_days(cube)
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/_multimodel.py", line 168, in _datetime_to_int_days
    for cell in cube.coord('time').cells()]
  File "/users/modellers/ledm/workspace/ESMValToolTest/ESMValTool/esmvaltool/preprocessor/_multimodel.py", line 168, in <listcomp>
    for cell in cube.coord('time').cells()]
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/cf_units/__init__.py", line 1989, in num2date
    return _num2date_to_nearest_second(time_value, cdf_utime)
  File "/users/modellers/ledm/workspace/ESMValToolTest/miniconda/minicondaLee/envs/esmvaltool/lib/python3.6/site-packages/cf_units/__init__.py", line 601, in _num2date_to_nearest_second
    time_values % 1. == 0.5)
TypeError: unsupported operand type(s) for %: 'cftime._cftime.DatetimeNoLeap' and 'float'

@ledm please don't change these lines -- the code labeled with TODO works only for iris => 2.0 and this is exactly why that snippet stays there and is commented out, so we use it when we will be switching from 1.13. With iris 1.13 it works as expected and there are no errors

This feels a bit like a catch 22, @valeriupredoi. You told me to upgrade iris, and it broke several parts of the preprocessor and the setup.py test code, but now you're telling me I can't fix them.
You also instructed me to add a version dependent code to my plotting tools, which I've done, but I can see whether it works, or whether I can share it as I'm not able to run the setup.py test or get past the preprocessor!
Also, apologies to everyone for adding so many code changes at once!

hahahah @ledm sorry for confusing you, dude -- but what I said was rather clear: do not update the iris since we are still working with 1.13 in this release; change the just the plotting code that uses iris.quickplot (since that's the only bit affected by the transfer from netcdf to cftime) in such a way that doean't break your diagnostic - how you do that is up to you entirely, what I did for the autoassess diag that uses quickplot as well: I put a check on the iris version - if = 2.1 execute the plotting, if not, return an empty plot

hey @ledm man, cheers for making the needed changes, now the namelist is running fine! A bit on profiling, I have had the namelist crash a couple times with MemoryError and have noticed that certain diags may take up to 18G of memory. The functions that affected this are:
_volume_pp:
  depth_integration(): iris.analysis.SUM
  volume_average(): iris.analysis.MEAN

If I were you I'd be careful with such things because a user may be running the diag on a node that hasn't much memory (note that I ran with max_parallel_tasks:1 in config). Where possible, don't rely on the iris.analysis since that one is not playing well in terms of resources with large numerical arrays. Anyhoo, the thing is still running now (the third attempt) and albeit memory still went to 14G a few times, Jasmin sci1 seems to cope with it :grin:

Thanks a lot for all the help and advice, @valeriupredoi, @jvegasbsc, @bouweandela, @mattiarighi, and @koldunovn. As far as I can tell, I've addressed all the comments that you've given so far, @valeriupredoi has tested the diagnostic scripts. From here, it looks like the merge pull request is ready to go.
There are still quality review failures, but there's little that I can do about the remaining Codacy issues. The remaining issues are things like:

codacy wants the allvars class n the _fixes files  to be capitalised,
codacy not being able to import various modules (like iris),
Actual real problems with functions that I did not author, like trajectory_cube in preprocessor/_area_pp.py (formely in preprocessor/_time_area.py).

The problems with Memory that @valeriupredoi mentioned could be addressed by iterating these functions over the time or plev dimension, but neither of us think that this is an essential task needed before the merge can be completed.

Yup, the old f(g(x)) problem - very good point that I missed @mattiarighi
Dr Valeriu Predoi.

In esmvaltool/preprocessor/init.py
https://github.com/ESMValGroup/ESMValTool/pull/438#discussion_r199066818
:

 'average_region',

'average_volume',
'time_average',

I agree that resources could be an issue, but it's conceptually not
correct to apply time/area statistics before the multi-model. This is
because, in general, the multi-model of some stats is not equal to the
stats of the multi-model.
Since the multi-model-mean/median is treated like another dataset, it's
important that the preprocessing is applied consistently to all of them.
That's why I'm insisting on keeping the order of operations as originally
planned. I understand that some diagnostic developers (like @ledm
https://github.com/ledm in this PR) would like to customize this order,
but as I said this is not yet support and until then we need to stay with
the most general approach.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/438#discussion_r199066818,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo9wnQTRoXPgPVfQCnASu1qzaWUlLks5uBcz6gaJpZM4Utv3M
.

I'm convinved too. I've moved the time average after the multimodel
preprocessor and pushed the change.
Please let me know if you spot anything else to do,
Lee
Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine
We have updated our Privacy Notice which is available at http://www.pml.ac.uk/System-pages/Privacy_Notice
Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth  PL1 3DH, UK.
This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.

Thanks @ledm.
I would also move the other grid-point operations (depth_integration, average_region, average_volume) and seasonal_mean after multi-model, for the same reasons.
The extract operations can stay before, they should be not critical in this sense (extracting before or after multi-model does not affect the results, but it saves memory)

@mattiarighi I think so, on my way to work and will double check when I get
there
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
On Fri, 29 Jun 2018, 09:26 Mattia Righi<EMAIL_ADDRESS>wrote:

Sorry, I didn't notice that season_mean was already there.
@valeriupredoi https://github.com/valeriupredoi where does it come
from? Was it part of your very first implementation of time-area?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/pull/438#issuecomment-401285818,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCowFk3qpbdhNyBnkBHbugohgMQ5vCks5uBeSygaJpZM4Utv3M
.

@mattiarighi sorry, being a big mouth I just ran into Grenville and had to give him a very detailed updated on ESMValTool :grin: yes, the seasonal_mean() from _time_area.py is a function I put in about a year ago, but it still seems to do the trick OK, @ledm are you using it or you've implemented your own seasons?

I haven't looked at the seasonal mean function at all. I hate to speak for everyone, but ocean scientists are typically are happy with monthly or annual means.
I should also point out that the time_average is an extremely simple function which collapses the cube along the time dimension. Perhaps it would be worth weighting the average according to the length of each time step, in the case of uneven time dimensions, but it doesn't do that yet.

yea no probs @ledm - I am autoassess-assing now and the land-atmos diags need the seasons; anyway, the current multimodel stats function is not applying weights (yet) so I reckon you don't need weights yourself just yet. Let's not postpone more this PR just now, it's already a lot of stuff that needs to get in anyway :smiling_imp:

See e.g. the file tests/integration/preprocessor/_regrid/test_regrid.py for examples of how to implement tests.

Hi @bouweandela, thanks for the extensive review comments, they are very helpful. I've address most of these points now and I'll be submitting these changes in the near future. I have yet to start adding tests, but I'll see what I can do on that front.
One issue I'm seeing is that I'm finding that the reordering the preprocessor operations in __init__.py  breaks several aspects of the ocean analysis (commit info).
The region_average and volume_average reduce a 2 or 3 dimensional array to 1D array (with dimension: time), which works fine if run before the multi-model mean. If we try to run these preproccessors afterwards, then the multi-model mean will try to take the mean of several models which all have different model grids shapes, and it fails.
This means that we now need to regrid everything in order to perform any kind of multi-model comparison of area or volume averages. As we all know, regridding can introduce uncertainties and increased running time, and it would be better to keep the average_volume and average_area preprocessors based on their naive data grids (before the multimodel preprocessor). Sorry to open this up again @mattiarighi and @valeriupredoi.

Fine with me to introduce customizable order of operation now, if it is not too much work.
But I think this would require to also introduce some checks to make sure the different modules of the preprocessor are interfacing correctly. Or, as @bouweandela suggests, we leave this to the user at his/her own risk.

I've created an issue for this feature request here: #495.

With regards to the ordering, I've also found that putting the area_average, volume_average and time_average preprocessor after the multi-model preprocessor means that the multi-model dataset hasn't had the same manipulations as the other models. Ie, in my case, multi-model data still contains all time points, where as the rest of the models are a time average. Similarly, the volume_average should have time-only dimensional, where as the multi-model contains latitude, longitude, time and depth dimensions.
This means that without re-ordering the preprocessors, very few of the ocean temperature examples will work (with multimodel data).

Created issue #497 from the above comment.

Sorry, I'm working on some early tests and accidentally pushed a draft commit, but it's far from complete.
Also, I'm adding unit tests instead of integration tests as these are more suited to test individual preprocessors.

Hi @bouweandela,
I've added several unit tests, and luckily these highlighted some conceptual issues with the preprocessor code which have now been fixed. Thanks for asking me to do them - it has resulted in significant improvements!
Bearing in mind issue #495 and #497 (which currently mean that the standard preprocessing order does not work with the ocean temperature yml recipe), I think that this code is now ready for the PR.
However, there may well be some problems as this PR might conflict with @jvegasbsc's PR #500, as we have split the _time_area.py into two separate files.
(Also, I won't be able to work on ESMValTool until the end of next week, so fingers crossed that this is sufficient to get it over the line. )
Lee

Hi @ledm,
Thanks for adding the tests, looks really good!
Could you also add numpy style docstrings to the preprocessor functions that you added?
I noticed that you cleaned up one of the two namelists, but not the other one (recipe_OceanBGC.yml). Does that one work yet?
Could you also add a documentation section to your namelists? See .e.g recipe_perfmetrics_CMIP5.yml for an example. You can do this by moving the stuff that is in the comments at the top of the namelist into the proper yaml tags (add any tags used to esmvaltool/config-references.yml as well). This does not do anything yet, but when I get round to implementing provenance next month it will be used for that, see #240 for more information.

Hi, I've added longer documentation for the preprocessors (though I must say that very few of the other preprocessors had this level of detail.)
I've also added the documentation section to the two recipes. As provenance is no yet implemented, I've just added a rough outline, but it should be sufficient for now. I'm happy to go into more depth when that part of the code is implemented. Either way, I intend to continue to add new part to these recipes and other recipes.

Thank you for adding the documentation. You are right that the documentation for many of the other preprocessor functions also needs to be updated, we have an open issue on that #360. In order not the make the issue worse, I asked you to already provide it now. Could you check that your formatting is according to the numpy docstring style? At the moment it looks e.g. like this:
Arguments
---------
    cube: input cube.

while it should be:
Arguments
---------
cube : iris.cube.Cube
    Input cube.

As shown in the examples here: http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html

Thanks for fixing the documentation.
Regarding the main functions in your scripts: instead of reading in the variable metadata yourself, you can simply use the metadata provided in cfg['input_data']. If you would like to group and/or sort it, you can now (since PR #508) use the function group_metadata, see for a simple example esmvaltool/diag_scripts/examples/diagnostic.py.

hey fellas @bouweandela @mattiarighi @ledm cheers for all your inputs and great work on this one -- I strongly suggest to approve this PR before suggesting any more changes -- it is already blowing up to humongous size and I feel that if suggestions keep coming this will be a neverending story, since the code base also keeps changing. Lee's effort is needed in since this will be a very useful set of diagnostics that a lot of scientists are already asking (at lest) me about. Lee can always branch out after this gets in and make the other changes needed to align the diags with the new implementations. I had a look through the Codacy issues and they are fine, nothing more than what I've had when merging the first Autoassess PR. In the future, as it was the case with the multimodel PR, I strongly suggest we limit the PR size and plug things in incrementally.

yo guys, any opinions on merging this now and letting Lee work further on it on the branch? :grin:

I've made a little push with minor changes, and I've commented out all the diagnostics that don't work without the custom preprocessor order. I

Hi, I've just merged the latest version2_development branch into this pull. The custom order ocean temperature is now fully functional and tested. I think that the pull request is ready to complete.
Also note that the failures in Codacy are beyond the scope of this PR.

Can we please complete this pull request @valeriupredoi @bouweandela. Everything worked yesterday and was ready to go. Ten this morning, there was two conflicting files, so I just pulled in the latest changes and now I have 22 test errors.
The underlying development branch continues to move on and I can't afford the time to merge, debug, test, run and resubmit this.

Hi Lee,
I'm fine with merging. The failing tests are probably because you made a mistake with the merge. Javier recently added some time operations to the preprocessor and also tests for those, that's what's causing the conflict (and probably your test failures). I performed the merge for you and uploaded it in the REFACTORING_OceanTemperature2 branch, because I didn't want to mess with your branch. Maybe you can check if that looks good/works and then we can merge that one? I think someone independent from you should also test that your recipes actually work. @mattiarighi, did you already have a look?

Thanks, the REFACTORING_OceanTemperature2 seems to work fine for me. I know that @valeriupredoi has been able to run this before.

Closing this PR and recreated it as #526.


[SYSTEMML-485] Rename unusual file extensions

Rename unusual file extensions (.Rt -> .R, .dmlt -> .dml, .data -> .csv, etc.) to help standardize file naming in project.

Could you please delete the L2SVM files? The dmlt and Rt extensions are generally used to indicate temporary files - for algorithm testcases, we do not generate them, so this must be a relict from a long time ago.
That reminds me, down the road, we should remove the entire "old codepath" from our testsuite. There are still various tests that use something like $$op1$$ + $$op2$$, replace these with input arguments, write the temporary dmlt script, and finally execute the script. This stems from a time when we did not have any input arguments (>5 years). Maybe we find a good way to automatically replace these tests.

Sounds good. I deleted the L2SVMTest.Rt and L2SVMTest.dmlt and also added a needed update to WriteCSVTest.dml.

LGTM - thanks @deroneriksson


More flexibility in querying Java toolchains

I'd like to have more flexibility in querying toolchains, similar to one that javaToolchains task has. To be able to query all detected toolchains and select one that matches some predicate (not necessarily by exact version and vendor) or to provision a JDK by some fallback toolchain spec if none of detected JDKs matched.
Context
My use case is following: I'm building most modules in my project with JDK 8, but for some compilation tasks, namely those that build module-info.java, I need to use JDK 9 or higher (specifying -release 9 option to produce compatible bytecode).
So for that, I want to select any JDK that has the version greater than or equal to 9. If no such JDKs are detected, I want to ask Gradle to provision me JDK version 11 (because it's LTS and there are more options for this versions than for 9). Of course, Gradle can deny that request if auto-provisioning is not enabled.
I also may want to make a part of build optional if no required JDK is found: for example, avoid running tests on that JDK.
For that I need to query a toolchain without triggering its provisioning when it's not found.

Still relevant


Doc for generation with multiple blocks of Api

what's new
Mention the new  example in Section 3.Features.
Add a new doc for detailed explanation on why and how to generate Api within multiple blocks(files).
related pr
https://github.com/fzyzcjy/flutter_rust_bridge/pull/516
https://github.com/fzyzcjy/flutter_rust_bridge/pull/527
related issue
https://github.com/fzyzcjy/flutter_rust_bridge/issues/511

@all-contributors please add @dbsxdbsx for doc

Good work! LGTM, except some phrasing, which I will modify after merging.

Btw, https://app.grammarly.com is quite helpful to check and improve grammar automatically.

/cc @Desdaemon who wrote that wonderful tutorial part

what is explicit and implicit api?

what is explicit and implicit api?

If API is defined by users, it is explicit, others like new_uint8_list are implicit API. Frb would throw panic with conflict on explicit API, but no warning but automatically add suffix starting from 0 to avoid implicit API conflict.

Ah, those are in the code details and I also do not remember very clearly currently.
I guess it is by simple string manipulation without fancy data structures.
Feel free to refactor them!

Great writeup @dbsxdbsx. I'm thinking whether it would be fruitful to pursue a crate-wide, explicit declaration process a la wasm_bindgen, which could solve a lot of the problems described here.

Great writeup @dbsxdbsx. I'm thinking whether it would be fruitful to pursue a crate-wide, explicit declaration process a la wasm_bindgen, which could solve a lot of the problems described here.

I have no idea what "a crate-wide, explicit declaration process" means here. But I guess it is something related to support for the web platform, right?

I am not sure, I guess it may be another toml configuration file specifying things instead of using command-line?


fatal: unable to access 'https://go.googlesource.com/go/': Unknown SSL protocol error in connection to go.googlesource.com:1080

OS X is 10.8.5
shell is zsh
I enter
gvm install go

ERROR: Couldn't download Go source. Check the logs /Users/MasterMac/.gvm/logs/go-download.log

vim  /Users/MasterMac/.gvm/logs/go-download.log

fatal: unable to access 'https://go.googlesource.com/go/': Unknown SSL protocol error in connection to go.googlesource.com:1080

me too

me too

me too

替换了源，在国内可以正常使用， 地址 https://github.com/mosaic101/gvm


Asks the input method to use the vertical candidate window when there is a long candidate.

Recently we added the input macro feature. It may cause longer candidates appearing in the candidate window, and it could be hard to use when the window stays horizontal.
The PR lets the candidate to be always vertical when there is a candidate whose length is larger than 8.

Awesome, thank you!


Travis-CI build failed because it automatically routed the builds to Ubuntu Trusty

Travis-ci build failed because it automatically routed the builds to Ubuntu Trusty, which is the Ubuntu distribution used by default now.  However, the builds is supposed to work in Ubuntu Precise which has reached the end of life.
check following
https://blog.travis-ci.com/2017-07-11-trusty-as-default-linux-is-coming
related Issue
https://github.com/angular-ui/bootstrap/pull/6624

Looking into this.  Thanks.

That's not why the build failed.  I'm pretty sure it's this:  https://github.com/jasmine/jasmine/issues/1327

Ok, I've played around with this for an hour and I'm calling it.  I don't have time to debug this.  If you want to, please feel free.  I've tried both precise and trusty as well as the timeout/retry settings in the issue above and pinning jasmine-core at 2.5.2, none of which works.  Maybe there's a magic combination that gets it passing, and maybe there's not.  This project is EOL'd and there was a 9 month span between the last successful build and the first failed one.


Rename blog files to include date in names

/kind cleanup

/lgtm
/approve

Preview:
https://deploy-preview-15386--kubernetes-io-master-staging.netlify.com/blog/2019/04/19/introducing-kube-iptables-tailer/

/approve


Feature: single select and multi select list items

Will there be a multi select widget and table widget?

Compare #99 and https://github.com/andlabs/libui/pull/343


Use bindata to populate operand defaults

Consider using go-bindata to load templates of the components managed by the cluster-image-registry-operator. This is a common practice amongst other OpenShift core operators - e.g. ocm-o.
Not required for 4.1

Which generator do you want to replace with a static file?

If we choose to go down this path (and I'd argue this is lower-priority tech debt), I'd start with the resources that aren't impacted by the operator config, such as RBAC, the internal service, etc.
Downside is this introduces dual paths for implementing the same behavior if say the Deployment needs the generator.


Domain should be optional

When starting a new project, it is common to experiment without a domain first. Standard AWS urls should be sufficient here.

Fixed in version 0.1.2.


cleanup: Use hex constants for i8/i16/i32/i64 limits.

The rest of the code uses mostly hex constants, which are much less error-prone than a complicated number like<PHONE_NUMBER>834775807 (in which I made a mistake, but it's hard to spot).
I've only put UINT64_C around the 64 bit constant, because those macros aren't used anywhere else in the code. This avoids some warnings in overly strict compilers (cstd <IP_ADDRESS> allows large integer constants and puts them into whatever type can represent them, but some compilers warn about it anyway and ask you to put suffixes like ULL).

Should we use INTxx_MAX/MIN, instead?

Should we use INTxx_MAX/MIN, instead?

Nah; these are MessagePack-specific constants, not platform constants; they shouldn't change based on what your machine thinks the biggest int (etc.) is.

I tried really hard to get any compiler on Godbolt to complain about this, but nothing did so I think we're fine here. I'll merge and see if anyone experiences any issues.

Should we use INTxx_MAX/MIN, instead?

Nah; these are MessagePack-specific constants, not platform constants; they shouldn't change based on what your machine thinks the biggest int (etc.) is.

I mean INT32_MAX etc. They have exact definitions and aren't platform-specific.


Various work on nav bars and theater.

Tweaks to search bar in mobile mode
Fixed width of left nav in mobile mode.
Fixed position of bottom pagination links after matrix table.
Tweaked theater to only load medium-sized image if can't display larger.
Moved theater JS out into new file, since it's gotten rather large.
Thumbnail partial passes in image width and height if available, allowing JS to choose and position image very precisely in most cases.
Have theater hide hamburger controls in mobile.

Ray, I added the search pulldown back in (mobile mode).  My reasoning is that the search bar is usually hidden, so it doesn't matter if it takes up multiple lines on mobile devices.  I think people use the pulldown often enough that it's worth ensuring that it's present even in mobile.  If you disagree, please feel free to return to the way it was.
I think the left nav issue is fixed now.  We might want to add more padding to the elements of the left nav.  You had added extra margin to the entire div in mobile mode, but that looks bad to me -- I prefer when the lines and hover background take up the full width of the nav bar.
I think the theater works really well now... and I've tried to test the mathematics very carefully... but I'm worried that it's too complicated now and dangerous.  I'd understand if people want to switch back to the old one, which always loaded the 960-size image, left it centered at the top of the page -- the js is way simpler... just so long as you don't mind that it doesn't center vertically (minor), and that it over-inflates images past 960 on large screens, potentially making users think the photograph itself is crappy quality, when in fact it's just dumb css not knowing to switch to the next higher resolution when the space available to the image grows beyond its full 960x640 resolution (more serious in my opinion).
I'm going to have to stop working on MO (I'm sure Ray is rejoicing! :).  I will try to keep the test server up-to-date if anyone creates pull-requests, at least this weekend.  I might not be able to help at all Monday through Thursday this coming week, though.  Sorry.

This all sounds great.  I don't mean I sound critical of your work in my
responses. They are just short because I'm on an iPhone.
Thanks so much for doing all of this, much improved.
On Saturday, March 7, 2015, Jason Hollinger<EMAIL_ADDRESS>wrote:

Merged #142
https://github.com/MushroomObserver/mushroom-observer/pull/142.
--
Reply to this email directly or view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/pull/142#event-248035204
.

--
Ray Suelzer
Mobile:<PHONE_NUMBER>


feat: add onKeyDown event to allow user to bail

Fixes #17.

Motivation

Acting on my own after reading #17. Might be totally wrong!
Test plan

Verified by adding an onKeyDown event.
// rest of the props
  onKeyDown={e => {
   // override: prevent custom things on Cmd+Enter
   if (e.metaKey && e.key === 'Enter') e.preventDefault();
  }}

GIF shows on pressing just Enter the text goes down but Cmd+Enter it doesn't

Thanks for the PR. Merged in feat: add onKeyDown event to allow user to bail


Function WriteBigEndian needs an offset of 3

WriteBigEndian was wrong.
If I want to reach the last of 4 bytes,
I must add 3.

Cool looks like that was a bug. Have you been able to test this change in any way?

Yes, but it was a short test only.
I tried to manipulate the xing header.
My intention was to fix the length
for mp3-files with variable length.

Yes, but it was a short test only.
I tried to manipulate the xing header.
My intention was to fix the length
for mp3-files with variable length.


PanelBuilders: Typed API for VizPanel creation

This PR introduces an API for building VizPanels using core visualizations easily. It supports code support for core panel types for Options and FieldConfig.
We need to merge https://github.com/grafana/grafana/pull/69678 first before proceeding with this.
TODO:

[ ] Add API methods for remaining core panels
[ ] Add default values handling (provided via @grafana/schema)
[ ] Add documentation

@dprokop how does  .override('numeric', 2) know it's a custom property and not a standard one?

@torkelo because override is typed with TFieldConfig - which represents panel-specific field config (https://github.com/grafana/scenes/blob/panel-builders-api/packages/scenes/src/core/PanelBuilders/FieldConfigOverridesBuilder.ts#L79). It only accepts keys/values from this interface, and this interface describes .custom shape only.


Syntax warning

Not important, typo?
2021-06-12 21:01:44 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.signal_messages is taking over 10 seconds
/config/custom_components/hasl3/sensor.py:370: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if sensorproperty is 'min':
/config/custom_components/hasl3/sensor.py:380: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if sensorproperty is 'time':
/config/custom_components/hasl3/sensor.py:389: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if sensorproperty is 'deviations':
/config/custom_components/hasl3/sensor.py:392: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if sensorproperty is 'updated':
/config/custom_components/hasl3/sensor.py:447: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if self._unit_of_measure is not '':

Perfect find! It is amazing how much I can miss sometimes. Will fix to next version which should be out in a few days.

Resolved in 3.0.0-beta.2


[SPARK-18918] [DOC] Missing  in Configuration page

What changes were proposed in this pull request?
The configuration page looks messy now, as shown in the nightly build:
https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/configuration.html
Starting from the following location:

How was this patch tested?
Attached is the screenshot generated in my local computer after the fix.
Configuration - Spark 2.2.0 Documentation.pdf

cc @rxin @srowen

Test build #70317 has started for PR 16327 at commit 81f6eb7.

Test build #70317 has finished for PR 16327 at commit 81f6eb7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70317/
Test PASSed.

Sure, I will merge this to master/2.1. It renders fine in Chrome for me. I don't think this is nearly a blocker for 2.1.0


[Rust][Dev] Cache packages and/or compilation in docker images

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-9359
To improve the local development experience, it would be nice if the Docker setup for Rust builds could cache packages from run to run.

Comment from Jorge Leitão(jorgecarleitao) @ 2020-07-14T13:22:14.139+0000:
One idea that I often use:

{code}
# use specific version here to freeze layer
FROM rust:1.44.1-buster AS backend-builder

# freeze the version for deterministic builds
RUN rustup update nightly-2020-07-03 && rustup default nightly-2020-07-03

# Compile a dummy program, so that the dependencies are compiled and cached on a layer
# see https://stackoverflow.com/a/58474618/931303 for details
RUN echo "fn main() {}" > dummy.rs
## if cargo.lock or cargo changes, we trigger a build of all dependencies.
COPY backend/Cargo.lock backend/Cargo.toml /
RUN sed -i 's@src/main.rs@dummy.rs@' Cargo.toml
RUN cargo build --release
RUN sed -i 's@dummy.rs@src/main.rs@' Cargo.toml

# compile the actual program
COPY ...

RUN cargo build --release
{code}

Comment from Jorge Leitão(jorgecarleitao) @ 2020-07-19T15:47:36.675+0000:
I believe that this is a duplicate of ARROW-7136

I think this is referring to a build system that no longer exists, or exists in a different repository, so closing


TypeError: list indices must be integers or slices, not str

Hi there, thanks for launching this exciting tool! I am trying to modify the training parameters in the yaml a model_definition file and I am getting an error that the dictionary is a list and not a dictionary.
YAML File
# ludwig yaml test
input_features:
  -
    name: image_path
    type: image
    encoder: stacked_cnn

output_features:
  -
    name: class
    type: category

training:
  -
    epochs: 01

Stack Trace:
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/cli.py", line 86, in main
    CLI()
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/cli.py", line 64, in __init__
    getattr(self, args.command)()
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/cli.py", line 67, in experiment
    experiment.cli(sys.argv[2:])
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/experiment.py", line 548, in cli
    experiment(**vars(args))
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/experiment.py", line 159, in experiment
    model_definition = merge_with_defaults(yaml.load(def_file))
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/utils/defaults.py", line 148, in merge_with_defaults
    default_training_params[param])
  File "/home/mendeza/Documents/uber-ludwig/py3env/lib/python3.6/site-packages/ludwig/utils/misc.py", line 136, in set_default_value
    dictionary[key] = value
TypeError: list indices must be integers or slices, not str

I think it's just a format error. Try this:
# ludwig yaml test
input_features:
  -
    name: image_path
    type: image
    encoder: stacked_cnn

output_features:
  -
    name: class
    type: category

training:
    epochs: 01

Thanks, confirmed removing the dash worked!
Why does the training configuration in the yaml file not need a dash? Also is there more information on how to format yaml files for ludwig correctly. Having some larger yaml files on the github repo would be a helpful resource.
Thanks!

The dash is needed to read a list of elements. In the case of input_features, there may be multiple features but for training, it's only one set of options.
We will add more examples which have a longer config file. Thanks for raising this issue.


Write shortest possible code: python how to query json (python)

Please write shortest code example for this question:
python how to query json in python
How to do it:

Go to python codes
Create new file (named in underscore case, should contain key words from title) with md extension (markdown file).
Propose new file with following content (please use all three blocks if possible - title, code itself and explanations list):

# python how to query json

```python
code part1 part2 part3 ...
```

- part1 - explain code part 1
- part2 - explain code part 2
- ...

More advanced template for examples and linked solutions. More docs here.

How to query json


Co-write Makeability Lab Manifesto/Ethos

This isn't really necessary for the webpage but it's something I want to do (whether we post to webpage or not) and I wanted to remind myself to do this, so posting here.
I've been thinking about this for a long time, but Zach Lieberman's talk re-prompted it in my mind. Here's some inspiration for us:

https://youtu.be/bmztlO9_Wvo

We wrote something. Yay!
https://docs.google.com/document/d/1YiiDsfpiolpXjUTj8xWrQwQQUzqrfqT9bocOpaYDrtI/edit


Clamshell Timer

Clamshell mode is helpful through Evolution, and I would like to have it on a timer to avoid leaving it on accidentally and running out the battery in a bag or similar.
I couldn't figure out how to do this in Evolution, I tried a sleep command but then it just hangs the entire screen.
Something like OneSwitch offers this type of capability.

Hi @rabbly ,
Thank you for your suggestion. Since many users gave feedback that they expect a timer for different switches and buttons, I plan to create a common timer for all items including native switches, shortcuts and Evolution.


provide way for collective to override the seo/socialmedia metadata

The content on the collective (see https://opencollective.com/reactiveui) page for desktop/mobile is not the best content to be displayed in seo/social media situations. For example I'm planning on putting together a image+video embed at the top but am putting this off as the markdown code would in theory flow through into Slack or appear on Google.

The right way to implement this is to allow you to post updates on your collective page (and have a unique page/URL per post that you can share on social media). We'll get there.

re: posting updates on our collective page
Our project already has a blog and my preference is to concentrate all activities/updates/communication to take place on the website as that provides the maintains insights of who in the community is engaged and where ReactiveUI is being used.
Have also considered hiding backer updates unless please login with OpenCollective/GitHub account [some form of primary identifier]:


option to disable cert validation

There is no option to disable cert validation.
Specially in test/dev environments it can be a problem.
For example I fail to connect to the test instance :
Connect failed: Host name '<IP_ADDRESS>' does not match the certificate subject provided by the peer (CN=elasticsearch)

@UweW Got it, I will add this option in next version, and release in this week(with cheat sheet feature).

@UweW edql 1.9.9 released, default skip ssl verification, could you try and if any problems, pls let me know
https://github.com/chengpohi/edql/releases/tag/v1.9.9

I am not able to download and test, latest available for download is 1.9.8

Hi @UweW , Jetbrain is still reviewing the new version.  you try manual download edql plugin and install it or wait Jetbrain approve it.
https://plugins.jetbrains.com/plugin/16364-elasticsearch-query--edql/versions/stable/265120

I can confirm, it works after your update.
Thanks a lot


Switch to using shared-source ObjectMethodExecutor

See https://github.com/aspnet/Common/pull/217
We should use this instead of our copy. Should be a simple swap.

So one small problem possibly is our ObjectMethodExecutor supports static methods, and it doesn't look like the common one does :/

Send a PR then :)

If we're supporting static methods in Hubs, then Controllers should probably support them too.

https://github.com/aspnet/SignalR/commit/1da4e07fff263c39edb6382d448fac65f358020b


How to get function name given match string?

For example : https://gist.github.com/steve21124/c13c9203f1e5280611335032ada65a41

match string "ImageStack.Notifications.stackDidReload" return "func subscribe()"
2.match string "self.cameraController.takePicture" return "takePicture()"

is possible using this to get above?

This is just general regex question rather than an issue of this project. I suggest you to ask this on StackOverflow.


[4.0] [sil-opened-archetypes-tracker] Remove only those archetype definitions that are registered with the current SILOpenedArchetypesTracker

It may happen that a valid opened archetype was not registered in the SILOpenedArchetypesTracker, because the tracker was created without scanning the whole function and thus may not aware of all opened archetypes of the function. This often happens e.g. during inlining. Therefore, remove the archetype definition only if it was registered before.
Fixes rdar://problem/33513110
Explanation: Remove only those archetype definitions that are registered with the current SILOpenedArchetypesTracker and do no assume that each tracker knowns all opened archetypes of the current function.
Scope of Issue: Due to a missing check a compiler assertion would happen.
Risk: Minimal, it is a simple bug fix.
Reviewed By: Erik Eckstein
Testing: Additional test was added.
Bug: https://bugs.swift.org/browse/SR-5553
Radar: rdar://problem/33513110

@swift-ci please test

@eeckstein Could you please review for 4.0 branch?

LGTM!
Thanks for fixing!


✨ Allow using dependables with functools.partial()

Discussion: https://github.com/tiangolo/fastapi/discussions/9744
Using functools.partial() with dependables feels like a very obvious idiom. partial() already works for dependables that are plain function, but it doesn't work consistently, partial of async and callable classes didn't work consistently.
This PR makes partial() work consistently in all combinations.
test_dependency_class.py have been renamed to test_dependency_types.py to reflect its bigger scope now.
There's quite a bit of repetition in the tests, part of me wants to just refactor those; but another part of me also wants to keep the tests simple.

This would be nice to have

Yeah - I also just hit this! Please merge 😅

@tiangolo Are we any closer to a merge on this? 🙂


docs: update limits doc

Incorporate feedback from #1421

@achingbrain any chance we can address comments and close this out?

Is it possible to control the max amount of data buffered across all js-libp2p connections in total?

Yes, but it's the result of a calculation rather than a single config value.  Each connection has a multiplexer, each multiplexer has a buffer for raw incoming data, that incoming data is parsed into messages for each stream, each stream has a message buffer.  Each multiplexer has a stream limit.
The max amount of data is the raw incoming data buffer size, plus the stream message buffers multiplied by the stream limit, all multiplied by the max number of connections.

If not, is at least possible to set a max buffer limit at the connection level? I'm assuming not.

Yes, but it's controlled by the multiplexers rather than the connection.  It's not currently possible to vary the limit between connections, if that's what you mean.

I wonder if it would be easier for a user to reason about if we had a limit like maxConnectionBufferSize or maxMuxerBufferSize.

Perhaps, but I think it might be quite hard to start with the total limit and derive useful sublimits.  By which I mean, it would be nice to say "each connection can only consume 60MB", but the muxer needs to divide that 60MB and allocate some to the incoming data buffer, some to the stream buffers, etc.
I guess you could say "the incoming data buffer is always 4MB" and "stream buffers are always 4MB" so that means of 60MB, 4MB goes to the incoming data buffer, 56MB is left over so I can only have 14 streams so max 7 incoming/7 outgoing?  What if you have some streams that are always consumed quickly and others that aren't?  You might want to let the slow streams have a bigger buffer than the fast streams.
It's all really fiddly - I think it's best if users tune the limits to their particular application needs.

LIMITS.md answers specifics about js-libp2p. I think it would be good to link out to docs.libp2p.io/reference/dos-mitigation, which give a wholistic take on DoS mitigation (potentially providing useful context).

This is linked to in the opening section above the ToC - I've tried to make it a bit more obvious.

Maybe add a FAQ entry about "Is there a way to limit the number of file descriptors?".

There's no FAQ as such yet, but I've added this as a section to the doc

Close out on the DoS vector mentioned in https://github.com/libp2p/js-libp2p/pull/1421#discussion_r999898956 . At the minimum lets create the backlog item.

The backlog item is #1508 and it's referenced from the doc now

Add more clarity about connection limits per https://github.com/libp2p/js-libp2p/pull/1421#discussion_r1034149930

I've added the suggested change.
I think we should merge this now, any extra tweaks can be applied as further PRs?


[FLINK-23862][runtime] Cleanup StreamTask if cancelled after restore …

What is the purpose of the change
Move cleanUp call from StreamTask to Task so that the latter
can decide to call it if the former is cancelled between restore and invoke.
Verifying this change

the actual failre is caught by JdbcExactlyOnceSinkE2eTest (FLINK-22889) by causing many failures with limited resources (jdbc connections not closed)
unit tests added to TaskTest to verify invokable is cleaned up
unit tests removed from StreamTaskTest that were checking that StreamTask calls cleanUp()
tests for emission from close removed as it requires cleanUp() call and doesn't comply with FLIP-147 (hotfix commit)

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? no

Thanks for the review @akalash and @pnowojski,
I'll squash the commits and merge the PR then.


Lifting useLoadTableColumnsToCache out of Flyouts

Description

Flyouts had an issue where startLoading would poll indefinitely when the flyout gets closed. Lifting the useLoadTableColumnsToCache hook out of flyouts stops this behavior.
Updated UI on empty associated objects table

Issues Resolved
N/A
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Fixes with #1598


dir_mask{}.png file not available

Hi Team,
I am excited to see this paper. I was trying to run your code. When I run the code, the dir_mask{}.png which the code expects to generate is unavailable. Kindly help me out with this.
Usage of the file: https://github.com/TZYSJTU/Sketch-Generation-with-Drawing-Process-Guided-by-Vector-Flow-and-Grayscale/blob/1cf79425ae51e0a89c55aed41ea26980d54ea214/cat.py#L126

Is this file not generated or can't be opened?

Its not getting generated. I dont see where it would get generated as well

ETF_filter = ETF(input_path=input_path, output_path=output_path+'/mask',\
     dir_num=direction, kernel_radius=kernel_radius, iter_time=iter_time, background_dir=background_dir)
ETF_filter.forward()
print('ETF done')

Please improve your coding skills

Thanks for suggesting about my coding skills.
Executing the code, Seems another issue has been raised for the same issue(https://github.com/TZYSJTU/Sketch-Generation-with-Drawing-Process-Guided-by-Vector-Flow-and-Grayscale/issues/2). Can you run the code once and see for yourself. The code you show is creating mask{} files. The file that is missing is dir_mask{}.

    for i in range(self.dir_num):
        if i==0:
            minimum = -90
            maximum = -90+length/2
            mask1 = 255*(((angle>minimum)+(angle==minimum))*(angle<maximum))
            maximum = 90
            minimum = 90-length/2
            mask2 = 255*((angle>minimum)+(angle==minimum))
            mask = mask1 + mask2
            cv2.imwrite(self.output_path+'./dir_mask{}.png'.format(i),np.uint8(mask.numpy()))
        else:
            minimum = -90+(i-1/2)*length
            maximum = minimum + length
            mask = 255*(((angle>minimum)+(angle==minimum))*(angle<maximum))
            cv2.imwrite(self.output_path+'./dir_mask{}.png'.format(i),np.uint8(mask.numpy()))

The corresponding code is in the file ETF/edge_tangent_flow.py
Please check why these dir_mask{}.png files are not generated.
In addition, my code can run on most people's machines.


What's the different between tiflow/cdc/api and tiflow/pkg/api?

Before asking a question, make sure you have

[X] Searched existing Stack Overflow questions
[X] Googled your question
[X] Searched open and closed GitHub issues
[X] Read the documentation

What is your question?
Seems that the two packages are used to implement some parts of TiCDC OpenAPI.

@asddongmen Can we merge the two packages?

@lonng,tiflow/cdc/api provides the OpenAPI implementation, while tiflow/pkg/api supplied with a rest client is used to interacting with the OpenAPI. At present, tiflow/pkg/api is mainly used by cli commands. In the future, it can be extracted to another standalone project such as 'tiflow-client-go'. What's your opinion? @asddongmen @sdojjy

@lonng,tiflow/cdc/api provides the OpenAPI implementation, while tiflow/pkg/api accompanied with a rest client is used to interacting with the OpenAPI. At present, tiflow/pkg/api is mainly used by cli commands. In the future, it can be extracted to another standalone project such as 'tiflow-client-go'. What's your opinion? @asddongmen @sdojjy

Yes, I believe that we can merge these two packages in the near future.


Any good ideas how to implement count results?

I need to be about to count the ES hits, currently I am doing this:
trait ElasticCountable
{
    public static function elasticCount($filters)
    {
        $model = new static();
        $host = config('elasticsearch.hosts');
        $client = ClientBuilder::create()->setHosts($host)->build();
        $params = [
            'index' => $model->searchableWithin(),
            'type' => $model->searchableAs(),
            'body' => [
                'query' => [
                    'bool' => $filters
                ]
            ]
        ];
        return $client->count($params)['count'];
    }

Models that are elaticCountable can then do:
Article::elasticCount(['must' => ['term' => ['is_published' => true]]])

The elasticCount is very flexible, but not fluent, and doesn't utilize Scout. Any ideas?

I know this issue is closed but when you use the paginate method the total hits are returned.
{
	"current_page": 1,
	"data": [
		{
			"your_data": 1461295,
			"more_data": 1461295
		}
	],
	"from": 1,
	"last_page": 5,
	"next_page_url": "http://example.com/search?query=public&page=2",
	"path": "http://example.com/search",
	"per_page": 20,
	"prev_page_url": null,
	"to": 20,
	"total": 98
}


Done. Added a checkbox to force the adv indicator on skills/talent

Looks good, could you add a tooltip for the checkbox (include the whole div so you don't need to hover specifically on the checkbox)?

Ok, will do!


Remove test targets and test helper libraries from Makefile

Remove test / tool targets from the Makefile. We don't need them because everything can be built with cmake and bazel instead.
Removal of the test targets and test util libraries greatly reduces the size of the Makefile and opens up lot of opportunities for simplifications. The goal is to end up with a Makefile than only has the basic libraries and its structure is very simple and understandable (currently that's far from the truth).
Some interesting changes (see commit history)

switch "Microbenchmark diff" and "Trickle diff" test jobs to use bazel for building the microbenchmarks
remove support for pkgconfig files generation in Makefile (no longer necessary because make install is no longer supported)
remove lots of unneeded metatargets
make generated Makefile more readable by adding regions that make it easier to see where the generated code is coming from.

The "benchmark diff" jobs should pass once https://github.com/grpc/grpc/pull/23832 is merged.

@veblush  this is now ready for review.

CC @hcaseyal  @vjpai  in case you wanted to take a look at the changes in bm_build.py (should be harmless, it just switches the build from Makefile to bazel, but anyway).

adhoc artifacts - packages - distribtests: https://sponge2.corp.google.com/f69b003e-eefb-4da7-bf2d-40cbdacf1b40

Basic tests python failed with an infrastructure issue (seems transient and it has passed before AFAIR)
fatal: clone of 'git://github.com/google/re2.git' into submodule path '/Volumes/BuildData/tmpfs/src/github/grpc/third_party/re2' failed
Failed to clone 'third_party/re2' a second time, aborting


small error in PKGBUILD

There seems to be an error in the beginning of the PKGBUILD:
"https://api.github.com/repos/rates/lunush/releases/latest"
should be changed to:
"https://api.github.com/repos/lunush/rates/releases/latest"

Thank you for reporting the problem! It is now resolved


Allow more than one digit range in ol.TileUrlFunction.expandUrl

To be able to have number range like {9-13}

The code would be easier to read if you would not assign match inside the if statement.

I've moved the assignment outside of the if

Thanks for the review


Control process hangs

Several reports of users finding the control process 'stuck' or 'hanging' several hours or longer. Both Richard & Dave have seen these issues. Maybe related to the 'process still running after 5mins' message seen in the client logs in stderr on some tasks. Unclear.
However, these stuck control code processes are unkillable, kill -9 does not remove them. Which suggests it's waiting on some device. Restarting the client does solve it.

See full description on the pull request.


WatchOS target

Is there any reason why there is no watchOS target? ReactiveKit also build for all platforms

WatchOS is not in the plan for now. However, ReactiveKit now provides inline bindings and key path bindings that can be used to solve much of the binding problems without the need for Bond.


Create deploy.yml

action to build and deploy docker images on push to develop branch

We probably want to merge these changes into develop branch, not master. @besumi


Zipkin to external upstream fails

I am trying to send traces via the builtin Zipkin tracer in Envoy directly to NewRelic Trace API which would need some custom headers.
I am not sure what I am missing here, tried with multiple combinations of configuration. Any help is much appreciated!!
Config
  - name: zipkin
    connect_timeout: 30s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: zipkin
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: trace-api.newrelic.com/trace/v1?Api-Key=<LICENSE_KEY>&Data-Format=zipkin&Data-Format-Version=2
                port_value: 443
tracing:
  http:
    name: envoy.tracers.zipkin
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
      collector_cluster: zipkin
      collector_endpoint: "/trace/v1"
      collector_hostname: "trace-api.newrelic.com"
      trace_id_128bit: true

Logs
[2023-08-21 13:44:04.710][46][debug][http] [source/common/http/conn_manager_impl.cc:1501] [C0][S16587043647254830363] encoding headers via codec (end_stream=false):
':status', '200'
'date', 'Mon, 21 Aug 2023 13:44:03 GMT'
'server', 'envoy'
'content-length', '2'
'x-envoy-upstream-service-time', '913'

[2023-08-21 13:44:04.710][46][debug][client] [source/common/http/codec_client.cc:112] [C1] response complete
[2023-08-21 13:44:04.710][46][debug][pool] [source/common/conn_pool/conn_pool_base.cc:159] [C1] destroying stream: 0 remaining
[2023-08-21 13:44:04.711][46][debug][http2] [source/common/http/http2/codec_impl.cc:970] [C1] stream closed: 0
[2023-08-21 13:44:08.685][8][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:174] DNS refresh rate reset for trace-api.newrelic.com/trace/v1?Api-Key=<LICENSE_KEY>&Data-Format=zipkin&Data-Format-Version=2, (failure) refresh rate 5000 ms
[2023-08-21 13:44:08.826][8][debug][main] [source/server/server.cc:200] flushing stats
[2023-08-21 13:44:10.168][46][debug][connection] [source/common/network/connection_impl.cc:638] [C0] remote close
[2023-08-21 13:44:10.168][46][debug][connection] [source/common/network/connection_impl.cc:242] [C0] closing socket: 0
[2023-08-21 13:44:10.169][46][debug][conn_handler] [source/server/connection_handler_impl.cc:164] [C0] adding to cleanup list
[2023-08-21 13:44:13.676][8][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:174] DNS refresh rate reset for trace-api.newrelic.com/trace/v1?Api-Key=<LICENSE_KEY>&Data-Format=zipkin&Data-Format-Version=2, (failure) refresh rate 5000 ms
[2023-08-21 13:44:13.826][8][debug][main] [source/server/server.cc:200] flushing stats

cc:  @wbpcode @Shikugawa

So, what's your problem? I didn't get it. And are your sure your address is a legal name for dns?

Hi @wbpcode , yes address is correct (documentation: https://docs.newrelic.com/docs/distributed-tracing/trace-api/report-zipkin-format-traces-trace-api/#existing-zipkin. I was able to send traces successfully on this trace API with an opentelemetry collector configured with zipkin receiver. For some reasons we need to make export traces directly from envoy without any collector.
I can curl to the trace api from within envoy container but spans being generated are not hitting API. Can't pinpoint where issue is from the logs. So just need some guidance if there is anything you see from my config which I might be doing wrong.

In the envoy, the address should be raw IP or pipe or hostname/dns domain. But in your config, you configured an complete url  as value of address. I think this is the most obvious issue I have found.


updated defendent claims on dashboard

JIRA link
https://tools.hmcts.net/jira/browse/CIV-6145
Change description
update status for defendant claims
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

reopen for testing

wrong pull request


Support enabling and disabling of easyTravel scenarios

Currently, there is no user-friendly way for enabling and disabling easyTravel performance scenarios.

Totally wrong project. Closing.


feat: add all networks to offchain treasuries

Summary
This is follow up to previous PR, which now will allow you to select any v1 network for offchain spaces treasuries.

Combobox now supports null (it didn't work previously, but it was bug in headless-ui, so had to bump it).
Combobox is used for both offchain and onchain treasuries (so we now have icons and search).
On offchain spaces v1 spaces list is used, on onchain spaces enabledNetworks are used.
Fixed issue when saving treasuries with duplicate detection (editing strategy without changing anything would show duplicate message).

Towards https://github.com/snapshot-labs/sx-monorepo/issues/830
Depends on https://github.com/snapshot-labs/sx-monorepo/pull/840
How to test

Go you your offchain space treasury settings.
You can select among all networks.
On onchain space settings you have limited numbers of networks available.

Screenshots

Selector looks good, it's just missing Starknet. (or Starknet Sepolia for testnet deployment).
I also got the same issue than the one reported here: https://github.com/snapshot-labs/sx-monorepo/pull/840#issuecomment-2394691231 when I try to update / rename, remove or add a new treasury and click "Save".

Handled both things, if you find more edge cases I'd say we should take it in different PR to see if it's something that we are mishandling or maybe a bug in headless UI.


Question about map_fusion_fission_client.py

Dear Dr. Charlottewright:
Hello! When I ran the code you provided to infer the lepidoptera chromosome fusion event, an error occurred and I do not know how to solve it. The following is my code and error information:
"python3 map_fusion_fission_client.py -i fusion_finder/ -tree ../syngraph/4707.newick -t 1 -o ./ -f output -l Ture" The following information is displayed:
[+] Running map_fusion_fissions.py with a threshold of 1
[+] Parsing chromosome assignment files
[+] Parsing the tree
[+] Summarising status of each chromosome
Traceback (most recent call last):
File "~/unmasked_genome/LFFF/map_fusion_fission_client.py", line 45, in
df_combined, spp_list = gather_stats_and_make_table(file_list, input_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/unmasked_genome/LFFF/merian_tools.py", line 117, in gather_stats_and_make_table
for query_chr in file.query_chr:
^^^^^^^^^^^^^^
File "~/miniconda3/lib/python3.11/site-packages/pandas/core/generic.py", line 6299, in getattr
return object.getattribute(self, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'query_chr'.
I put the lep_fusion_fission_finder.py results for each species in the fusion_finder folder, and the chromosome numbers for each species are 1-31.   Can you tell me how to solve it?
Another problem is that I use syngraph and the species at the base of my species tree as reference. Under the -m2 mode, I infer that chromosome fusion of my species is 13 times, while using LFFF method -w 17, I infer that my species is a butterfly for 12 times. May I ask how the conflict between these two results should be explained? I see you say in the article that the results of the two methods are almost the same, and my species is  also a butterfly of the  Lepidoptera.
Sincerely,
Xuan Zhang

Hi Xuan,
Thank you for trying out this tool! It seems the problem is accessing the output from the first command (lep_fusion_split_finder.py). Please could you check whether your output from this does include files with the format "SPP_chromosome_assignments.tsv" where the first line in the file has the header "query_chr status assigned_ref_chr" followed by declarations for each of your chromosomes? The line "AttributeError: 'DataFrame' object has no attribute 'query_chr'." makes me think this isn't being found. If they do look ok, feel free to send me your output from lep_fusion_split_finder.py and your tree I can try to look into it further.
The difference between the number of fusions detected by syngraph and lep_fusion_fission_finder is most likely due to the different thresholds of sensitivity that are being used by each tool. If you are not specifying a number of markers for a given chromosome to be detected with syngraph, then the default of 5 is being used, which is more sensitive than the threshold of 17 with lep_fusion_fission_finder. Playing with this number of markers with both tools should help you to get a feel of what changes with this and what you think the best cut-off should be. It may also help to visualise your fusions to see which fusion is being missed when you have 12 rather than 13, and whether this is a genuine fusion that should be detected. To do this, you could upload your BUSCO table for the species of interest here: https://charlottejwright.shinyapps.io/busco_painter/ or run it on the command line via this tool https://github.com/charlottewright/lep_busco_painter.
Best wishes,
Charlotte


[dio] 💡 Improve comments

[x] adapters
[x] dio
[x] transformers
[x] cancel token
[x] dio error
[x] dio mixin
[x] form data
[x] headers
[x] multipart file
[x] interceptors
[x] options
[x] parameters
[x] response
[x] redirect record

New Pull Request Checklist

[x] I have read the Documentation
[x] I have searched for a similar pull request in the project and found none
[x] I have updated this branch with the latest main branch to avoid conflicts (via merge from master or rebase)
[ ] I have added the required tests to prove the fix/feature I'm adding
[x] I have updated the documentation (if necessary)
[x] I have run the tests without failures
[x] I have updated the CHANGELOG.md in the corresponding package

Additional context and info (if any)

Thanks for the review @ueman. I was attending some offline meetings and paused the OSS work for about 2-3 weeks. Will continue soon, maybe the next Monday.

All code comments are updated (probably).

@ueman I'll be waiting until you've confirmed the above comments are resolved.


[WIP] Enable users to request default/row-major layouts on result nodes

is it possible to add a unittest for your PR using mem_reuse API. (mem reuse API has one unittest)


GUI browse of (D64) files goes wrong with the second time browsing

if i browse a d64 file (or amiga) and i push the back button, select a different (or the same) d64 file en push browse again i get an error:
error: Layout : if you use a sector count , you must specify exactly one start sector.
i suspect something is still not going as it should...

Bah, I forgot to clear the config when loading the new one.


Added Melon symbol + contractmap.json info + removed other listing

Sorry for the slowness, trying to keep the repo as clean as possible.
In the diff, it looks like we already had Melon in the directory, so there are two major changes here:

Renaming Melon Token to Melon (sounds fine)
Replacing the SVG with a PNG. Is this necessary? SVGs look much better on a wider variety of screens. Was our old logo bad?

Oh also, if the old image is bad, we should remove it from this repo, too.

@thehighfiveghost couple things to address, look forward to merging


[FLOC-2368] Revert accidental readme updates

Thanks @davidoliverSP2 . I can see that this only touches the README and I happen to know that that doesn't affect the build, so I'll merge.


Named aggregation doesn't work in DataFrame.agg

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Ubuntu 19.10

Modin version (modin.__version__):

<IP_ADDRESS>+31.ga0a3ebb

Python version:

Python 3.8.3

Code we can use to reproduce:

import numpy as np
import modin.pandas as pd
#import pandas as pd

df = pd.DataFrame({
    "col1": [ 1,  2,  1,  2,  1,  3,  3,  2,  3],
    "col2": [11, 12, 13, 14, 15, 16, 17, 18, 19],
    "col3": [ 4,  4,  4,  4,  5,  5,  5,  5,  5],
    "col4": [ 1,  1,  1,  1,  1,  1,  1,  1,  1]
})
print(df, '\n')

df1 = df.agg(new_col=('col2', max))
print(df1, '\n')
df2 = df.agg(new_col1=('col2', 'max'), new_col2=('col2', 'min'), new_col3=('col2', 'sum'),
              new_col4=('col4', 'max'), new_col5=('col4', 'min'), new_col6=('col4', 'sum'))
print(df2, '\n')

Describe the problem

Running this code results in an exception:
Traceback (most recent call last):
  File "agg_test2.py", line 13, in <module>
    df1 = df.agg(new_col=('col2', max))
  File "/localdisk/gashiman/modin/modin/pandas/base.py", line 504, in aggregate
    return self.apply(func, axis=axis, args=args, **kwargs)
  File "/localdisk/gashiman/modin/modin/pandas/dataframe.py", line 289, in apply
    query_compiler = super(DataFrame, self).apply(
  File "/localdisk/gashiman/modin/modin/pandas/base.py", line 716, in apply
    raise TypeError("{} object is not callable".format(type(func)))
TypeError: <class 'NoneType'> object is not callable

This bug may be related to how named aggregation worked in groupby before #2254 was fixed.
Source code / logs

When we're doing renaming aggregation we're passing functions to aggregate via kwargs, so the positional func argument will be None. However, if-statement that dispatches us to the dict-handling branch only checks that func is a dict, another condition to enter dict-handling branch should be func is None.
https://github.com/modin-project/modin/blob/f14a3c187fd2349348e74c9e5095884371ef8645/modin/pandas/base.py#L528-L530


using 2 grafikcards of different type

hey,
i´m running right now a test on a nvidia 1660i with 8GB ram and i had to go down to 480 resolution. i´m just wondering if it is possible to run a multi GPU system with this. Since direct12 a multi GPU system is possible on games.
could it work here too?
i have here a extr sapphire nitro+ with 8GB ram for example.

Multi-GPU is possible.
but saves time by dividing the given task rather than the concept of performance improvement.
Use the -g 0,1 command.
If there is a built-in graphic, for example, in the task manager, if the built-in graphic number is GPU 1, you must enter -g 0,2.

that sound promissing. just to be sure, you mean the graphic from the iGPU on the CPU?
I´m using Windows10 with a i7-10700 on an AsRock Z490 PG Velocita. I checked the Task-Manager, but ther is only one GPU, my GTX1660Ti visible.
I cant finde a tutorial for this purpose. Maybe you can help me out.
If i need a different OS, just tell me, i have a currently unused NVMe SSD on board.

If only 1 separate GPU was installed and 2 GPUs did not appear in the task manager, iGPU was already disabled.
In order to use multi-GPU, in addition to GeForce GPU, did you install sapphire nitro+ with 8GB GPU on your PC?

Right now i have only the GTX installed. So I have to enable the iGPU?
But I could Install the Nitro+ also. Then i have 3 GPU's 😁
What do you suggest is the most promissing way?

You do not need to turn on the iGPU.
Using iGPU as a DAIN multi-GPU can rather increase RAM usage and create a work bottleneck.
After installing your Radeon GPU use the -g 0,1 command.
However, if black frames occur in some frames during interpolation, it is due to GPU acceleration, so you may need to turn on iGPU for display output.
In this case, you must enter the -g command without the built-in graphic number.
The way you turn on the built-in GPU varies from motherboard to motherboard, so you need to find the setting yourself and enable it.

ok, so i try first the Nitro+
You mentioned before the -g command. Could you explain that a little more? I´m not familiar with that.
I was for the internal GPU already very happy, because i found this view on the taskmanager. Thats a lot of V-RAM.
From the 64GB RAM the DAINApp is using only 7 oh so.
I attach a view screenshots.

Dain-App and Dain-ncnn are two different programs

@Mar2ck
thats not really correct "ncnn implementation of DAIN, Depth-Aware Video Frame Interpolation."
It is the same DAIN App, but this version is also for Windows/Linux/MacOS Executable for Intel/AMD/Nvidia GPU

Theyre based on the same thing but they function differently. Dain-ncnn is a CLI program so you'd need to run it from the command line and use the -g argument


Daily forecast from EC offset by 1 day

Describe the bug
In rare occasions, the data returned from the MSC Datamart for the daily forecast is mislabelled, resulting in the forecasts being offset by 1 day.
Code in question: daily_factory.rb.
To Reproduce
Steps to reproduce the behavior:
Only occurs in rare circumstances when the data is not formatted as expected.
Expected behavior
Regardless of input data, the forecast is correctly produced.
Additional context
Originally spotted by @williamlauzon in lfroms/clouds#72.

Just a heads up, I noticed this issue happening last night (February 11th) and tonight (February 12th). The forecast displayed is correct, but the dates displayed are incorrect. For example, today's weather and tomorrow's weather appear to match what is on the EC website, but today is labelled as February 13th and tomorrow is labelled as February 14th. Also, in case it helps to provide additional context, another app using the EC data (but not WeatherCAN) is correctly displaying the dates right now. I just thought I would provide some details in case it helps identify this issue better!

Thanks, @williamlauzon. I've confirmed that this is a parsing issue in Nimbus. The code made certain assumptions about the data that did not always hold true.
I'll be deploying a fix shortly.

Reopening temporarily as there appears to be a bug in the code where the dates are incorrect on certain days of the week.


#1330 I've slightly changed the code to get AuthID

#1330

This has to be part of a major update since it's a breaking change. There's no real migration path either so I'm going to close this.


Hoverfly prepends a "./" to proxied request's query parameters if any parameter value contains a colon ":"

Start a simple HTTP server locally:
const http = require('http');
const hostname = '<IP_ADDRESS>';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World\n');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Start hoverfly locally in capture mode, on the default port 8500:
hoverfly -capture

Make a request to the server, with at least one query parameter whose value contains a colon :
curl -x 'http://localhost:8500' 'http://localhost:3000/hello?c=b&a=d:e'

Hoverfly log:
INFO[2017-09-20T13:09:52-07:00] request and response captured                 mode=capture request=&map[scheme:http destination:localhost:3000 path:/hello query:map[c:[b] a:[d:e]] headers:map[User-Agent:[curl/7.43.0] Accept:[*/*] Proxy-Connection:[Keep-Alive]] body: method:GET] response=&map[error:nil response]

Actual request as seen in Wireshark:

As can be seen in the above capture, Hoverfly rearranged the query parameters, but also pre-pended a ./ to them, in effect causing the a parameter to become invalid.
This is breaking any such proxied requests that contain a colon in their query parameter values.
Is there a workaround possible for this?
Hoverfly version:
hoverfly --version
v0.14.0

Hey @kodeninja,
I am able to recreate this issue locally and am looking into fixing it. I anticipate a fix being in the next release of Hoverfly, v0.14.1, which I expect will be released Friday or Monday.
I do not think there is a workaround to this.
I will update this issue when I've fixed this issue.

Thanks, @benjih, for the prompt response!

The fix had been released as v0.14.1. Please try it out.

@tommysitu, @benjih I can confirm the issue is fixed with the latest hoverfly (I tested with v0,14,2)!

Thanks a lot for your help!


Node.js bindings: Added JS/lowlevel bindings for AIO and build stuffs

This patch adds native/lowlevel bindings for AIO and using the same
JS API for AIO is implemented and includes the related build changes.
Signed-off-by: Srinivasa Ragavan<EMAIL_ADDRESS>Signed-off-by: Sudarsana Nagineni<EMAIL_ADDRESS>

@barbieri Can you review the AIO apis?

looks good, however I miss tests or samples on their usage. I was to comment on that for other IO apis, but feel free to add them later.

Thanks @barbieri . We will be adding samples & tests as separate patches.


[Proposal] Integrate 101 as a Data Provider

Integrate 101 as a Data Provider will enable to create new Groups from holders of a specific 101 Course NFT.
Specifications
{
  "name": "101",
  "iconUrl": "",
  "providerClassName": "101Provider",
  "functions": [
    {
      "name": "Get holders of course NFT",
      "functionName": "TODO",
      "countFunctionName": "TODO",
      "description": "Returns all holders of a specific 101 course",
      "args": [
        {
          "name": "course",
          "argName": "TODO",
          "type": "string",
          "required": "true",
          "example": "TODO",
          "description": "A specific course identifier"
        }
      ]
    }
  ]
}

Tips on how to integrate the Data Provider
A 101 Course Token is a non-transferrable ERC-721 NFT minted on Polygon or Celo Chain with these contract addresses.
The easiest way to extract addresses for each 101 Course NFT is by using 101 Subgraph.
Have a look at Sismo documentation to understand what is a Data Provider, and checkout the tutorial to know how to integrate a Data Provider.
Have fun!

Hey @mme022
Again, thanks a lot for your contribution to the Sismo Hub! 💜
You can still contact me on Discord (martingbz.eth#3809) or on Twitter (@0xMartinGbz) to discuss about the bounty payment process. :)


Possibility to remove total number line

Category

[x] fzf binary
[ ] fzf-tmux script
[ ] Key bindings
[ ] Completion
[ ] Vim
[ ] Neovim
[ ] Etc.

OS

[ ] Linux
[ ] Mac OS X
[ ] Windows
[ ] Windows Subsystem for Linux
[ ] Etc.

Shell

[ ] bash
[ ] zsh
[ ] fish

It's just a question, is it possible to eliminate the line that shows number of items?

No option for that currently. Alternatively, you can consider using --inline-info or change the color of the line using --color option. See man fzf for details.

Yeah I ended up using --inline-info but I still think it's nice to be able to remove it altogether instead of inlining it.

Any chance of an option being added that hides these numbers? --inline-info doesn't make things much better; to me the numbers are just random noise I'd rather not have to perceive.


@Cache

I think it would be nice to use lombok for method caching.
@Cache
private String fetchBlacklist() {
return blacklist; (content is static)
}
Lombok should wrap the method and execute it only once and save the return value in a private variable or so.
If the method is called again, the cached value should be returned instead of executing the method again and again.

I think this behaviour could be accomplished by using @Getter(lazy=true): https://projectlombok.org/features/GetterLazy.html

@thisisthekap oh great, thanks. Did know that feature. But in this case i have to write the variable boilerplate and the calculation occurs if the object gets constructed, not if the method gets called.

Using @Getter(lazy=true), the calculation is triggered by the first call of the generated getter (not on object-construction).

Oh you'r right. This is quite confusing that lombok removes the private variable. Thank you very much

Lombok still keeps a private variable around. If we were to add this feature we'd.... still have a private variable. We need to store that cache somewhere!
More generally this should be called @Memoize and should in theory at least also be applicable to methods with arguments, mapping the argument(s) onto the memoized result. This gets you into discussions on how long to cache this for, etc.
Very complex stuff. Let's stick with what @Getter(lazy = true) can do for now.


Feature Request: Highlight evaluation

Hi, I'm enjoying playing with Conjure!
I was trying to add a brief highlight when evaluating an expression, I hacked something in Lua (My fennel and Lua knowledge is almost non-existent)
It looks something like this. Here I'm evaluating this root-form:

Or check this asciinema:

I got:
diff --git a/lua/conjure/eval.lua b/lua/conjure/eval.lua
index ed21063..f27b746 100644
--- a/lua/conjure/eval.lua
+++ b/lua/conjure/eval.lua
@@ -128,11 +128,35 @@ do
   client_exec_fn = v_0_
 end
 local eval_str = nil
+local function highlight_range(opts)
+  if not (type(opts.start) == "table" and type(opts.finish) == "table") then
+    print("arguments start or finish: expected table")
+    return nil
+  end
+  local start = opts.start
+  local finish = opts.finish
+  local start1 = {}
+  start1[1] = start[1] - 1
+  start1[2] = start[2]
+  local finish1 = {}
+  finish1[1] = finish[1] - 1
+  finish1[2] = finish[2] + 1
+  local bufnr = opts.bufnr or 0
+  local higroup = opts.higroup or "IncSearch"
+  local timeout = opts.timeout or 500
+  local NAMESPACE = vim.api.nvim_create_namespace "conjure_highlight"
+  vim.highlight.range(bufnr, NAMESPACE, higroup, start1, finish1)
+  vim.defer_fn(
+    function() vim.api.nvim_buf_clear_namespace(bufnr, NAMESPACE, 0, -1) end,
+    timeout
+  )
+end
 do
   local v_0_ = nil
   do
     local v_0_0 = nil
     local function eval_str0(opts)
+      highlight_range({start=opts.range["start"], finish=opts.range["end"]})
       local function _3_()
         if opts["passive?"] then
           return opts

The function accepts a range (Conjure range and Neovim range have an offset, nothing difficult), a buffer number, a highlight group and a duration.
Is this something you think would be valuable in Conjure (Maybe behind a config flag)? The problem is: I don't know how to write the fennel code that outputs this Lua.
Thanks!

This is released! The help text for it can only be found on develop at the moment though.