3D convolution and pooling for R

I would like to know if 3D convolution and pooling is available for R? The convolution can take in kernel and stride with 3 dimension but the pooling is not able to do so. Further more, when I tried to use the infer_shape() function to verify the shape of the convolution, the function only accept data with 4 dimension, which is 1 dimension less for a 3D convolution.
Would appreciate a if anyone can show me how to enable 3D convolution and pooling in R.

@EddieIT Can you also show your code? I will look into it.

@EddieIT @thirdwing
#I think this script recreates the problem?
require(mxnet)
#===============Example Data=================
dataGen <- function(a,b,c,d,e){data <- as.logical(c(round(runif(9000)-a,0),
round(runif(9000)-b,0),
round(runif(9000)-c,0),
round(runif(9000)-d,0),
round(runif(9000)-e,0)))
return(data)}
class0 <-dataGen(.1,.1,.5,.1,-.2)
class1 <-dataGen(.1,.4,.5,-.4,.1)
class2 <-dataGen(.5,-.1,.2,-.1,.1)
train.array <- rbind(class0,class1,class2)
dim(train.array) <- c(3,3,5,3000)
train.y <- as.array(c(rep(0,1000),rep(1,1000),rep(2,1000)))
trainIter = mx.io.arrayiter(data = train.array, label = train.y)
===============Symbolic Model=================
data <- mx.symbol.Variable(name='data')
drop1 <- mx.symbol.Dropout(data=data,p=.1)
first conv
conv1 <- mx.symbol.Convolution(data=drop1, num_filter=50, kernel=c(1,1,5))
relu1 <- mx.symbol.LeakyReLU(data=conv1)
second conv
conv2 <- mx.symbol.Convolution(data=relu1, num_filter=30, kernel=c(2,2,1))
relu2 <- mx.symbol.LeakyReLU(data=conv2)
flat
flatten <- mx.symbol.Flatten(data=relu2)
fc <- mx.symbol.FullyConnected(data=flatten, num_hidden=300)
drop2 <- mx.symbol.Dropout(data=fc,p=.5)
final fullc
fcOut <- mx.symbol.FullyConnected(data=drop2, num_hidden=3)
myNet <- mx.symbol.SoftmaxOutput(fcOut)
===============Generate Solver================
model <- mx.model.FeedForward.create(symbol=myNet,
array.layout='auto',
X=trainIter,
ctx=mx.cpu(),
num.round=100,
wd=0.00001,
initializer=mx.init.Xavier(),
eval.metric=mx.metric.accuracy,
learning.rate=0.01,
optimizer = 'adagrad',
verbose = T)
Resulting Error:
[11:27:22] D:\Program Files (x86)\Jenkins\workspace\mxnet\mxnet\dmlc-core\include\dmlc/logging.h:300: [11:27:22] d:\program files (x86)\jenkins\workspace\mxnet\mxnet\src\operator./convolution-inl.h:423: Check failed: dshp.ndim() == 5 (4 vs. 5) Input data should be 5D in batch-num_filter-depth-y-x
Error in symbol$infer.shape(list(...)) :
Error in operator convolution22: [11:27:22] d:\program files (x86)\jenkins\workspace\mxnet\mxnet\src\operator./convolution-inl.h:423: Check failed: dshp.ndim() == 5 (4 vs. 5) Input data should be 5D in batch-num_filter-depth-y-x

@haimeh Sorry for the late reply. You can get the information from the python API document: http://mxnet.io/api/python/symbol.html#mxnet.symbol.Convolution
In 3D convolution, the shape of input should be (batch_size, channel, depth, height, width).
You can try to change dim(train.array) <- c(3,3,5,3000) to dim(train.array) <- c(3,3,5,1,3000).


Infrastructure Design

As described in the PID: "Create a workflow design from model egress into data
hosting environment, with supplemental tool integrations."
This appears to be getting worked on by via #5

@jemrobinson The note for the SAS token is to supply the full connection_string including the Endpoints, in addition to the SAS token itself.


proposal: add kubernetes custom resources for values and variable expansion

ProblemStatement: As an Kubernetes engineer, I would like to extract values from one or more custom resources & use them to build other custom or native resources. I would like to have a kubernetes native solution instead of templating & out of band solutions.

Draft Ideas:

These are some new custom resources:

kind: ConfigFormation
kind: Config

Make use of kind: GenericController from metac
ConfigFormation is something that will be configured to extract values from one or multiple resources
Config is the output that gets built based on ConfigFormation


Added -U flag to autoload

This is for #66 .
For the zsh experts out there, I hope the -U flag can be explained. From what I understand, you need the -U to ensure that the function you're calling corresponds to a file with that filename somewhere in your bin path?

Thanks for the fix!


added docs

this is just Gerard teaching Lely to write docs and PR the to the main code base.

thanks looks good. thanks you.
LGTM


Laravel 5.3 Error: Unauthenticated Guard issue, could be an easy fix

Laravel Version: 5.3.28
PHP Version: 7.0.2
Database Driver & Version: mysql

I've come across an interesting issue with our system. I'm running Laravel 5.3.28 and am successfully using the initial authentication routes that come with laravel out-of-the-box. In addition I've successfully set up and am using passport as well. A person is able to both log in and/or register onto the system and receive an access token in both cases, however:
The problem
The GET /user route keeps returning an 401 unauthenticated error.
What I've done
Inside of my App/Http/Middleware/Authenticate.php file I have a method authenticate(array $guards).
    /**
     * Determine if the user is logged in to any of the given guards.
     *
     * @param  array  $guards
     * @return void
     *
     *  @throws \Illuminate\Auth\AuthenticationException
     */
     protected function authenticate(array $guards)
     {
     if (empty($guards)) {
     return $this->auth->authenticate();
     }

     foreach ($guards as $guard) {

     if ($this->auth->guard($guard)->check()) {
        return $this->auth->shouldUse($guard);
     }
     }
     //Error Thrown on line below
     throw new AuthenticationException;
     }

As noted above in the comment, the error is thrown there. What I'm confused about is that the $guard variable contains the guard string: 'api'. When it's ran through the check() method the $this->user() (used inside of check()) returns null which then makes $this->auth->guard($guard)->check() return false.
Note
$this->user() is used in check() which is a method found in vendor/laravel/framework/src/Illuminate/Auth/RequestGuard.php.
In the end, I'm a little stuck here and in need of some assistance. If more information is needed please ask and I will provide as much as I can. Is there something specific I may have missed here? Maybe in the Login or registration?
Thank you in advance.
Update 1
Diving a bit further I went into the vendor\illuminate\auth\RequestGuard.php file and found that the attempt to get the user for the check method mentioned above was returning null:
RequestGuard.php Class
    namespace Illuminate\Auth;

    use Illuminate\Http\Request;
    use Illuminate\Contracts\Auth\Guard;

    class RequestGuard implements Guard
    {
    /**
     * Get the currently authenticated user.
     *
     * @return \Illuminate\Contracts\Auth\Authenticatable|null
     */
    public function user()
    {
        // If we've already retrieved the user for the current request we can just
        // return it back immediately. We do not want to fetch the user data on
        // every call to this method because that would be tremendously slow.
        if (! is_null($this->user)) {
            return $this->user;
        }

        return $this->user = call_user_func($this->callback, $this->request); <-- returns null here
        }

method the $this->user() (used inside of check()) returns null

It means the user isn't authenticated, make sure you pass a valid token while attempting the request.


minikube addons enable ingress

🔎  Verifying ingress addon...
❌  Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
😿  If the above advice does not help, please let us know:
👉  https://github.com/kubernetes/minikube/issues/new/choose

It seems to be the minikube issue？

It seems to be the minikube issue？

closing as it does not seems to be an issue of kubevela


some minor issues

This is very good but recently got some problems:

No option on where to save the final .ovpn file. I'd like it to be in the same folder that the script is located so I can easily have access to move it to more safe places. Right now /root folder is hard to reach.

It does not give the flexibility of choosing key sizes and encryption method while older version had that feature.

Thanks a lot for this awesome script by the way.

The script defaults to the home directory for saving the .ovpn because that's statistically the place most likely to have write permissions and to be useful for the user. Unless someone comes with a better idea, this will likely not change, I don't like the idea of creating an additional prompt for this, sorry.
No version of this script has ever implemented those features, because the proper settings are already selected for the user. You are probably confused because
a  copy  of  this  script  advertised by someone unrelated to this project cluelessly offers that choice. I say cluelessly because from OpenVPN 2.4 the cipher is negotiated except if negotiation is explicitly disabled, which he doesn't do.

I hope that this clarifies things a bit. I like to keep the script simple and user friendly, so advanced or niche features are not likely to be implemented.

ok, now that I have you available, is it possible to quickly change the server ip after the installation? For me the server is at home so the ip changes once in a while, so I have to uninstall ovpn and start over to give it the new ip. Is it easy/possible to have an option for updating server IP?

If your server is inside a private network, it is easy because its private IP address doesn't change, only the public IP address which is not directly exposed to the server. See #456 if this is your situation. In that case, you should probably use a dynamic DNS service to help with this.
If your server is directly exposed to the internet (without NAT) situation is a bit more complex because you need to update the server configuration, but there has been discussion on this issue tracker with some guidance about it if you are interested.


[do not merge]: reverts custom service account

This PR was generated using Autosynth. :rainbow:
Synth log will be available here:
https://source.cloud.google.com/results/invocations/33c5c220-78f5-4bb3-bc40-7c0125925327/targets

[ ] To automatically regenerate this PR, check this box.

we should add these config files as excludes in synth.py.


Need pg_parser_browserified.js for embedding

Should it be included in the docs?
https://docs.blitzboard.io/how-to/share#embed-into-your-pages

The dependent libraries are now packaged.
The docs is also updated.
https://github.com/blitzboard/gitbook/commit/2747ace326076b24e6b8e4a27dc57e68090a764d


Feature Request: Set background color and/or image

Current slide design features only one 'layer' ontop of white canvas.

Add option to set background color of canvas.
Add option to set a background image (per slide)
Add option to set default background image for all slides (only if no background image was set before)

Changing background colors is something I've been meaning to implement. I need to look into it sometime.
I can certainly implement background images aswell.
This will be solved when the features described in #28 are implemented.

I implemented two new tags: [bgimg] for changing background images and [bgcolor] for changing the background color. These tags will ship in v1.1.0 but they are already merged in next and included in nightly Docker builds if you want to give them a try.


Publish website to aws workflow

Completed workflow to publish the resulting website for each push or PR to aws s3.

Can we get a comment on the pull request with the link (if we are on a pull request ?)

Comment AWS S3 URL for pull requests added.
New aws environment added with deployment URL set to the commented URL

Corrected non-matching commit sha


Welcome to the SingularityNET Ambassador Program

Welcome to the SingularityNET Ambassador Program

The SingularityNET Ambassador Program is dedicated to growing awareness of the importance of decentralized AI/AGI and the SingularityNET ecosystem. Through the program, we aim to provide structure and rewards to community members who are committed to contributing towards a beneficial AGI.
For background information on the SingularityNET Ecosystem, see https://singularitynet.io/ecosystem. Our main drivers are transparency, decentralization, and involvement. To uphold these values, community members are fully involved in decision-making processes, including how tokens are spent. This is achieved through regular Town Hall meetings, which are open and accessible to all members of the community. Together, we can work towards creating a world where AGI benefits all of humanity.
Currently this space is in development. It aims to be the single source of truth for the Ambassador Program.
Besides this page you can check the available tasks on Dework - app.dework.xyz/singularitynet-ambas
You can check the Getting started section, including the sign-up form, the meeting schedule and notes, and the rewards distribution.


plot arrays across faces without cutout

This issue closes #410 and #409
The following are tasks that need to be completed before merging this PR. Until then, consider this as a work in progress

[x] Add plot function to code base that works.
[x] Include some tests.
[x] Update API / documentation that clearly demonstrate how it works.
[x] Include a link to an example notebook

I will work on completing the rest of the tasks sometime this week, or the weekend at the latest.

Hey Tom. catalog_ECCO.yml it is just a catalog on my local computer that I use for development since it points towards the test data from oceanspy. I think it might be better to run the notebook on sciserver. I will make the proper modifications to do that (3 cells)

@ThomasHaine I changed a bit the notebook in the hyper link above to run in Sciserver, it also has description on how to install the correct oceanspy version (this branch) in a sciserver environment, and also added improved description of the notebook.  There are about 20 examples.
The notebook successfully ran in Sciserver with ECCO. I'd expect the same with LLC4320.

@Mikejmnez thanks. I'll check...

With LLC4320 it errors at the first plot with:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File <timed exec>:8

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/oceanspy/plot.py:1474, in _plotMethods.faces_array(self, **kwargs)
   1472 @_functools.wraps(faces_array)
   1473 def faces_array(self, **kwargs):
-> 1474     return faces_array(self._od, **kwargs)

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/oceanspy/plot.py:1127, in faces_array(od, Ymoor, Xmoor, varName, xoak_index, face2axis, **kwargs)
   1121 if xoak_index not in _xoak.IndexRegistry():
   1122     raise ValueError(
   1123         "`xoak_index` [{}] is not supported."
   1124         "\nAvailable options: {}"
   1125         "".format(xoak_index, _xoak.IndexRegistry())
   1126     )
-> 1127 ds_grid.xoak.set_index(["XC", "YC"], xoak_index)
   1128 cdata = {"XC": ("mooring", Xmoor), "YC": ("mooring", Ymoor)}
   1129 ds_data = _xr.Dataset(cdata)  # mooring data

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/xoak/accessor.py:101, in XoakAccessor.set_index(self, coords, index_type, persist, **kwargs)
     98 self._index_type = index_type
     99 self._index_coords = tuple(coords)
--> 101 coord_objs = [self._xarray_obj.coords[cn] for cn in coords]
    103 if len(set([c.dims for c in coord_objs])) > 1:
    104     raise ValueError(
    105         'Coordinates {coords} must all have the same dimensions in the same order'
    106     )

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/xoak/accessor.py:101, in <listcomp>(.0)
     98 self._index_type = index_type
     99 self._index_coords = tuple(coords)
--> 101 coord_objs = [self._xarray_obj.coords[cn] for cn in coords]
    103 if len(set([c.dims for c in coord_objs])) > 1:
    104     raise ValueError(
    105         'Coordinates {coords} must all have the same dimensions in the same order'
    106     )

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/xarray/core/coordinates.py:289, in DatasetCoordinates.__getitem__(self, key)
    287 def __getitem__(self, key: Hashable) -> DataArray:
    288     if key in self._data.data_vars:
--> 289         raise KeyError(key)
    290     return self._data[key]

KeyError: 'XC'

@Mikejmnez any ideas about how to fix?

With LLC4320 it errors at the first plot with:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File <timed exec>:8

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/oceanspy/plot.py:1474, in _plotMethods.faces_array(self, **kwargs)
   1472 @_functools.wraps(faces_array)
   1473 def faces_array(self, **kwargs):
-> 1474     return faces_array(self._od, **kwargs)

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/oceanspy/plot.py:1127, in faces_array(od, Ymoor, Xmoor, varName, xoak_index, face2axis, **kwargs)
   1121 if xoak_index not in _xoak.IndexRegistry():
   1122     raise ValueError(
   1123         "`xoak_index` [{}] is not supported."
   1124         "\nAvailable options: {}"
   1125         "".format(xoak_index, _xoak.IndexRegistry())
   1126     )
-> 1127 ds_grid.xoak.set_index(["XC", "YC"], xoak_index)
   1128 cdata = {"XC": ("mooring", Xmoor), "YC": ("mooring", Ymoor)}
   1129 ds_data = _xr.Dataset(cdata)  # mooring data

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/xoak/accessor.py:101, in XoakAccessor.set_index(self, coords, index_type, persist, **kwargs)
     98 self._index_type = index_type
     99 self._index_coords = tuple(coords)
--> 101 coord_objs = [self._xarray_obj.coords[cn] for cn in coords]
    103 if len(set([c.dims for c in coord_objs])) > 1:
    104     raise ValueError(
    105         'Coordinates {coords} must all have the same dimensions in the same order'
    106     )

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/xoak/accessor.py:101, in <listcomp>(.0)
     98 self._index_type = index_type
     99 self._index_coords = tuple(coords)
--> 101 coord_objs = [self._xarray_obj.coords[cn] for cn in coords]
    103 if len(set([c.dims for c in coord_objs])) > 1:
    104     raise ValueError(
    105         'Coordinates {coords} must all have the same dimensions in the same order'
    106     )

File ~/mambaforge/envs/Oceanography/lib/python3.10/site-packages/xarray/core/coordinates.py:289, in DatasetCoordinates.__getitem__(self, key)
    287 def __getitem__(self, key: Hashable) -> DataArray:
    288     if key in self._data.data_vars:
--> 289         raise KeyError(key)
    290     return self._data[key]

KeyError: 'XC'

Hey @ThomasHaine! Apologies, I thought I wrote but my message stayed as draft.
The issue with LLC4320 is a non-issue really. The Keyerror means that the variable XC (and YC) are not defined as coordinates for the xarray.dataset. This is fixed by setting them as coordinates, for example:
ds = ds.set_coords(['XC', 'YC'])

that does it. Otherwise xoak can't find them.
However, last week as I ran this code with LLC4320 on SciServer (Oceanography (Integrated Viewer)) and found that:

it remains pretty slow. Most of the time is on building the tree with xoak. I tried loading the variables XC, YC into memory (since these are 2D) but it didn't work. It has to do with building the tree everytime there is a query for coordinates. I need to work on perhaps reusing the build of the tree (via xoak) so that we don't continuously re-build it, which seems unnecessary.
Perhaps once Grendel is ready things will be faster and it will be a non-issue.

The plotting function has some bug or something. When I use the same coordinates from the ECCO examples but now on LLC4320, the plots are off, which is weird since both datasets have the same topology.

I need to spend some time on it. I am not too concern about 2), but 1) is a bummer (that is slow with LLC4320).  Again, perhaps with Grendel things will run much much faster. Right now I am using the basic Oceanography (Integrated Viewer) volume on Sciserver (so pretty basic).

Thanks @Mikejmnez. I find the same issues (1) and (2). Grendel is still available, so it may be possible to test now. Otherwise, the data transfer to the new ceph cluster will happen soon (I hope).
Is it possible to build the xoak tree once and store it?

Is it possible to build the xoak tree once and store it?

Not sure, but if it is that would significantly speed up the plotting in this PR, subsample.mooring_array and subsample.stations!! I need to spend some time exploring these options, and finding the bug on the plotting func in this PR.

2. The plotting function has some bug or something. When I use the same coordinates from the ECCO examples but now on LLC4320, the plots are off, which is weird since both datasets have the same topology.

Fixed this problem with 2) above. It was minor issue but now plotting works as expected with LLC4320, DYAMOND and all other LLC-geometries. You can find some examples with LLC4320 (and how long it took to run) on this notebook.
note

I had to download LLC4320 and DYAMOND data to my local computer in zarr format, to speed things up. The 2D Grid and single snapshot of ETA. And so the notebooks run locally to the data, which is as fast as it will get on a single (8core) computer.

I did some performance test of xoak on my local computer. Not bad: xoak builds in about 20 secs with LLC430 (4secs with DYAMOND) and so things are pretty fast when data is local to the compute. After than, selecting range of coordinates is pretty fast. I didn't test if storing the grid after xoak was built bypasses the need to build xoak within the grid dataset everytime. Will check that soon.

I need to do some more benchmark: preliminary data on this notebook.

Another thing that slows down things on LLC4320 is the amount of points to be plotted. 4320^2 per face, and when there are multiple faces involved, many more grid points. And so a lot of the time spent by the function was simply plotting (up to ~1 minute). And that is when data is local to compute notebook. On Sciserver (at least the interactive node) I would expect things to be much slower. But perhaps with Grendel (optimize transfer of data) things will be much faster.

TODO: Still need to write tests and documentation (docstring).

Thanks @Mikejmnez . I'll take a look and review, then you can merge.

Thanks @Mikejmnez . I'll take a look and review, then you can merge.

Great. I updated the example notebook at the top of this PR conversation so you can see examples with LLC4320. You can see the timings there when the grid dataset is stored in zarr in my machine, next to the notebook.
The ds_grid + xoak combination is being build everytime there is a plot, and so those timings can probably be improved a bit. The notebook can be considered the baseline when data is stored locally next to compute environment.

@ThomasHaine BTW you should re-install oceanspy from the branch, since the code has changed since last time you installed it. The instructions to do that are in the example notebook that is referenced at the beginning of the PR conversation.
NOTE: The notebook uses a yaml file that is also available from the GH repository, but that yaml file defines the dataset locally to the notebook. To run that notebook from within Sciserver, you should instead use oceanspy's default catalog:
cell [4]:  ECCOod = ospy.open_oceandataset.from_catalog("ECCO")

and
cell[6]: LLC4320od = ospy.open_oceandataset.from_catalog("LLC4320")

@Mikejmnez I checked the notebook with ECCO and LLC4320 on SciServer, and all works fine. Please proceed with the merge!


Work in progress

merging wip to main for the proof of concept

I think this is good. If there's no conflict it's good then it's good to be merged.


Top direction is not working as intended

I'm making a messaging app like whatsapp, I used infinite loading to load a conversation messages, when selecting a conversation it's supposed to view from the last message and scroll up to view more, to do this I used
<div class="d-flex flex-column-reverse">
  <!-- load the messages -->
</div>

this makes the scroll start from the bottom
if I put the infinite-loading component after the messages it keeps loading even without scrolling
if I put it at the top before the messages it will appear at the bottom and will also keeps loading
if I make a parent div and put the infinite-loading before the messages div as a sibling and put direction=top it works fine, but when selecting a conversation, the scroll bar starts from the top making the component load once, while it's supposed to start from the bottom and not load messages till I scroll up.

I think it can be solved by adding a new direction reverse-top for example, and add the infinite element under the list (it will appear at the top) and add the below to the getCurrentDistance() function
else if (this.direction === 'reverse-top') {
        const infiniteElmOffsetTopFromBottom = this.$el.getBoundingClientRect().top;
        const scrollElmOffsetTopFromBottom = this.scrollParent === window
          ? window.innerHeight
          : this.scrollParent.getBoundingClientRect().top;

        distance = scrollElmOffsetTopFromBottom - infiniteElmOffsetTopFromBottom;

what do you think ? @PeachScript


Support array operations in update

Now that we released the new update syntax, we can start supporting more operations. Here is the PostgreSQL list:
http://www.postgresql.org/docs/9.1/static/functions-array.html
@michalmuskala where is the list for MongoDB? I think we can at least support concat or push?

Here's a list: http://docs.mongodb.org/manual/reference/operator/update/
We could support mul for numeric fields (since Mongo 2.6), for arrays we could support append, prepend (>= 2.6) and concat. Please not that in Mongo those can only be used with constant values (no columns, fields, etc - but we're already dealing with that problem).

Sweet, I think we can support $push and $pull.

In master.


chore: update deps to include evaluate

Please do not create a pull request without creating an issue first.
Changes need to be discussed before proceeding, pull requests submitted without linked issues may be rejected.
Please provide enough information so that others can review your pull request. You can skip this if you're fixing a typo – it happens.

[ ] I have added tests to tests to cover my changes.
[ ] I have updated docs/, if necessary.
[ ] I have updated the README.md, if necessary.

What existing issue does this pull request close?
Put closes #issue-number in this pull request's description to auto-close the issue that this fixes.
How are these changes tested?
This pull request includes automated tests for the code it touches and those tests are described below. If no tests are included, reasons why must be provided below.
These changes are tested with [...]
Demonstration
Demonstrate your contribution.
For example, what are the exact commands you ran and their output, related screenshots, screen-recordings, test runs, anything that can showcase.
Provide additional context.
Provide as much relevant context as you like.

Codecov Report

Merging #593 (e33887e) into main (da9865a) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #593      +/-   ##
==========================================
- Coverage   84.87%   84.86%   -0.02%
==========================================
  Files         161      161
  Lines       12587    12585       -2
==========================================
- Hits        10683    10680       -3
- Misses       1904     1905       +1

see 3 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[SPARK-29950][k8s] Blacklist deleted executors in K8S with dynamic allocation.

The issue here is that when Spark is downscaling the application and deletes
a few pod requests that aren't needed anymore, it may actually race with the
K8S scheduler, who may be bringing up those executors. So they may have enough
time to connect back to the driver, register, to just be deleted soon after.
This wastes resources and causes misleading entries in the driver log.
The change (ab)uses the blacklisting mechanism to consider the deleted excess
pods as blacklisted, so that if they try to connect back, the driver will deny
it.
It also changes the executor registration slightly, since even with the above
change there were misleading logs. That was because the executor registration
message was an RPC that always succeeded (bar network issues), so the executor
would always try to send an unregistration message to the driver, which would
then log several messages about not knowing anything about the executor. The
change makes the registration RPC succeed or fail directly, instead of using
the separate failure message that would lead to this issue.
Note the last change required some changes in a standalone test suite related
to dynamic allocation, since it relied on the driver not throwing exceptions
when a duplicate executor registration happened.
Tested with existing unit tests, and with live cluster with dyn alloc on.

Test build #114038 has started for PR 26586 at commit a14c4ab.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/18899/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/18899/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18899/
Test FAILed.

Test build #114038 has finished for PR 26586 at commit a14c4ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114038/
Test PASSed.

test this please

Test build #114743 has started for PR 26586 at commit a14c4ab.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19565/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19565/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19565/
Test FAILed.

Test build #114743 has finished for PR 26586 at commit a14c4ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114743/
Test PASSed.

test this please

Test build #114817 has started for PR 26586 at commit a14c4ab.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19640/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19640/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19640/
Test PASSed.

Test build #114817 has finished for PR 26586 at commit a14c4ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114817/
Test FAILed.

test this please

Test build #115055 has started for PR 26586 at commit a14c4ab.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19873/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19873/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19873/
Test PASSed.

Test build #115055 has finished for PR 26586 at commit a14c4ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115055/
Test PASSed.

Test build #115125 has started for PR 26586 at commit 6c8d4cd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19938/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19938/
Test FAILed.

retest this please

Test build #115125 has finished for PR 26586 at commit 6c8d4cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115125/
Test PASSed.

Test build #115131 has started for PR 26586 at commit 6c8d4cd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19944/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19944/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19944/
Test FAILed.

Test build #115131 has finished for PR 26586 at commit 6c8d4cd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115131/
Test FAILed.

retest this please

Test build #115138 has started for PR 26586 at commit 6c8d4cd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19950/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19950/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19950/
Test FAILed.

Test build #115138 has finished for PR 26586 at commit 6c8d4cd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115138/
Test FAILed.

retest this please

Test build #115267 has started for PR 26586 at commit 6c8d4cd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20075/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20075/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20075/
Test FAILed.

Test build #115267 has finished for PR 26586 at commit 6c8d4cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115267/
Test PASSed.

K8s integration failure is consistent, but it seems to be irrelevant to this PR.
- Launcher client dependencies *** FAILED ***
  The code passed to eventually never returned normally. Attempted 29 times over 2.006781855433333 minutes.  (DepsTestsSuite.scala:213)

- Launcher client dependencies *** FAILED ***
  java.lang.AssertionError: assertion failed:

Yeah, but I'd be more comfortable if I saw that same failure in another PR, and I don't remember seeing it. (These integration tests are also becoming pretty hard to run locally with all the external dependencies...)

Ok, the following is a different PR and fails the same test:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20076/artifact/resource-managers/kubernetes/integration-tests/target/integration-tests.log
(Also seems to have run around the same time as the run that failed for this one.)

Test build #115582 has started for PR 26586 at commit e6f8814.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20382/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/20382/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20382/
Test FAILed.

Test build #115582 has finished for PR 26586 at commit e6f8814.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115582/
Test PASSed.


Add an example

Hello, i created a simple game using LSTMTimeStep.
Code: https://github.com/arifikhsan/batu-gunting-kertas-nuxt
URL: https://rockpaperscissors-ai.now.sh/
Can this game be included in the example? Thanks.

Added. Thanks

how about updating it to work in nuxt 3 ? this is awesome!!!


Switch between tests based on runtime class

Rather than using the RUNTIMECLASS parameter to modify the contained config, use it to switch between different test files. These tests will then use the appropriate runtime class. The runtime class is a good way to identify different tests because it includes the hypervisor and the hardware that are supported.
This also adds the SEV tests that @ryansavino is working on. See https://github.com/kata-containers/tests/pull/5130
Depends on https://github.com/confidential-containers/operator/pull/80 for  the runtime class.
At some point in the future we might want a more robust way to link test file names to rurntime classes. It's also a bit annoying that we have to specify the names of each of the tests in this repo. That will mean we need extra changes when we push new tests to the tests repo.
@wainersm

/test


embedded_io::ReadReady/WriteReady not implemented for uart peripheral

I am working on a project where I need to do some non-blocking reading from the uart peripheral based on the embedded_io traits. The regular Read trait specifies a blocking read such that it will wait for at least one byte to be available to be read. If one wants to do a non-blocking read, you need to use the ReadReady trait function read_ready to first check  if there is data available before doing the read.
However, only the Read trait is implemented (in #727 and #781 ) for the split and unsplit peripherals, but not the ReadReady trait. Is there a reason this has been left out?
It looks like the implementation could be quite simple, something like this for the Reader split peripheral (not tested):
impl<D: UartDevice, P: ValidUartPinout<D>> embedded_io::ReadReady for Reader<D, P> {
    fn read_ready(&mut self) -> Result<bool, Self::Error> {
        Ok(is_readable(self.device))
    }
}

It might be simply because no one needed it before. Feel free to open a PR :)


MediaTranscoding sample "Unknown failure"

The MediaTranscoding sample shows a red "unknown failure" error when I try to transcode to UHD 2160p60 on an Intel machine without Quick Sync Video and with an Nvidia GPU. UHD 2160p60 transcoding succeeds on Intel hardware with Quick Sync Video.
Repro steps:

Build the MediaTranscoding sample (I built the C# version).
Run the sample on a machine with an Intel CPU without Quick Sync Video. I used a Core i7-5960X with an Nvidia GeForce GTX 980 Ti.
Select scenario #2 (Transcode with custom settings).
Select any source video file.
Enter Width=3840, Height=2160, Frame Rate=60
Select an output file.
Click the Transcode button.

You'll get a red "unknown failure" error.
If you change the target frame rate from 60 to 30, this scenario will succeed.
If you run this scenario on a Surface Pro it will succeed at both 60 and 30 frames per second. I used a Surface Pro LTE from mid 2018 that has an Intel Core i5-7300U with Quick Sync Video.
Perhaps this is some kind of platform bug. Or perhaps it's some kind of Intel and/or Nvidia driver bug. For what it's worth, I installed the latest driver from Nvidia (445.87). I am running Windows 10.0.18363.815.

I captured a trace with WPR and it showed one thing that might be a clue:
EventName: Microsoft-Windows-Runtime-Media/WinRTTranscode_PrepareTranscode_Task/Stop
Payload: ErrorCode="-1,072,861,856" FormattedMessage="PrepareTranscode operation 0x29ed6283fe0{Object} ended. -1,072,861,856{ErrorCode} "

That error code happens to be MF_E_TRANSFORM_TYPE_NOT_SET. The Media Foundation doc page for the H.264 encoder says this:

The output type must be set before the input type. Until the output type is set, the encoder's IMFTransform::SetInputType method returns MF_E_TRANSFORM_TYPE_NOT_SET.

So maybe there's a bug in how the OS sets up the graph when transcoding to 59.94 fps.

I have filed a ticket against the feature team to investigate this.

@oldnewthing Hey, is there an update on this? The media composition API is incredibly useful. But this bug is making it impossible to create the video editing app that I am trying to work on. I really don't want to switch QT as I simple love the way UWP and MediaComposition API works.

Just to add to this, I am using a mp4 video exported from GeForce Experience. When I convert to 1080p 30fps mp4 using the media composition API, I get this weird washed out colours. You can see an example here:

I should mentioned that none of these issues exists (including the 60fps issue mentioned by @Zofware) if use VC-1/WMV as the output.

I sent a reminder to the feature team.

@tahnik Just a few days ago the team communicated a workaround to me for the unknown failure when transcoding to 2160p60 in the scenario I described in the original post. Fortunately the workaround is fairly simple.
The issue is that some video encoders (e.g. Nvidia) have to be configured with a slightly higher encoding "level" to be able to process high frame rate UHD 2160p video. This Wikipedia article explains the various encoding levels required for H.264 video. Windows doesn't automatically set the higher level for you so you have to set it yourself.
When you set up the video portion of the media encoding profile for high frame rate UHD 2160p H.264, you need to set MF_MT_VIDEO_LEVEL to eAVEncH264VLevel.eAVEncH264VLevel5_2. In scenario 2 of the C# version of the the sample, I believe you can do something like this on or around line 158 of Scenario2_Custom.xaml.cs:
// Workaround: set MF_MT_VIDEO_LEVEL to 52 (XAVC level 5.2 which supports 4k60)
_Profile.Video.Properties[MF_MT_VIDEO_LEVEL] = (UInt32)eAVEncH264VLevel.eAVEncH264VLevel5_2;

where MF_MT_VIDEO_LEVEL and eAVEncH264VLevel.eAVEncH264VLevel5_2 are declared to match the declarations from the Windows SDK. Like this:
// %ProgramFiles(x86)%\Windows Kits\10\Include\10.0.19041.0\um\mfapi.h
static readonly Guid MF_MT_VIDEO_LEVEL = new Guid("{96F66574-11C5-4015-8666-BFF516436DA7}");

// %ProgramFiles(x86)%\Windows Kits\10\Include\10.0.19041.0\um\codecapi.h
enum eAVEncH264VLevel : UInt32
{
    eAVEncH264VLevel1         = 10,
    eAVEncH264VLevel1_b       = 11,
    eAVEncH264VLevel1_1       = 11,
    eAVEncH264VLevel1_2       = 12,
    eAVEncH264VLevel1_3       = 13,
    eAVEncH264VLevel2         = 20,
    eAVEncH264VLevel2_1       = 21,
    eAVEncH264VLevel2_2       = 22,
    eAVEncH264VLevel3         = 30,
    eAVEncH264VLevel3_1       = 31,
    eAVEncH264VLevel3_2       = 32,
    eAVEncH264VLevel4         = 40,
    eAVEncH264VLevel4_1       = 41,
    eAVEncH264VLevel4_2       = 42,
    eAVEncH264VLevel5         = 50,
    eAVEncH264VLevel5_1       = 51,
    eAVEncH264VLevel5_2       = 52
};

[* Full disclosure: Since making the original post I started working at Microsoft but I am not commenting here in that capacity. ]

@Zofware You sir, are a wonderful person for taking your time to reply me. And yes, it worked! So I could not find the MF_MT_VIDEO_LEVEL and eAVEncH264VLevel.eAVEncH264VLevel5_2 in the UWP project that I am working on. I found the GUID for MF_MT_VIDEO_LEVEL from a dodgy CS file floating in the internet. Here is the line of code that I added if anyone is looking at this issue in future:
profile.Video.Properties[Guid.Parse("96f66574-11c5-4015-8666-bff516436da7")] = (UInt32)52;

However, the colours in the converted video is still washed out. Do you have any idea why that might happen?
Also, congratulations on getting a job at MS!

Never mind, I should probably create a different issue as that is not directly not related to this issue. I think it has some thing to do with the colour profile Nvidia is using when recording the screen. Thank you for your help.

Thanks @tahnik!
That's right, the GUID and the encoding levels aren't defined by default in the UWP C# environment. I pulled those definitions out of two of the C header files in the Windows SDK. The specific files are noted in the comments above.
Be aware there are encoding levels defined for HEVC/H.265 video that I didn't include above but they are in the SDK header files also. This is speculation, but I wonder if certain HEVC encoding settings require a level to be set? It's something to be aware of if you run into problems with HEVC in the future.
Cheers,
Adam
P.S. I don't know the cause of the washed out colors you're seeing but your theory sounds reasonable to me. Good luck!

I will work with @oldnewthing to add this workaround to the MediaTranscoding sample when I have some time.

Fixed in v8.0.12. Thanks, @Zofware

Thanks for seeing this through @oldnewthing!


Prevent Block ticking on edge chunks

Traditionally, in Vanilla Minecraft, all blocks of all loaded chunks will have a chance to tick. The unfortunate side effect of this is that there are corner cases where certain blocks (like liquids) will end up ticking and causing random chunk loads/generations because they are simply at the edge of the loaded chunk range.
In previous server mods, it was common practice to prevent the edge chunks from even considering to tick blocks and entities (CraftBukkit did this and to a greater extent, Spigot does this).
The question though is when in certain circumstances some mods (or to an extent redstone machines rely on this) may rely on neighbor chunk loads to successfully function.
So, I ask:
If there are some general arguments that will suffice to say that neighbor chunk loading during a block tick should still function and cause the ensuing prevention of edge chunk block ticking to be ignored OR just go ahead and mainline the prevention of edge chunk block ticking to be suppressed.

My biggest thing about Sponge and how its different than the other mod platforms is that we do NOT break Vanilla gameplay, whatsoever.
If this breaks Vanilla redstone gameplay then I think the path is clear.

That being said, that doesn't prevent a mixin config toggle (in the optimizations plugin) for those admins who simply don't care about that functionality working.

Couldn't this even be done on a per-world level? That would allow servers with a specific world for redstone to optimize the rest of the worlds.

This has been done in commit
https://github.com/SpongePowered/SpongeCommon/commit/a66367c4d9067b5fcc0c087139aa6e5dccf9376f
I'll handle any issues as they come up.


Form Pristine & Dirty values are wrong when matInput has placeholder attribute in IE 11

What is the expected behavior?
on loading page, form should have the following default value with matInput and placeholder attribute in IE11
prisitine - true
dirty - false
Note: Its working fine as expected without placeholder attribute in matInput.
What is the current behavior?
Onload it has below value in IE
pristine - false
dirty - true
which is different from Chrom and Firefox
What are the steps to reproduce?
https://stackblitz.com/edit/form-status
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Angular version: 6.01
Browser: IE 11
Language: TypeScript
Is there anything else we should know?
NIL

This looks to be the same issue as #14440. Closing since Material doesn't gave anything to do with when the form control is marked as dirty.

Try with adding placeholder like below


Feauture/moare additions corrections

Some added rules, some corrections.
Be extra careful when you go 'down' the tree, e.g. for schorsingsbesluit a GO has many EB's but the decision should only end up in one EB. We need more info on that; i.e. the forms will need this extra information.

merging so testing can test


Delete .DS_Store

remove file (sorry, OCD)

Thank you for submitting a pull request on this HMRC repository. A member of the HMRC team will review your changes and action accordingly.
HMRC reviewer: please reference this runbook for guidance on how to proceed.

Thank you for submitting a pull request on this HMRC repository. A member of the HMRC team will review your changes and action accordingly.
HMRC reviewer: please reference this runbook for guidance on how to proceed.

Thank you for submitting a pull request on this HMRC repository. A member of the HMRC team will review your changes and action accordingly.
HMRC reviewer: please reference this runbook for guidance on how to proceed.

thanks @jezd-axyl


Add disconnectpeer subcommand

Disconnect peer with disconnectpeer <pubkey>.
Checks all channels are closed with peer before disconnecting.

I saw that the command existed in LND to disconnect a peer but the sample node only has connectpeer and listpeers

Sorry for the delay, thanks for adding this!


Stream auto

Auto manage the stream
Start after x time
switch windows after x time etc

[x] UI
[x] Auto save config
[x] Event handler (read config, listen game events, setcurrent scene with timeout, start stream and stop stream)

#14

#14

#16


Crate generating code with bindgen failed to build

Crate name
libproc-rs
Build failure link
https://docs.rs/crate/libproc/0.14.9/builds/1380760
Additional details
Build script is here: https://github.com/andrewdavidmackenzie/libproc-rs/blob/master/build.rs
on macos the generated file is included by lib.rs https://github.com/andrewdavidmackenzie/libproc-rs/blob/master/src/lib.rs
with
#[cfg(target_os = "macos")]
#[allow(warnings, missing_docs)]
mod osx_libproc_bindings {
    include!(concat!(env!("OUT_DIR"), "/osx_libproc_bindings.rs"));
}

and that fails to build with:
[INFO] [stderr]  Documenting libproc v0.14.9 (/opt/rustwide/workdir)
[INFO] [stderr] error: couldn't read /opt/rustwide/target/aarch64-apple-darwin/debug/build/libproc-305d8c67d250c2df/out/osx_libproc_bindings.rs: No such file or directory (os error 2)
[INFO] [stderr]   --> src/lib.rs:67:5
[INFO] [stderr]    |
[INFO] [stderr] 67 |     include!(concat!(env!("OUT_DIR"), "/osx_libproc_bindings.rs"));
[INFO] [stderr]    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[INFO] [stderr]    |
[INFO] [stderr]    = note: this error originates in the macro `include` (in Nightly builds, run with -Z macro-backtrace for more

Is there any way I can fix this?

I'm not 100% sure what the problem is, but let's walk through it.
More information about the build environment: https://docs.rs/about/builds
docs.rs is using cross compilation to build docs for non linux targets, so xcode headers won't exist.
My hunch would be that this is the reason for this error, though I'm not sure why you are not seeing an error from the build script.
So might try running the build on a linux host, cross compiling to mac.

So, maybe I could detect docs.rs in build.rs and use vendored headers
instead of the generated ones....?
Would want to never actually compile with possibly out of date headers....
On Sun, Oct 6, 2024, 2:55 PM Denis Cornehl @.***> wrote:

Uploading your own docs is currently not possible due to several reasons
(see #93 https://github.com/rust-lang/docs.rs/issues/93 ).
The typical solutions for this are:

vendor the necessary headers
use conditional compilation to have the docs build without the
headers or library

the latter assumes that very often the generated bindgen code isn't part
of the public interface of the library, so is not part of the documentation.
—
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/docs.rs/issues/2627#issuecomment-2395430499,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKF4LBZ3A7MKM5762MXRI3Z2EXNRAVCNFSM6AAAAABPNJ4B5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGQZTANBZHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

So, maybe I could detect docs.rs in build.rs and use vendored headers instead of the generated ones....? Would want to never actually compile with possibly out of date headers....

That, or don't include any headers on docs.rs, don't generate any bindings,
and use conditional compilation (#[cfg(docsrs)]) to make the code compile.
( assuming the generated bindings are not part of the public interface of your library)

Is there anyway to test the build, without me having to do a release
each time?
On Mon, Oct 7, 2024, 7:47 AM Denis Cornehl @.***> wrote:

So, maybe I could detect docs.rs in build.rs and use vendored headers
instead of the generated ones....? Would want to never actually compile
with possibly out of date headers....
That, or don't include any headers on docs.rs, don't generate any
bindings,
and use conditional compilation (#[cfg(docsrs)]) to make the code compile.
( assuming the generated bindings are not part of the public interface of
your library)
—
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/docs.rs/issues/2627#issuecomment-2395959911,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKF4LDF5IYOFATCJKJGCCDZ2IN7XAVCNFSM6AAAAABPNJ4B5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVHE2TSOJRGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

https://docs.rs/about/builds#testing-builds-locally
https://github.com/rust-lang/docs.rs/blob/master/README.md#build-subcommand

( though to set it up is a little time intensive, and I'm not 100% certain if the documentation is 100% up-to-date)

I have fixed this using a pre-generated version of the macos bindings from a manual run of bindgen when building on docs.rs, otherwise I use the generated version as before, using #[cfg(docsrs)] in build.rs
The result can be seen here: https://docs.rs/crate/libproc/latest
and now I have the two platforms available in the drop-down.
Thanks for your help.


librespeed-cli: init at 1.0.7

Motivation for this change
This is a command line interface for LibreSpeed speed test backends, written in Go.
https://github.com/librespeed/speedtest-cli
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 116986 run on x86_64-linux 1

1 package built:

librespeed-cli


Fix#153

fixes: #153

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

5 of 12 (41.67%)  changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.007%) to 65.065%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

Pharo/PharoJs-Base-JS-CoreLibraries/PjObject.class.st
0
7
0.0%

Files with Coverage Reduction
New Missed Lines
%

Pharo/PharoJs-Base-JS-CoreLibraries/PjObject.class.st
1
14.8%

Totals

Change from base Build<PHONE_NUMBER>:
-0.007%

Covered Lines:
13350

Relevant Lines:
20518

💛  - Coveralls

Enhancement introduced in a different commit


otpGraphQL: supportedTransitModes not documented

I found that in https://github.com/public-transport/transport-apis/blob/236d88d00f3bbdfaf8b623a8556b9fae6a22cf61/data/de/ulm-otp.json#L19 and https://github.com/public-transport/transport-apis/blob/236d88d00f3bbdfaf8b623a8556b9fae6a22cf61/data/de/stadtnavi-otp.json#L19 an option called "supportedTransitModes" is used. However it is not mentioned in the protocol specific options: https://github.com/public-transport/transport-apis#open-trip-planner-with-graphql
So imho it would make sense to add it there and maybe link to the possible values (e.g. this https://docs.opentripplanner.org/en/dev-2.x/RoutingModes/ if I'm not mistaken).
If you agree I could make a little PR adding the documentation

I agree that it can be removed as a breaking change.


crypto/tls: degraded performance from 1.8 to 1.9 version

What version of Go are you using (go version)?
go version go1.9 linux/amd64
go version go1.10 linux/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build843745536=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I use a very simple program created in December 2016 which use aws-go-sdk to get a metric on ELB.
It has been compile with go 1.8 and there is no problem until now.
I had to re-compiled it recently (with go 1.9) to add changes and I noticed a degradation of the performance.
Since I also tested go 1.10 and there I noticed the same behavior as for 1.9 version.
The code is exactly the same but 1.9/1.10 binary version execution time is more than 2x greater than with the 1.7/1.8 binary version.
package main

import (
	"fmt"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/cloudwatch"
	"os"
	"time"
)

func main() {

	region := os.Args[1]
	elb_name := os.Args[2]
	metric_name := os.Args[3]
	metric_stat := "Sum"

	if len(os.Args) > 4 {
		metric_stat = os.Args[4]
	}
	svc := cloudwatch.New(session.New(), &aws.Config{Region: aws.String(region)})
	dur, _ := time.ParseDuration("300s")
	params := &cloudwatch.GetMetricStatisticsInput{
		StartTime:  aws.Time(time.Now().Add(-dur)), // Required
		EndTime:    aws.Time(time.Now()),           // Required
		MetricName: aws.String(metric_name),        // Required
		Namespace:  aws.String("AWS/ELB"),          // Required
		Period:     aws.Int64(300),                 // Required
		Statistics: []*string{ // Required
			aws.String(metric_stat), // Required
		},
		Dimensions: []*cloudwatch.Dimension{
			{ // Required
				Name:  aws.String("LoadBalancerName"), // Required
				Value: aws.String(elb_name),           // Required
			},
		},
		//Unit: aws.String("Count"),
	}
	resp, err := svc.GetMetricStatistics(params)

	if err != nil {
		panic(err)
	}

	fmt.Println(resp)
}

i.e. result of a test run for each version :
for 1.8.7 version :

time ./elb-check.go187 eu-west-1 test-elb Latency Average
{
Datapoints: [{
Average: 0.15102177378775059,
Timestamp: 2018-02-20 11:01:00 +0000 UTC,
Unit: "Seconds"
}],
Label: "Latency"
}

real 0m0.057s
user 0m0.028s
sys 0m0.008s

for 1.10 version :

time ./elb-check.go19 eu-west-1 test-elb Latency Average
{
Datapoints: [{
Average: 0.15102177378775059,
Timestamp: 2018-02-20 11:01:00 +0000 UTC,
Unit: "Seconds"
}],
Label: "Latency"
}

real 0m0.123s
user 0m0.104s
sys 0m0.000s

I suspect it is linked to the ssl request made to cloudwatch and so the issue #23727
Here is a pprof cpu pdf for 1.8 version :
out-go18.pdf
And the same for 1.10 version :
out-go110.pdf
What did you expect to see?
Almost the same performance / time execution.
What did you see instead?
An increase of the time execution and the resource consumption
Thanks

@xp-1000 - Did you get a chance to check this with 1.11.2 ?

Hello @agnivade,
I just tested :
time ./zabbix-aws-cloudwatch-1.8.7 -region=eu-west-1 -metric=HealthyHostCount -namespace=AWS/ELB -dimensions='Name=LoadBalancerName,Value=elb-test' -stat=Minimum -no-data-value=0
1

real    0m0.051s
user    0m0.040s
sys     0m0.004s

time ./zabbix-aws-cloudwatch-1.11.2 -region=eu-west-1 -metric=HealthyHostCount -namespace=AWS/ELB -dimensions='Name=LoadBalancerName,Value=elb-test' -stat=Minimum -no-data-value=0
1

real    0m0.128s
user    0m0.108s
sys     0m0.008s

it seems not better.
I confirm the code is the same, only golang version changes.

AFAIU, your code makes a network request to cloudwatch and gets some results. This is very much dependent on your network speed. If you say that the problem is with crypto/tls in general, then any tls connection should show the same delay. Ideally, we would want the network part out of the equation by setting up a local server. Your current repro, as it stands is a very generalised one without pointing towards any particular part of crypto/tls.
I will leave it for others to investigate.

@xp-1000 - Just wanted to check in on this. If you can shorten the code to something that does not need to have an aws cloudwatch setup, it will help us investigate this faster.


Expose path of currently executing view

The path of the currently executing view would be extremely helpful when tracing information RE view rendering, e.g. from Tag Helpers.
It will likely not work on ViewContext as that flows down to things like partial views. Perhaps a new type is needed, e.g. ViewExecutionContext, on which we make it available. We could investigate making the entire tree of view execution available, e.g. view -> layout -> partial view -> etc.

We should consider doing something special for templates (as templates are similar to partial views, but they have a different context), so perhaps some indication of the type of view that the code is executing in (enum? or set of well known strings?)


Problem when get_readonly_fields() is used

django-admin-view-permission is really cool :-)
Something is strange: with read_only -> OK but with get_readonly_fields() -> KO
Here is the detail. With just a view permission on a Model.
(1) when I use the attribute 'readonly_fields' as below
class MyModelAdmin(model.Admin):
    ...
    readonly_fields = (a,b)

The result in the admin "change view"  : all the attributes are displayed in readonly (expected).
(2)  when I use instead get_readonly_fields(self, request, obj=None)
class MyModelAdmin(model.Admin):
    ...
    def get_readonly_fields(self, request, obj=None):
        return (a,b)

--> all the fields that are in this list (here a,b) are readonly (as expected), but all other fields are read/write (this is unexpected). The rest is ok, no button save, etc.
I'm missing something ?
Thks

Hi @escribis,
Instead of returning directly the tuple of the fields, you could do something like this:
def get_readonly_fields(self, request, obj=None):
      # the following command will calculate the readonly_fieds based on the admin_view_permission
      # package (eg: if the user has view permission it will return all the fields otherwise it will return
      # those fields based on the django `ModelAdmin` or you custom class)
      readonly_fields = super(MyModelAdmin, self).get_readonly_fields(request, obj)

      # Here you can put your code and change the readonly_fields based on extra logic or extra permissions

      return readonly_fields

Thanks,
Lefteris

Thnks ! Works like a charm :+1:


Fix fileheader analyzer issues

Fixes #1356
Fixes #1432

:question: Can you squash these two commits, since they really combine to form the work here?

Updated and squashed