How to correctly modify OpenWhisk’s gateway blocking limit from source code (currently 60s)

Because our project requires a large number of HTTP request tests and our functions (modern machine learning workflows) have relatively long execution times, our team believes that it is necessary to compile the Docker image of OpenWhisk from the source code and modify the 1min gateway blocking limit.
We have made a number of source code modifications for this purpose, as follows:

In the source code part of the controller, we modify application.conf, set request-timeout = 36000s

In the source code part of the controller, we also modified reference.conf:

In the common Scala module, we also modified application.conf:

After completing the above three modifications, we used ./gradlew :core:controller:distDocker to compile the Docker image and replaced the image used by our K8s OpenWhisk cluster. We modified value.yaml .

We ensure that the content written in values.yaml meets the requirements:

We also made some modifications to nginx, including the default.conf inside the nginx:1.21.1 image, and the read-only nginx.conf configured using nginx-configmap:

However, when we used curl to test, we found that it did not seem to work completely. The blocking of the nginx gateway was overcome, but the following information was printed:

So, what modifications do we need to make to overcome this problem? Looking forward to your guidance!

@style95 Could you please give me some guidance?

It seems you are using the API gateway, could you check again without the API gateway first?

It seems you are using the API gateway, could you check again without the API gateway first?

You are supposed to be able to invoke the action with wsk. I think that's the starting point to look into.

I'm sure that my gateway can correctly ensure that the end-to-end response is greater than 60 seconds. I have also explicitly configured the controller. I can't seem to find any more places where I need to configure the timeout. Can you give me some suggestions?

wsk does work, but only the information recorded by the activation can be viewed. We prefer to complete our services directly through the gateway HTTP request.

At the same time, how should I correctly configure the execution time limit? What I wrote in value.yaml is 500 minutes, but when I use wsk to set the upper limit of execution time of 500 minutes, the invoker log prints 600s?

@QWQyyy
First, IIRC, the timeout of the Kubernetes client in the above log is related to the pod creation.
It's not related to the execution of an activation.
The action timeout controls the execution timeout against the pod(container).
https://github.com/apache/openwhisk/blob/5529cc49d31f135dfdac4f2a2072ca46bfd754de/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerProxy.scala#L834
I think you need to ensure you can invoke your action successfully with the wsk action invoke command.
If you can successfully invoke your action without the API gateway, then the culprit is the API gateway.
If your action is invoked well but it is changed to the asynchronous(get 202 response) at some point, it's related to the controller configuration. If you can't even invoke your activation in the asynchronous mode as well, you may not configure the action timeout properly.

@QWQyyy First, IIRC, the timeout of the Kubernetes client in the above log is related to the pod creation. It's not related to the execution of an activation. The action timeout controls the execution timeout against the pod(container).
https://github.com/apache/openwhisk/blob/5529cc49d31f135dfdac4f2a2072ca46bfd754de/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerProxy.scala#L834
I think you need to ensure you can invoke your action successfully with the wsk action invoke command. If you can successfully invoke your action without the API gateway, then the culprit is the API gateway. If your action is invoked well but it is changed to the asynchronous(get 202 response) at some point, it's related to the controller configuration. If you can't even invoke your activation in the asynchronous mode as well, you may not configure the action timeout properly.

Okay let's try it!

@QWQyyy First, IIRC, the timeout of the Kubernetes client in the above log is related to the pod creation. It's not related to the execution of an activation. The action timeout controls the execution timeout against the pod(container).
https://github.com/apache/openwhisk/blob/5529cc49d31f135dfdac4f2a2072ca46bfd754de/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerProxy.scala#L834
I think you need to ensure you can invoke your action successfully with the wsk action invoke command. If you can successfully invoke your action without the API gateway, then the culprit is the API gateway. If your action is invoked well but it is changed to the asynchronous(get 202 response) at some point, it's related to the controller configuration. If you can't even invoke your activation in the asynchronous mode as well, you may not configure the action timeout properly.

It is true that I can successfully access my functions using wsk, but it is only limited to functions within 60 seconds. For functions that take longer to execute, the wsk client also returns an Oops--504 error, but I found through the resource manager htop that the function code is still During execution, after the function is executed, you can see from wsk activation that the function ends correctly. This confuses me.


Bazel for Eta

Description

It would be great to have Bazel support for Eta and will help Eta's adoption in large startups and companies who tend to follow monorepo model.
This issue is to track Bazel support for Eta and to discuss how to overcome any issues.
Tagging @jin and @johnynek who have both shown interest in writing the bazel rules.
A starting point of the discussion is: https://github.com/typelead/eta/issues/853

@jin do you want to make a google doc and/or publish your WIP code?
I have followed rules_haskell fairly closely. There it seemed to require quite a bit of knowledge about how ghc-pkg worked in order to implement the rules. They didn't just call ghc.
To me, ideally, eta would have enough CLI flags that we can use it directly, since having etlas and bazel both in play make it complex to deal with the fact that etlas expects to own state directories, but bazel also expects to own all the state.
In addition to hopefully only using eta, it would be great if we can bootstrap the eta from source, or in the worst case only fetch jars and run eta on the jvm.
finally, it would be great to look at: https://github.com/FormationAI/hazel to see if we can use that to generate bazel dependencies for haskell packages that we can build with eta.
A doc, with some design goals would be great to help us split this work up.
I think a good v0 would be the ability to compile eta code locally and depend on other bazel java rules (java, scala) to use java and scala libraries.
a v1 should be able to consume haskell/eta libraries.

@johnynek you can substitute ghc-pkg with eta-pkg
For Nix:

I configure the package.conf.d with all the Eta dependencies
Eta and Etlas do create files under $HOME - I made Nix set HOME to a specific directory during builds
I do use Etlas, because that uses the project's build information
I put a a script called coursier on the PATH which just lists the Maven jars

I don't know Bazel pretty much at all but it sounds like the same strategy might work.
https://github.com/eta-lang/eta-nix/blob/etaPackages/eta-modules/generic-builder.nix

@johnynek I got java_binary.deps <- eta_library to work. PTAL: https://github.com/jin/rules_eta

I've proposed a bunch features in the README as a start It also licontainsts the Java/Eta sandwich.

In addition to hopefully only using eta, it would be great if we can bootstrap the eta from source, or in the worst case only fetch jars and run eta on the jvm.

It took me over 15 minutes to compile eta on a Macbook Pro, so I don't recommend this. It'll be great if we have a direct distribution of the eta binary that we can pull with a repository_rule.

Eta and Etlas do create files under $HOME - I made Nix set HOME to a specific directory during builds

I believe this is why I had to pass the Bazel flag --action_env=HOME to make it work. If not, I get this error:
$ bazel build //examples/basic:main
..
ERROR: Compiling main-eta using Eta failed (Exit 1)
eta: eta: panic! (the 'impossible' happened)
  (Eta version 0.8.6b2):
	v_unsafeGlobalDynFlags: not initialised

Please report this as a Eta bug: http://github.com/typelead/eta/issues

Target //examples/basic:main failed to build
..

@Jyothsnasrinivas can eta be made to not depend on the $HOME env var?

@jin Great work on getting a small example working already!

It took me over 15 minutes to compile eta on a Macbook Pro, so I don't recommend this. It'll be great if we have a direct distribution of the eta binary that we can pull with a repository_rule.

Here are the links to the binaries:

Linux
Mac OS
Windows

You can substitute the last part of the links with eta-pkg and eta-serv.jar for the respective binaries.
Please note that Windows is an exception and non-JAR binaries have a .exe suffix.

@Jyothsnasrinivas can eta be made to not depend on the $HOME env var?

Yes that can be done. We can add another flag to configure the .eta directory.

@jin Just added -homedir flag. If you run eta -homedir somdir Main.hs it will expect that somedir/package.conf.d/ is a valid package database that has all the boot-libraries.
We distribute binary versions of all the boot packages in every release. And they are downloaded and installed using Etlas. You are probably better off using Etlas instead of Eta because you'll end up duplicating the same work Etlas is doing.

Thanks! That makes it significantly simpler as we can use Bazel's non-hermetic repository rules to bootstrap the required directories within Bazel's execution root for Eta to function, and pass those locations into the rules.
I haven't really looked deeply at how Etlas work, but from my initial experience I'm echoing @johnynek's thoughts:

To me, ideally, eta would have enough CLI flags that we can use it directly, since having etlas and bazel both in play make it complex to deal with the fact that etlas expects to own state directories, but bazel also expects to own all the state.

So far, I've been trying to implement the simpler logic with primitive calls as much as possible without using a more powerful tool like Etlas. It helps to keep things simple and we can easily track the granularity of decomposed actions using eta_library targets. From my experience so far, I think Etlas will be used to manage the dependencies during the (pre-)loading phase, and Eta for the actual granular compilation.

Sure that sounds good!


K8SPSMDB-774 describe updating certs

Should we add as example
spec.secrets.ssl
spec.secrets.sslInternal to deploy/cr.yaml file.?


mac os error

i get this error with my mac

Seonghoonui-MacBook-Pro:peatio seonghoonju$ gem install mini_racer -v '0.2.0'
Building native extensions. This could take a while...
ERROR:  Error installing mini_racer:
ERROR: Failed to build gem native extension.
current directory: /Users/seonghoonju/.rvm/gems/ruby-2.5.0/gems/mini_racer-0.2.0/ext/mini_racer_extension

/Users/seonghoonju/.rvm/rubies/ruby-2.5.0/bin/ruby -r ./siteconf20181017-11290-1pq5qc0.rb extconf.rb
checking for -lpthread... yes
checking for -lobjc... yes
creating Makefile
current directory: /Users/seonghoonju/.rvm/gems/ruby-2.5.0/gems/mini_racer-0.2.0/ext/mini_racer_extension
make "DESTDIR=" clean
current directory: /Users/seonghoonju/.rvm/gems/ruby-2.5.0/gems/mini_racer-0.2.0/ext/mini_racer_extension
make "DESTDIR="
compiling mini_racer_extension.cc
clang: warning: argument unused during compilation: '-rdynamic' [-Wunused-command-line-argument]
In file included from mini_racer_extension.cc:2:
In file included from /Users/seonghoonju/.rvm/rubies/ruby-2.5.0/include/ruby-2.5.0/ruby.h:33:
In file included from /Users/seonghoonju/.rvm/rubies/ruby-2.5.0/include/ruby-2.5.0/ruby/ruby.h:2036:
/Users/seonghoonju/.rvm/rubies/ruby-2.5.0/include/ruby-2.5.0/ruby/intern.h:47:19: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
void rb_mem_clear(register VALUE*, register long);
^~~~~~~~~
/Users/seonghoonju/.rvm/rubies/ruby-2.5.0/include/ruby-2.5.0/ruby/intern.h:47:36: warning: 'register' storage class specifier is deprecated and incompatible with C++17 [-Wdeprecated-register]
void rb_mem_clear(register VALUE*, register long);
^~~~~~~~~
2 warnings generated.
linking shared-object mini_racer_extension.bundle
clang: warning: libstdc++ is deprecated; move to libc++ [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mini_racer_extension.bundle] Error 1
make failed, exit code 2
Gem files will remain installed in /Users/seonghoonju/.rvm/gems/ruby-2.5.0/gems/mini_racer-0.2.0 for inspection.
Results logged to /Users/seonghoonju/.rvm/gems/ruby-2.5.0/extensions/x86_64-darwin-18/2.5.0/mini_racer-0.2.0/gem_make.out

from xcode 10

Building with libstdc++ was deprecated with Xcode 8 and is not supported in Xcode 10 when targeting iOS. C++ projects must now migrate to libc++ and are recommended to set a deployment target of macOS 10.9 or later, or iOS 7 or later. Besides changing the C++ Standard Library build setting, developers should audit hard-coded linker flags and target dependencies to remove references to libstdc++ (including -lstdc++, -lstdc++.6.0.9, libstdc++.6.0.9.tbd, and libstdc++.6.0.9.dylib). Project dependencies such as static archives that were built against libstdc++ will also need to be rebuilt against libc++. (40885260)

https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes
seems like you guys need to update

Well... We did -- v0.2.3 addresses this.

Oh!!! thx didnt know... :) 👍


IP Camera works with ONVIF in Zoneminder not in Python-Onvif library

From #4:

continuous_move.py is just an EXAMPLE. It works well ONLY with MY CAMERA. you should use onvif-cli or ONVFIService of this lib to test your camera.

Might i suggest adding those file names to a .gitignore so that we (the public) do not get your private test scripts or as a much better alternative set up a generic framework that can be setup for a broader spectrum of ONVIF compatible devices.
Traceback (most recent call last):
  File "./probe.py", line 89, in <module>
    mycam = ONVIFCamera('<IP_ADDRESS>', 80, 'admin', 'admin', '/etc/onvif/wsdl/')
  File "/usr/local/lib/python2.7/dist-packages/onvif/client.py", line 227, in __init__
    self.update_xaddrs()
  File "/usr/local/lib/python2.7/dist-packages/onvif/client.py", line 237, in update_xaddrs
    services = self.devicemgmt.GetServices({'IncludeCapability': False})
  File "/usr/local/lib/python2.7/dist-packages/onvif/client.py", line 31, in wrapped
    raise ONVIFError(err)
onvif.exceptions.ONVIFError: Unknown error: <urlopen error timed out>

as it stands im not sure if this is a problem with my python programming or if this is a problem with the library. My OS is Kali (Debian based 4.9.0-kali4-amd64 #1 SMP Debian 4.9.30-1kali1 (2017-06-06) x86_64 GNU/Linux) after running the install using pip install onvif I fired up the python console and imported the library. however once running the Example on the README.md provided with the project it failed telling me that several WSDL documents were in fact missing. I found them all in the repository under wsdl and copied them to etc. now it seems to load up an instance fine however i still cannot interface with the camera. I have over 150 cameras that i would like to manage all were purchased at the same time from the same manufacture (Zhejiang Dahua Technology Co., Ltd.) I'm running Zoneminder for CCTV recording and picks up ONVIF compatible devices i would rather not have to walk through every camera with the zoneminder install wizard. Providing i can communicate with the camera using this library i should be able to obtain all of the needed information and build the Zoneminder database with Python. Once the MySQL database has been built with the camera information i also have to change Static IP Address assignments from the Camera Web-Interface as well as System Clock settings and administrative logon. Asking to do this manually on that number of devices is a big task i would like to solve this using a python application to automate the mind numbing task.

I just tried this lib because my zoneminder did not want to control my camera. As fas as the WSDL i agree. The lib should use the shipped WSDL as default unless override by the user.
As for your use-case. I would configure one camera in Zoneminder using the interface than copy the settings from mysql. Given all the camera's are the same. All the settings except for IP:port user/pass (PTZ Control Address) are the same. So copy them in mysql wouldn't be to hard.

After inventing a creative solution I have fixed all of my issues and settled on a design for the program. Copy paste using MySQL is still a lot of work by hand. So I will be using the db connector to change all those settings. For the ONVIF side of things I've also engineered a way to combine your library to create a CLI version of the ONVIF device manager with added feature of copping settings between devices as long as the settings are non conflicting I.E. static IP addresses.

For reference: what onvif version of zoneminder did u use? As the one called "Onvif" is no onvif at all.

Version number for zoneminder is v1.30.4 installed using the ppa:iconner/zoneminder

Correction: ppa:iconnor/zoneminder. Apologies on spelling phone keyboard and all that :)

What control script did u use in zoneminder


New project calculator

Describe the project you want to add with tech stack
A calculator is made with html, css and js
Expected behavior

It's a basic calculator which evalutes mathematical expressions
Screenshots (optional)

Additional context (optional )

GSSoC'23
Assign yhis issue to me so i can work on it

I know flutter so I can make a Calculator app.
can you please assign me this issue under GSSoc 2023. I really want to work on this issue.

I know html, css, and javascript. I know how to create it.
Please assign this issue to me so that i can work on it.

@Saksham0205 if you want to upload your project of flutter than try to open new issue and get it assigned so you can make pr !!

I want to work on this issue under the GSSOC '23. Please Assign this issue to me.

Project already present


Add GPUDatasetIterator to daskdataset access via to_iter function

rerun tests

Thanks for attacking this Julio.  I have some thoughts here that I'd like to get your perspective on (you/others may disagree)...

We should probaby use this oportunity to move the DaskDataset code into nvtabular/io.py (and rename it to Dataset or NVTDataset). I think the two general goals here should be (1) to merge the dask/iterator APIs into one, and (2) to trim redundant code wherever possible.
We may need to add (something like) a to_iterator method to Dataset, and avoid turning the class itself into an iterator.  I say this because I feel the current purpose of DaskDataset is to process a list of file paths and produce a dask.DataFrame (for a specific column selection) when needed by the undeerlying nvt workflow.  It seems natural to extend this to also produce an iterator (for a specific column selection) when needed by the undeerlying nvt workflow.  It is true that the column selection is more imortant for dask-based statistics gathering, but I still worry that it will become confusing to iterate on the Dataset object itself.

Any thoughts here @benfred ?

@rjzamora I agree with both of your points - I think we should move the dask dataset code into nvtabular/io.py, and have an explicit 'iter' call.
I also we should consolidate the 'engine' code: we have a ParquetDatasetEngine and a CSVDatasetEngine in the dask/io code. Likewise we have a '_get_read_engine' in the nvtabular/io code that switches between the parquet/csv iterator implementations.
Instead, I think we should have an 'iter' method on the ParquetDatasetEngine class which calls ParquetIterator for each file, and likewise a iter method  on CSVDatasetEngine which calls the CSVIterator (rather than have the base class call GPUDatasetIterator). The GPUDatasetIterator class can be deleted after this change.

rerun tests

@benfred - Any other requirements for this step, or can we merge?

LGTM - thanks!


tools: fix coverage.sh

Fix missing double quote
Use POSIX sh instead of bash

@thaidn can you review this?


fix typo in content collections example

What kind of changes does this PR include?

Minor content fixes (broken links, typos, etc.)

Description
Fixes typo in a content collections example.

Thanks for this, great catch @fflaten!  🙌