User application testing facility

Does aiohttp provide functionality to test application written with its HTTP client? It is in the sense of packages like httpretty, responses and this SO question. Basically it may be some mock connector to say here's the next response your should return on any request. Or it may be more elaborate with separate queues for HTTP methods, base URLs, et cetera. It's useful for integration and system testing. Workarounds are possible, but require to construct complete server to imitate the target and to patch the application's URLs, which can take some substantial effort.
Even though it's not in my direct interest, the same thing may also apply to HTTP server. Some sort of base test case class that provides means to start user's app, do requests and make assertions about them with ease. Here is how CherryPy does it.
I've looked at the test suite and aiohttp.test_utils but there seems to be nothing what I'm looking for exactly. If I have not overlooked it and there's no such functionality at the moment, this can be treated as a feature request.

I believe all mentioned projects are exist because it's hard to write test server in classical blocking way.
You should execute it in separate thread and synchronize testing code with this server.
However in asyncio world co-existing of client and server API's in the same test case is pretty easy.
We do use it in test suite for aiohttp, sure.
Please take a look on our typical test file: https://github.com/KeepSafe/aiohttp/blob/master/tests/test_client_functional_newstyle.py

I don't think that internal complexity really matters for an end user. She has certain use cases and there are common approaches to solve them without too much of boilerplate. Your answer, in other words, is DIY, which is actually an answer. Anyway, I want to point to some things.
Client testing
Specifically it is a white-box testing. A system written with aiohttp has endpoints (URLs) to which it performs requests during its normal operation. Unless you patch all the endpoints to localhost or if the design allows application-wide configuration, you can't test it. And ease is of composition isn't relevant here. Fake connector, which you can pass a number of pre-fabricated responses, is.
Server testing
The threaded server, CherryPy, didn't convince you. Here's my next attempt: http://tornado.readthedocs.org/en/latest/testing.html. You can see the same unittest API no mater if the things is blocking IO or not.

Well, I see your point.
For client testing fake connector that doesn't resolve host but connects to specific local port would be helpful.
Also we can introduce a TestCase good for aiohttp usage.
Currently we duplicate setUp/tearDown procedure in every our test file as well as find_unused_port and graceful server shutdown procedure.
Personally I don't feel the strong request for those changes but I'm open for Pull Request.

massive +1 for docs and/or helper methods for unit testing a server app.
In my few days starting to use aiohttp this has been the biggest stubbing block and the closest I've come to saying "aiohttp is not ready, lets go back to django/flask".
It's really important there's an easy and powerful way to make fake http requests, where one can:

interrogate the response
get proper stacktraces from server errors
mock bits of the server on a per test basis.
setup and teardown databases on the server between tests.

I've made a start on doing this myself by copying bits of aiohttp's own tests but it's not obvious and there ends up being quite a lot of code dedicated to setting up the tests which I would expect to normally like in the test framework. (eg. most of conftest.py).
If the expected approach at present is to use aiohttp's own tests as an example some more comments particularly in conftest.py would be great, eg. docstrings in methods like pytest_pycollect_makeitem the purpose of which is not immediately clear.

is it possible to use pytest-asyncio to replace some of the setup in conftest.py?

Saying aiohttp is not ready, lets go back to django/flask sounds like you only look in the spotlight, without really caring about practical qualities of the libraries/frameworks. Non-blocking IO doesn't come for free. It's good for one things, it's bad for others. If you're really looking for a non-blocking library/framework it's quite more reasonable to go back Tornado. It's the "now". aiohttp is more the "future".

aiohttp tests was started from classic unittest style.
We used it for about 2 years but I really don't very happy with this approach.
Then we tried to make pytest.
Looks much better. But before making our test suite tools public (e.g. document it) we need to collect better experience in the area.
Perhaps instead of trying to publish code is better to add documentation chapter about writing tests in both unittest and pytest ways.
pytest-asyncio is interesting project but we have a bit different views on how globallness of the loop. Again I don't want to contribute into pytest-asyncio until get satisfactory common solution for aiohttp.
Anyway, reading aiohttp test suite code is not so hard, isn't it?

We at Center for Open Science developed a "port" of httpretty for aiohttp: https://github.com/CenterForOpenScience/aiohttpretty . It's in its early days, but it's certainly come in handy for aiohttp client testing.

I have created a testing pattern that seems to work (based on very limited testing). Not sure if this is a very sane way to do it, but using this technique the tests and event loop are all running in the same main thread, so it should allow you to properly mock things.  I've included a full working example inline below.
import socket
import asyncio
import threading
from unittest import TestCase

import aiohttp
from aiohttp import web
from aiohttp.hdrs import METH_GET, METH_POST

async def handle(request):
    text = "Hello world"
    return web.Response(body=text.encode('utf-8'))

async def handle_post(request):
    data = await request.post()
    testing = data['testing']
    return web.Response(body=testing.encode('utf-8'))

async def init(loop, port=None, bind_all=False):
    if port is None:
        port = 8080
    ip = '<IP_ADDRESS>' if bind_all else '<IP_ADDRESS>'

    app = web.Application()
    app.router.add_route('GET', '/hello', handle)
    app.router.add_route('POST', '/post', handle_post)
    srv = await loop.create_server(app.make_handler(), ip, port)
    print("Server started at http://{}:{}".format(ip, port))
    return srv

class AioTestClient:
    def __init__(self, base_url, loop):
        self.base_url = base_url
        self.loop = loop

    def request(self, method, url, data=None):
        test_response = None
        async def async_request():
            full_url = ''.join((self.base_url, url))
            async with aiohttp.request(method, full_url, data=data, loop=self.loop) as response:
                response.body = await response.read()
                try:
                    response.text = response.body.decode('utf-8')
                except UnicodeDecodeError:
                    response.text = None
                return response

        def run_from_thread():
            nonlocal test_response
            future = asyncio.run_coroutine_threadsafe(async_request(), self.loop)
            test_response = future.result()
            self.loop.call_soon_threadsafe(self.loop.stop)
        thread = threading.Thread(target=run_from_thread)
        thread.start()
        self.loop.run_forever()
        return test_response

    def post(self, url, data=None, **kwargs):
        return self.request(METH_POST, url, data=data, **kwargs)

    def get(self, url, **kwargs):
        return self.request(METH_GET, url, **kwargs)

class AioTestCase(TestCase):
    client = None

    @classmethod
    def setUpClass(cls):
        loop = asyncio.get_event_loop()
        port = cls._get_available_port()
        loop.run_until_complete(init(loop, port=port))
        cls.client = AioTestClient('http://{}:{}'.format('<IP_ADDRESS>', port), loop)

    @staticmethod
    def _get_available_port():
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.bind(('<IP_ADDRESS>', 0))
            return s.getsockname()[1]

    def test_one(self):
        response = self.client.get('/hello')
        self.assertEqual(response.status, 200)
        self.assertEqual(response.text, "Hello world")

    def test_two(self):
        response = self.client.post('/post', data={'testing': 'abc123'})
        self.assertEqual(response.status, 200)
        self.assertEqual(response.text, "abc123")

@dgelvin it may look surprising but you really don't need a separate thread for running client requests: they may share the same event loop.

@asvetlov I believe using my technique the client requests are performed on the same event loop. The thread is just used to schedule the request on the event loop and then shut it down.

I feel your pain, guys.
The main challenge for me is introducing the test framework, not particular classes/functions but the complete solution.
Moreover, we switched (well, partially) to py.test instead of standard unittest.
Thus end users need two frameworks ideally.
Test framework means it should provide carefully calibrated tools for end users, covering not only aiohttp test needs but acceptable for application developers.
In other words I need to think about testing API as strong as about aiohttp itself.
We have plenty tests for aiohttp itself and aio-libs libraries. I should just get used techniques together and document the approach -- documenting is crucial part, you may copy-paste fixtures from tests/conftest.py for example but need to figure out what they does first.
I hope to introduce something next month -- perhaps for py.test first.

@dgelvin but you really don't need a thread at all -- please take a look on https://github.com/KeepSafe/aiohttp/blob/master/tests/test_urldispatch.py tests

@asvetlov Thanks, very helpful.  I see that simply using run_until_complete with the client request works great.

I have some fixtures for pytest and aiohttp. They're based on the ones from aiohttp itself but are significantly simplified as they concentrate on testing the application not the framework.
I might try taking them out of the private repo they're currently in and create them as a gist or PR.
Obviously the test fixtures should end up as a reusable library rather than just available to be copy & pasted. If we're using pytest (which I really like) this should obviously be done as a plugin.
@asvetlov in an ideal world do you think this plugin should be part of aiohttp or a separate package?

In fact, just found I'm using them in a public repo: https://github.com/em-2/em2-core/blob/master/tests/http/conftest.py demonstrates a very simple conftest.py.
The tests are only really a sub so far, but they demonstrate the idea:
async def test_missing_conversation_valid_html(client):
    headers = {'Authorization': 'Token 321', 'Actor'<EMAIL_ADDRESS>    data = '{"key": "foobar"}'
    r = await client.post('/123/messages/add/', data=data, headers=headers)
    assert r.status == 400
    content = await r.read()
    assert content == b'ConversationNotFound: conversation 123 not found\n'

Fixed by #902 and #910


'charcters' typo in splash image

https://i.imgur.com/UrDb9qc.png

Funnily enough, I made a typo in the issue title :-P

do u mean char[a]cters ?


[RMP] Multi-GPU Data Parallel training for Tensorflow in Merlin Models

Problem:
Single GPU training takes significantly longer than multi-gpu.  Customers would like to be able to accelerate their training workflows by distributing training across multiple GPUs on a single node.
Goal:
Enable customers to do data parallel training within Merlin Models training pipeline.
Constraints:

Single node
Embedding tables fit within the memory of a single gpu
Use NVIDIA best practices; aka Horovod

Starting Point:

Enable Horovod within the Merlin Models API

@marcromeyn can you flesh this out a little further.

This is a related ticket #764 - [BUG] Models does not support to tf.distribute.MirroredStrategy() for data parallel training

@viswa-nvidia @EvenOldridge
We need to add following success criteria :

Analyze scaling factor by using multiple GPUs: If we go from 1x GPU -> 2x GPUs -> 4x GPUs -> 8 GPUx - how much higher is the throughput?
Provide performance metrics (accuracy / AUC / etc) to show that there is no negative effect in the model performance
Provide guidance how to set global batch size, batch size per GPU and learning rate when scaling

If we provide only the technical functionality without testing the points above, we cannot guarantee it is working. I get these questions often from customers.

I provided an example to show that Merlin Models work with horovods: https://github.com/NVIDIA-Merlin/models/pull/778
However, we need to address the points above + the bug ( https://github.com/NVIDIA-Merlin/models/issues/765 ).
In addition, we should make it more user-friendly.

Noted. @EvenOldridge , please review and add to the goals. I am not sure if this ticket is fully defined.

I've been doing some experimentation with horovod integration with Models API, based on @bschifferer's example https://github.com/NVIDIA-Merlin/models/pull/778, and I fully agree with his point on all the success criteria he listed above, and also the need for dataloader to produce equal number of batches across partitions, as mentioned in https://github.com/NVIDIA-Merlin/models/issues/765.
Some additional notes on https://github.com/NVIDIA-Merlin/models/issues/765, it seems that the dataloader produces an unequal number of batches when the dataset is partitioned, which is problematic for horovod because one worker might finish processing all the batches and wait idle and/or hang and/or time out while the other worker(s) are still processing their batches. There might be some workaround like seeding from the dataloader side as mentioned in the issue, or from horovod side using hvd.join(), but the best solution is to have dataloader produce equal number of batches when partitioned.

I provided following example based on the current code: https://github.com/NVIDIA-Merlin/models/pull/855 - as I am OOO next week.
@edknv did a great job to provide the horovod functionality in Merlin Models.
I think we need to review the current flow of multi-GPU. I think it is not fully user-friendly / end-to-end integration:

I am not sure, if the issues with unequal batch size for the data loader is solved: https://github.com/NVIDIA-Merlin/models/issues/765 - if the solution is moved to the data is generated, correctly - how das that work? How are we ensure it with NVTabular? How about users who do NOT use NVTabular?
The unittest is written that each worker runs through the FULL dataset per epoch. That is incorrect. If we have 1M data points and 2 GPUs, each GPU should run only through 500k data points. I wrote the example that NVTabular produces distinct files per worker - however, that is not a solution which guarantees the point above.

We havent look on the points here:

Analyze scaling factor by using multiple GPUs: If we go from 1x GPU -> 2x GPUs -> 4x GPUs -> 8 GPUx - how much higher is the throughput?
Provide performance metrics (accuracy / AUC / etc) to show that there is no negative effect in the model performance
Provide guidance how to set global batch size, batch size per GPU and learning rate when scaling

@viswa-nvidia, I don't think that the tasks under Staring Point and Examples are the relevant items. Can you please follow up with @edknv and @bschifferer to capture tasks that enabled this roadmap ticket?


Connect nodes with different types

How to prevent users from connecting nodes that aren't with the same type,
for example, prevent user to connect a str output to an int input?

You can try switching on type_safety.
flowfunc.Flowfunc(
        id="input",
        config=fconfig.dict(),
        type_safety=True,
)


Skin URL

Can you make the skin the URL link, as in OgarUnlimited to the skins file and there you can upload your skin:
nameskin ; http://i.imgur.com/*******.png (or any link on the image)
When you log into the game to write: "nameskin" nick
All customers well and not all
" = <>

Client
sided
We
Can't
Do
that

Out of the scope of this project. Although this server parses skin names in <> tags, the client is the one who is supposed to get the skin image and draw it, not the server.

^^

@Luka967 u cant help me with ur client?
There is a lot of LAG when the zoom is increased compared to your normal configuration, when it is in 1 cell of more than 100k
u can fix that? or where work that functions?

[Violation] 'requestAnimationFrame' handler took 340ms

@Luka967 i think is recommended to use persistent FPS rather than shifting.

@gyeyoqu Please DM him about this instead of saying it on an issue that is not related.

var = [skinurl ,id ,idbot], [x23/x54/x32]
This is cilent sided

Have the skins instead of making the flag the skin name into a URL? This isn't supported by this project.


add column first_name

develop
GitHub Board
Issue link
#499 Sort publications by author
Story link
[#323 Story]( Sort publications by author)
Code reviewers

[ ] @teaFunny

Summary of issue

[ ]   add column first name to db,
[ ]   add trigger on update, insert users

Summary of change

[ ]   added column first name to db,
[ ]   added trigger on update, insert users

temporary solution. just for presentation. Sorting will be optimized.


Bump version to Omni Core <IP_ADDRESS>-rel

This bumps the version to <IP_ADDRESS> and prepares the release notes.

@zathras-crypto: I started with filling out the blanks here: 0fc1775a64ac1f24cc488ee88d08dd3d5d9e9b25.
I think we can get away with recycling the descriptions for the fee system and cross-property STO, instead of mentioning that we just fixed some stuff. What do you think?
Most of the release notes can probably also be used for 0.2.

@zathras-crypto: what do you think about the release notes? Please let me know, if there is anything to change, or add. :) Feel free to type as you like..

Hey bud, this all looks good to me - though I'm not sure we're making it clear enough that this is a mandatory upgrade.  For example:

This version is built on top of v0.0.11, which is a major release and consensus critical in terms of the Omni Layer protocol rules. An upgrade is mandatory, and highly recommended. Prior releases will not be compatible with new behavior in this release.

I read that as 0.0.11 is the major release and required upgrade, which may give the impression that this isn't mandatory.  Since activation will specify <IP_ADDRESS> as a min client version for fees & cross-prop STO, clients that haven't been upgraded will stop working.
TL:DR; I think we should be a bit clearer that this version is required once we start activating things :)

Maybe we should call it 0.0.12, to have at least some consistency, in terms of bumping the version number for consensus critical stuff?

Sounds perfect, I don't think we have to worry about version confusion with Bitcoin Core now - 0.0.12 would be fine :)

Here we go: https://github.com/OmniLayer/omnicore/releases/tag/v0.0.12

To compile that you need to let users know to set CXXFLAGS with fPIC, it needs exported as passing
CXXFLAGS=-fPIC ./configure will not be enough


:bug: forceload vector now requires an Integer

Fix #698

I noticed that the Pull Request contains some bugs, so please merge after the Re-request review.

Btw after this PR you can create new branches in this repository directly instead of using your fork, and you can push to the master branch without creating a PR when you feel confident ;D

From now on, I will create a branch directly and work on it, but I will do it with PR for a while. I'm not so confident so lol

:tada: This PR is included in version 3.0.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Allow EmptyList for prepared query parameters of type List<?>

Now if I try setting the value of a query parameter of the type List<?>, the server returns that EmptyList cannot be assigned to a parater of the type List<?>. It is annoying when you have a single prepared query which can get the empty list as its parameter.
For example
DECLARE $x AS List<uint32>;
DECLARE $y AS List<uint32>;

SELECT * FROM X WHERE x IN $x
UNION ALL
SELECT * FROM X WHERE y IN $y

@shnikd first, we need to implement the unit test to reproduce the issue and then we need a fix.


Installation issue

Hi I placed both the components and VST3 in their respective folders in Library but DDSP is nowhere to be found In Live and Reaper.
Reaper shows them as a failed to scan plugins(vst3) clicking on the manual scan(from the list of failed to scan plugins) still doesn't make them visible.
What am I missing?
I am on Sierra 10.12.6 , if this makes any difference.
I hope there is a solution.
Thanks!

Unfortunately, the minimum required macOS version is Mojave 10.14. You can try building the plugins yourself if you are comfortable building C++ projects using CMake. You would just need to update this line with your OS version and it might work. Good luck, and please update this issue if you manage to get it working on Sierra!


I am not able to solve the last challenge in the sets category  (input_validation_bug)

in the code you used (\n) as a separate for the two commands but you could have used a semicolon
all the challenges doesn't support using single or double quotations you could use a regx to support all of that

i am working on it

Yeah just faced with issue!


OpenCL codegen: poor optimization of #VGPR used

On ROCm 2.10, RadeonVII
In GpuOwl, the change referenced below speeds-up a big kernel by more than 33% by passing two arguments referencing the same memory buffer (thus the same data) instead of one in order to disable the ROCm optimizer from caching the data, once read, into VGPRs. This reduces the number of VGPRs used by the kernel from 156 to 125 and thus increases occupancy from 1 to 2.

tailFusedMulDelta
Before
After

NumVGPRsForWavesPerEU
156
125

Occupancy
1
2

https://github.com/preda/gpuowl/commit/1e0ce1d8abf9f8b189373085a6cbdc2e2d814d33
This change is a work-around-the-optimizer, because it actively hides information from the compiler (in this case it hides the equivalence of the two buffer arguments).
The optimizer should be able to balance the desire to cache once-read data into VGPRs versus the VGPR pressure. In this particular case the register pressure is extreme (as it reduces occupancy to 1) yet the optimizer still uses a lot of VGPRs to "keep data around".

Another example of the same, please have a look at this commit
https://github.com/preda/gpuowl/commit/b23e0d538dcd6cd10d35a058084494099f14a5e5
which increases performance (by reducing the number of VGPRs and thus increasing occupancy) by adding a volatile qualifier to global memory to disable caching the values read in VGPRs.
The optimizer should make a better decision about whether to cache or not in VGPRs -- decreasing the occupancy from 3 to 2 is a very large cost to pay for some cached values.
Please note that the solution consisting in using "volatile" in imperfect, rough, with secondary costs; a good optimizer could do much better.

Also with ROCm 3.1,
https://github.com/preda/gpuowl/commit/612febd684938d3bd3716a928070a63c20d4a8e5
which applies the "duplicate buffers" trick against the optimizer, reduces the number of VGPRs for the k_tailFused from 112 to 81 (corresponding to occupancy 2 -> 3).
Really, the cost of just re-executing the memory read when the data is still in the L1/L2 cache is nothing compared to that wastage of VGPRs.

Hi @preda
Thanks for reaching out.
Request you to share an update on ROCm 4.1.
Thank you.

@ROCmSupport ROCm 4.1 is not working for me on Radeon VII with the upstream amdgpu driver.

@preda not sure I am using the upstream driver but I have upgraded to ROCm 4.1 and it detects my Radeon VIIs.

@valeriob01 did you install rocm-dkms?

Yes.

@ROCmSupport ROCm 4.1 is not working for me on Radeon VII with the upstream amdgpu driver.

Did you do a fresh install from scratch ?
upgrade is not working for 4.1

Yes.

@valeriob01 the fact that you installed rocm-dkms means that you are using the "custom" amdgpu driver that comes with ROCm. In my case I want to use the amdgpu driver that is part of the official Linux kernel. My goal should be aligned with AMD's position, as they activelly open-source and merge amdgpu to the Linux kernel. Thus the situation here, that ROCm contains a driver that is not merged to any Linux kernel, is a bit mistifying; hopefully a temporary situation, I'm waiting to be able to run ROCm on Radeon VII without needing dkms, as before.

Hi @preda
As we mentioned, rocm 4.1 does not work without dkms for now, which is a temporary situation.
Hence I recommend you to roll back to old rocm like 4.0 or rocm4.1 with dkms.
Thank you.

Hi @preda, please check latest ROCm Documentation and ROCm 5.7.1 to see if your query has been resolved.  If resolved, please close the ticket.  Thanks.


WOW .. just WOW .....

You should simply have to open, update NuGets and rebuild all .. this is a VERY TROUBLING DIRECTION!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 76f39eec-9dd1-6167-56ec-47b26e0b5063
Version Independent ID: f9f2b0f3-01eb-cf58-8767-194fe0fc0e44
Content: Migrate from ASP.NET Core 2.2 to 3.0
Content Source: aspnetcore/migration/22-to-30.md
Product: aspnet-core
Technology: aspnetcore-migration
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Thanks for the comment.


Update dependencies

To be able to use the Traits exported from Maud they need to be of the same Crate version. I've updated the dependencies so it works again now.

You need to also update Cargo.release.toml to match (hacky workaround till Cargo supports having dev-dependencies enabled via feature flags).

Alright! I've updated that to match Cargo.toml

Yeah, it's not the greatest of checks... Probably errors because the version isn't updated in the release file.

I did update it!

I mean the version = 0.2.0 line.

I just pushed an improvement to the check on master, you could rebase and it should give a usable error message now.

I see now what you meant, I fixed that as well.

Cool, thanks a lot :)

And pushed 0.2.2 to https://crates.io/crates/maud-pulldown-cmark now (had a couple small issues with my auto-packaging script).

Awesome!


T-shirt sizing tooling

Description

This PR adds tools for t-shirt sizing that makes it simple to bolt it on to any existing component.
Action Button has been retrofitted with the new tooling to highlight the differences
How does it work?

Write out each t-shirt size, call @remapvars {} with find and replace, then simply use the size-agnostic variables. Most components already use size-agnostic variable names, so its possible no additional code will have to be modified:

.spectrum-ActionButton--sizeS {
  @remapvars {
    find: --spectrum-actionbutton-s-;
    replace: --spectrum-actionbutton-;
  }
}

.spectrum-ActionButton--sizeM {
  @remapvars {
    find: --spectrum-actionbutton-m-;
    replace: --spectrum-actionbutton-;
  }
}

/* ... */

.spectrum-ActionButton {
  block-size: var(--spectrum-actionbutton-height);
  /* ... */
}

During processing, the postcss-remapvars plugin takes all every possible variable for each t-shirt size and remaps them to a variable that does not have a size prefix:

.spectrum-ActionButton--sizeS {
  --spectrum-actionbutton-height: var(--spectrum-actionbutton-s-height);
  /* ... */
}

.spectrum-ActionButton--sizeM {
  --spectrum-actionbutton-height: var(--spectrum-actionbutton-m-height);
  /* ... */
}

/* ... */

Next, the postcss-dropunusedvars plugin checks the file to see which variables are used, then drops everything we didn't use so we're not publishing thousands of variables we don't use.

Finally, the postcss-dropdupedvars will remove variables we have overridden so we don't publish unnecessarily re-defined variables.

The end result is a set of classes that modify only the necessary variables to change the t-shirt sizes of the component.

To-do list

[x] If my change impacts other components, I have tested to make sure they don't break.
[x] If my change impacts documentation, I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.

[ ] Write documentation for postcss plugins
[ ] Discuss whether plugins should be in a separate repo
[ ] This pull request is ready to merge.

@misterbrownlee @GarthDB @jianliao @bernhard-adobe I think these PostCSS plugins should live in their own, separate repos. For the purpose of reviewing this pull request, I'd like to keep them in the monorepo. However, after this PR is approved, I would like to move them out to separate repositories. Does that sound like a plan?

I'm good with moving the plug-in out to their own open source release.


Grammar for golang produces incorrect syntax highlighting

VSCode Version: 1.32.3
OS Version: Windows_NT x64 10.0.17763

Steps to Reproduce:
See  https://github.com/Microsoft/vscode-go/issues/2367

This is an issue with the grammar, since the grammar is functioning as written and categorizing the second string as variable.other.assignement.go. Moving to the repository that owns the grammar.

New issue: https://github.com/atom/language-go/issues/170


refactor: rename n_accounts in integration tests

This change is

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 80.80%. Comparing base (a106711) to head (d8f9550).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #371   +/-   ##
=======================================
  Coverage   80.80%   80.80%
=======================================
  Files          35       35
  Lines        1620     1620
  Branches     1620     1620
=======================================
  Hits         1309     1309
  Misses        245      245
  Partials       66       66

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


bug(files): Files are not downloading properly

Issue:
On all OS's there seems to be issues downloading files
MacOS:
When you download a file from chat it will take a really long time to appear at save location and when you try to open it, it asks you to search the app store for a app to open that unknown filetype or it will say file is empty
https://github.com/Satellite-im/Uplink/assets/93608357/ddaa169b-5a82-4d53-b71d-0ab6fde7c048
https://github.com/Satellite-im/Uplink/assets/93608357/310565b2-ba96-48cc-af37-2c79582ad2ed

Windows:
When you try to download the file from chat, it doesn't recognize the file type and will not save
https://github.com/Satellite-im/Uplink/assets/93608357/67eef8da-37f1-4985-8689-7e36ed81ec02
Linux:
Not sure if this is the true behavior since I am on a vm but it also takes a long time to appear at save location, when I tried to open the file in Linux it took me to a text editor
https://github.com/Satellite-im/Uplink/assets/93608357/34788a0c-3bed-417d-a952-dcd648f64451
https://github.com/Satellite-im/Uplink/assets/93608357/0216e224-7a8f-4d0b-85e1-fd501a8c83b5

@lgmarchi removed you because darius fixed this in his PR
https://github.com/Satellite-im/Uplink/pull/1288


UI and user location

Removed all code related to old markers and report form.
Changed UI to try and make it mobile friendly, including removing most google maps buttons.
Added user location functionality with a button, works but sometimes just sets user location as default Victoria coordinates.
Added icons for user and different disturbances/crimes.
Currently has no marker adding functionality.

lgtm! the only thing is that the "show my location" button kinda falls off screen when viewed on a desktop screen, but we can fix that later


DeckSwiper swipe on button press not working

My problem is  in reference to the issue here https://github.com/GeekyAnts/NativeBase/issues/416
I want to click a button and the action of it would cause the deck swiper move to the next item in the deck. I have followed the instructions in the above link, it doesn't seem to work
  `<DeckSwiper`
              ref={(ds) => {this.deckSwiper = ds;}}
              dataSource={cards}
              onSwipeLeft={this.callLeftSwipe}
              onSwipeRight={this.callRightSwipe}
              renderItem={item =>

                    <Card style={{ elevation: 3 }}>
                        <CardItem>
                            <Left>
                                <Thumbnail source={item.image} />
                            </Left>
                            <Body>
                                <Text>{item.text}</Text>
                                <Text note>NativeBase</Text>
                            </Body>
                        </CardItem>
                        <CardItem cardBody>
                                <Text style={{ width: screenWidth, height: screenheight }}>Cool</Text>
                                <TouchableOpacity  onPress={() => {
                                  this.setCurrentItem(item);
                                  this.setModalVisible(true,item)
                                  }} >
                                 <Image style={{ resizeMode: 'cover', width: screenWidth, height: screenheight }} source={item.image} />
                                </TouchableOpacity>

                                </CardItem>
                        <CardItem>
                            <Icon onPress={() => this.deckSwiper.swipeRight()} name="md-thumbs-up" style={{ color: '#ED4A6A' }} />
                            <Text>{item.name}</Text>
                        </CardItem>
                    </Card>
                  }
            />`

P.S I have noticed that I can access the props of the deckSwiper and call the custom methods I have written for handling a swipe such as onPress={() => this.deckSwiper.props.onSwipeRight()}  it ends up calling the callRightSwipe method. However this doesn't solve the problem wherein the card would change to the next one in the deck. Any idea what the problem could be?

I have the same issue.

try onPress={() => this.deckSwiper._root. swipeRight()}

Thanks but that command crashes the app because this.deckSwiper._root is undefined.

Can you check with v2.0.12 if it works?

Should be fixed.

This isn't working for me either. Is there something else I need to do to access _root.
When I console.log(this.deckSwiper) it has these properties:
_reactInternalInstance: ReactCompositeComponentWrapper
context: Object
isMounted: (...)
props: Object
refs: Object
replaceState: (...)
resolveConnectedComponentStyle: ()
setWrappedInstance : ()
state: Object
updater: Object
wrappedInstance: DeckSwiper
__proto__: ReactComponent

@ryannewton native-base version?

@shivrajkumar I was on 2.0.9. Just upgraded and that solved it.
Thanks!

This seems to have broken in 2.1.4.
Any idea what the new API is?

I'm on v 2.1.3 and same issue. this._deckSwiper._root.swipeRight() gives an undefined.

This isn't working for me either. Is there something else I need to do to access _root.
When I console.log(this.deckSwiper) it has these properties:
_reactInternalInstance: ReactCompositeComponentWrapper
context: Object
isMounted: (...)
props: Object
refs: Object
replaceState: (...)
resolveConnectedComponentStyle: ()
setWrappedInstance : ()
state: Object
updater: Object
wrappedInstance: DeckSwiper
__proto__: ReactComponent

Did you find a solution for this swiper not working? Or did a specific version upgrade work for you?

This seems to have broken in 2.1.4.
Any idea what the new API is?

@ryannewton  did you find a version that worked for the deck swiper?
MIne is giving undefined for v.2.1.3.


Problems installing zsh-git-prompt

Hi!
I have tried installing the git-prompt, but get some strange errors!
When I add source (..../zshrc.sh) I get a strange warninger for every blank line in my config file?!
Also a few errors that stands out, *colors* function definition not found, no such option: *PROMPT_SUBST* ^M and command not found: *add-zsh-hook*
t460y~/.zsh_plugins/zsh-git-prompt$(git_super_status) % zsh
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:2: command not found: ^M
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:7: command not found: ^M
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:9: command not found: ^M
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:12: colors^M: function definition file not found
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:13: command not found: ^M
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:setopt:15: no such option: PROMPT_SUBST^M
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:16: command not found: ^M
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:18: command not found: ^M
zsh: add-zsh-hook: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:19: command not found: add-zsh-hook
zsh: add-zsh-hook: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:20: command not found: add-zsh-hook
zsh: add-zsh-hook: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:21: command not found: add-zsh-hook
: command not found...
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:22: command not found: ^M
/home/richard/.zsh_plugins/zsh-git-prompt/zshrc.sh:25: parse error near `in^M'
<EMAIL_ADDRESS>

problem with my gitconfig, added ^M (CR) to all my files :-1:


Error in train_DeepCC_model due to deprecated keras code

Hi @gaofeng21cn and @zero19970,
I’m attempting to use your DeepCC package but am encountering an error related to deprecated code.
Error: ValueError: decay is deprecated in the new Keras optimizer, please check the docstring for valid arguments, or use the legacy optimizer, e.g., tf.keras.optimizers.legacy.Adam.
In addition, it also has the following warning about another deprecated parameter.
WARNING:absl: lr is deprecated, please use learning_rate instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.
My entire code related to DeepCC is below:

fs <- getFunctionalSpectra(df)
labels <- c("I","II","III","IV")
deepcc_model <- train_DeepCC_model(fs, labels)
df <- get_DeepCC_features(deepcc_model, fs)

I am using R 4.2.2, tensorflow 2.11 on macOS Big Sur (11.7.4). Will be happy to provide any additional details!
Thanks for all your hard work on this cool tool!

Hi Andrew,
It seems that you have installed the latest version of keras 2.11. The official statement shows that keras$optimizers$legacy have been replaced by keras$optimizers$legacy$Adam().
The fastest way to solve this problem is to install an older version of keras and tensorflow in R lower than 2.9.0.
Best wish,
Chenghang

Hi Andrew,
In R environment, running keras::install_keras(version="2.9.0", tensorflow="2.9.0") is okay.
Best wish,
Chenghang


Allow ENV usage in WordPress

I will integrate https://github.com/vlucas/phpdotenv OR https://github.com/scottjs/wp-dotenv
like described here
https://m.dotdev.co/secure-your-wordpress-config-with-dotenv-d939fcb06e24
https://erikpoehler.com/2020/12/30/using-an-env-file-for-database-and-other-credentials/
I will let you know how and maybe it is interesting enough for you to integrate.

@tiredofit you know your docker-nginx-php-fpm better: is there an easy way to modify the fpm config to set clear_env = no?


Abstract intermediate type for NonDisRES

Creating a new NonDisRES-like type (in combination with the EnergyModelsRenewableProducers package) which uses the curtailment variable creates some peculiar errors suggesting the key of such a node does not exist.
A solution could be to introduce abstract type AbstractNonDisRES <: EMB.Source end and then define NonDisRES through struct NonDisRES <: AbstractNonDisRES. If we then use
function EMB.variables_node(m, 𝒩ⁿᵈʳ::Vector{AbstractNonDisRES}, 𝒯, modeltype::EnergyModel)
    @variable(m, curtailment[𝒩ⁿᵈʳ, 𝒯] ≥ 0)
end

other NonDisRES-like types that also needs such a variable can use curtailment as well (I guess the type AbstractNonDisRES can be used for the other constraints as well that dispatches on NonDisRES) .
The workaround in the current setup is to define a new curtailment variable
function EMB.variables_node(m, 𝒩ⁿᵈʳ::Vector{PayAsProducedPPA}, 𝒯, modeltype::EnergyModel)
    @variable(m, curtailment_ppa[𝒩ⁿᵈʳ, 𝒯] ≥ 0)
end

Generally speaking, the workaround is entirely within the philosophy of EMB. In the current stage, it can remain that way. We plan to do soon a minor release based on the upcoming major release of TimeStruct. In this case, we can also introduce said abstract type.

Just a general EMX remark about the issue: Even though the workaround works just fine, the struggle to figure out the issue was quite tedious (and the core issue was still unclear to me and @dqpinel after some hours of debugging). A strong suggestion for such core packages (as EnergyModelsRenewableProducers) to introduce intermediate abstract types (as suggested) I think would resolve the issue and allow for better extensibility of the packages in general.

Correction to myself: Somehow, it worked with a different index set to add a variable with an existing name. This should by all means not be possible in JuMP, as far as I know. Anyhow, the danger part into the documentation is well deserved.


In GETTINGSTARTED.md there is a typo in the code directions, extra space after documents

In GETTINGSTARTED.md there is a typo in the code directions, extra space after documents:
const db = await orbitdb.open('my-documents-db', { type: 'documents '})
should be:
const db = await orbitdb.open('my-documents-db', { type: 'documents'})

Thanks for reporting this issue. We will address shortly. Alternatively, feel free to submit a PR.


Porting Fixes for Loc String And Using CasesInsensitive Keys in storin…

…g detals about FQDN and frontend Port for azure vm resources

Looks fine.


⚠ Bofrost.lu has degraded performance

In ff5277b, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 874 ms

Resolved: Bofrost.lu performance has improved in 6962907.


support package: imports

Related to https://github.com/dart-lang/dartpad_server/issues/43.

It would also hopefully enable dart pad to support something like: http://code.makery.ch/library/hello-dart/de/

Would it be possible in the meantime to have white list of packages which can be made available? A good place to start would be libraries such package:collection.

or polymer, core-elements, paper-elements. They also need a transformer :-/

It's about as much work to support whitelisted packages as arbitrary packages :(
We're concentrating on getting the gist sharing work plumbed in, then will probably address packages.
packages or workflows that actually require associated html (like polymer) would be more work, and be a follow-on task - not be part of the initial packages support. We wouldn't be able to get a running app just by compiled dart code on a server; we'd also need to concatenate a bunch of html.

closing - tracking this here: dart-lang/dart-services#43.


Use ServerT with polymorphic monad

Ideally, I think we could just use types instead of a full server, but this is a what I could quickly do.

So, I made the server fuzz not require a server value, just the type of the server. The same can probably be done with the client side.
Although, it's not clear to me what the division between server and client is.

This doesn't work on my production API though. It fails with a type mismatch error. Do with this what you will. (:
Feel free to close this PR.


will the model run faster when I use nngraph?

For example, I could get a 2-layer MLP with Container or nngraph, will it be a little bit faster when I use nngraph?

No. Just look at the difference in complexities of the respective updateOutputs. nn.Sequential is a simple loop whereas nngraph has several loops, allocations, and conditionals.

@nhynes thank you, I think it was something like theano before.


Github readme streak stats resetting despite having an ongoing streak

Issue faced
I have been consistently working on github from last 15 days my current streak was 13 till 2 days ago. Now it is only 2 but i didn;t break it. The readme stats is not showing the streak record properly.
Expected behaviour
The current streak should be of 15 days but is instead of 2 days

Screenshots / Live demo link (paste the github-readme-stats link as markdown image)

I have private contributions enabled and also have read the existing contribution guidelines of readme stats card.

The stats are taken directly from the Github API so there isn't anything I can think of that can fix this.
Also, when I view your profile, I don't see any contributions from 4 days ago.
It might be a timezone issue

The stats are taken directly from the Github API so there isn't anything I can think of that can fix this.
Also, when I view your profile, I don't see any contributions from 4 days ago.
It might be a timezone issue

Oh i can see the graph on my side displays the contributions properly. Thanks i will look into it <3 .

It's possible if you were to self-host it and use a GitHub token for your own account, it might display in your timezone. I have not tested this theory yet, so if you'd like to try it and let me know the results, that would be helpful.

@DenverCoder1 that's a good idea to check the effects of timezone , i will try it and tell the difference👍


[engsys] Upgrade dev dependency dotenv version to ^16.0.0 for selected packages

to have pipelines' validation.

API change check
API changes are not detected in this pull request.


Rotate DOWN around Z axis pointing down

This fixes bug that upside down blocks had the wrong rotation

@nsg Found another bug with rotations code... Something upside down should be rotated the opposite direction around its own up axis (since it is pointing down). Totally confusing (for me at least) :confused:

@petterarvidsson yeah, I understand the confusion :)
Good that you found and fixed a bug! :+1:


[XML Export] V2 improvements and bugfixes

THIS PR COVERS:

XML Transaction node: should reproduce one separated workbasket

XML Message node should reproduce one separated event of updates specified by workbasket (eg: message 1 - create measures, message 2 - add footnotes to measures, message 3 - add duties to measures so on)

Generated XML should not include blank nodes

XML generation attributes: should be same as in existing LIVE samples (eg: utf-8 should be UTF-8 so on)

Test generated XML examples from 'Create Measures V2' in Taric Importer on local

Bugfix for footnote's issue:
When adding new footnote to the system (not reuse) importing highlights conformance rule triggered 'F04: At least one description record is mandatory.'

TRELLO CARD

 [XML Export] V2 improvements and bugfixes


Test state only

Added unit test for state only

Improve the updated test, and add the status test for the "pending_order_daily" part of vlt under different conditions

Change pending_oder_daily part of the logic becomes non storage and direct operation.


Markdown code block issue

<space>+text inside a code block is parsed as project. For example:
(A) `This is code +text`

This is a little tricky.  As you know, todo.txt is intended to be a very simple format with minimal syntax.  Since Markdown is not part of the todo.txt "standard", any quoting inside the text of a todo is not recognized as special by todo.txt.  In practical terms, that means that the jsTodoTxt parser doesn't treat quotes as special and doesn't recognize Markdown constructs in todos.  I think that is why the  +word is being recognized as a project, before the Markdown processing even begins.   As I would expect, the closing quote is being included as part of the project name.

@zerodat you're right about this.
@9z0 I agree it is an issue, but I would raise the question if this is a relevant use case that can really cause pain to sleeks users and would justify the work of figuring out a solution either on sleeks side or maybe even involving upstream.

@zerodat , @ransome1
Thanks for explanations. No, it is not a major issue. I would of course prefer a fix, but I can work around it until a feasible solution presents itself.
Should I close the issue?

@9z0 up to you. Again I wouldn't want to work on it but I don't mind adding it to the backlog.

OK, I would appreciate if you make a note of it somewhere for reference, and I will close this issue.

OK, I would appreciate if you made a note of it somewhere for reference, and I will close this issue.


[storybook:build] default project fails on windows

when I create a new project and I run "npm run storybook:build", I get the error below:
Is this a known issue?
C:\Projects\mynewtest-project>npm run storybook:build

><EMAIL_ADDRESS>storybook:build C:\Projects\mynewtest-project
> build-storybook

[BABEL] Note: The code generator has deoptimised the styling of C:\Projects\mynewtest-project\node_modules\storybook-prebuilt\dist\storybook-prebuilt-351d7073.js as it exceeds the max of 500KB.
[BABEL] Note: The code generator has deoptimised the styling of C:\Projects\mynewtest-project\node_modules\storybook-prebuilt\dist\storybook-prebuilt-351d7073.js as it exceeds the max of 500KB.

The service worker file was written to C:\Projects\mynewtest-project\storybook-static\sw.js
The service worker will precache 10 URLs, totaling 3.62 MB.

'C:Projectsmynewtest-project.storybookpreview.js' is imported by inline-entry.0.js, but could not be resolved – treating it as an external dependency
'C:Projectsmynewtest-project
ode_modulesstorybook-prebuiltweb-components.js' is imported by inline-entry.0.js, but could not be resolved – treating it as an external dependency
[BABEL] Note: The code generator has deoptimised the styling of C:\Projects\mynewtest-project\node_modules\storybook-prebuilt\dist\storybook-prebuilt-351d7073.js as it exceeds the max of 500KB.
> 1 | System.register(['C:Projectsmynewtest-project.storybookpreview.js', 'C:Projectsmynewtest-project
    |                                                                    ^ Unterminated string constant
  2 | ode_modulesstorybook-prebuiltweb-components.js'], function () {
  3 |   'use strict';
  4 |   var configure;
(node:16812) UnhandledPromiseRejectionWarning: SyntaxError: Unterminated string constant
    at ee (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:19391)
    at m (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:20704)
    at C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:22849
    at C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:25257
    at Object.L [as input] (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:25596)
    at a (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:27626)
    at s (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:27680)
    at p (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:28094)
    at _ (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:28210)
    at X (C:\Projects\mynewtest-project\node_modules\terser\dist\bundle.min.js:1:42572)
(node:16812) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:16812) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

node is at version 12.15.0
npm at 6.13.4
windows version 10
Steps to reproduces:
npm init @open-wc
app or web-component:Web component
(*)Demoing
scaffold Demoing
name of the web/component: mynewtest-project
structure to disk: Yes
Install with: npm
"npm run start" and "npm run storybook" work, "npm run storybook:build" throws the error above.

Ive heard this complaint from other people as well. Will look into it.

This was fixed with https://github.com/open-wc/open-wc/pull/1380 thanks for reporting it!


5-D tensors as weight for Conv2d?

Thanks for this outstanding work, I'm wondering why the weightnet.py use 5D tensors as weights for F.conv2d?
Does this really work for F.conv2d?
Actually this will raise errors for PyTorch, so is there a different behavior using MegEngine?
e.g. the code here.
Looking forward to your reply.

MegEngine uses a 5D tensor as the grouped convolutional kernel, which is different from PyTorch API.


Remove unused code from the app

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@bb0a219). Click here to learn what that means.
The diff coverage is 0%.

@@           Coverage Diff            @@
##             master   #3276   +/-   ##
========================================
  Coverage          ?   7.34%
========================================
  Files             ?     270
  Lines             ?   11524
  Branches          ?     902
========================================
  Hits              ?     846
  Misses            ?   10616
  Partials          ?      62

Impacted Files
Coverage Δ

...rw/commons/filepicker/FilePickerConfiguration.java
0% <ø> (ø)

...ava/fr/free/nrw/commons/filepicker/FilePicker.java
0% <ø> (ø)

...ree/nrw/commons/filepicker/MimeTypeMapWrapper.java
0% <ø> (ø)

...va/fr/free/nrw/commons/filepicker/PickedFiles.java
0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb0a219...c38c8d1. Read the comment docs.


Improve method selection for script body in SecureASTCustomizer

This ended up being all that was required for 4_0_X and earlier, so I merged this PR on those branches. Only for master the improved solution catering for JEP445 was needed.


Loading... spins forever using docker image 1.1.1

Bug Description
Hi,
I've upgraded langflow to version 1.1.1 pulling the latest docker image, I can log in, see the flows I've created on earlier version, but each time I try to edit one, I just got a loading screen.
I've tried to flush everything to have a fresh new install, deleted both langflow and postgres container and volumes, restarted everything, and I still got the same problem. I've also tried to use docker image v1.1.0 but it's even worth, with 1.1.0 I got a bunch of "table creation error" when langflow tries to start.
I don't see any error in the browser console, I've tried Brave and Firefox private browsing/normal, always the same.
Is there any way to get debug logs to see what's going on ?
Reproduction

Set Langflow up using docker
Create a new flow
Wait forever

Expected behavior
A blank flow appear, with components on the side and everything.
Who can help?
No response
Operating System
Debian 12
Langflow Version
1.1.1
Python Version
None
Screenshot

Flow File
No response

I've been fiddling with things since this morning, and BAM 12 flows appeared all of a  sudden and now it works. I can't really tell what did the trick.
But it works fine now !!!!

I have the same issue, but I am running Langflow i1.1.1 n a Python venv; NOT via docker image.
It is sporadic and when it happens and I see /api/v1/refresh return an authentication error, even though I had just logged in a minute prior. Doing a browser reload often clears the issue.


NIFI-4073 - fix duplicated stack trace

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit?

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Reviewing...

Ran contrib-check and all tests pass. +1, merging.
I did start a two node cluster and ensure it was communicating, then used pf to temporarily block the cluster port to trigger the error message. I did not see the duplicated stack trace.
# Enabled port blocking

2017-06-23 11:12:55,991 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-06-23 11:12:55,987 and sent to localhost:10000 at 2017-06-23 11:12:55,991; send took 3 millis
2017-06-23 11:12:56,561 INFO [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 3ecf8509-0b0b-4602-ac55-1ec9057dc3c3 (type=HEARTBEAT, length=2344 bytes) from localhost:8081 in 1 millis
2017-06-23 11:12:57,293 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 6930 nanos
2017-06-23 11:13:00,433 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@7a80ebcd checkpointed with 1 Records and 0 Swap Files in 12 milliseconds (Stop-the-world time = 4 milliseconds, Clear Edit Logs time = 4 millis), max Transaction ID 0
2017-06-23 11:13:00,998 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request dc71898f-f3e6-4aaa-af2e-b0dae65e4f53 (type=HEARTBEAT, length=2344 bytes) from localhost:8080 in 1 millis
2017-06-23 11:13:00,999 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-06-23 11:13:00,995 and sent to localhost:10000 at 2017-06-23 11:13:00,999; send took 4 millis
2017-06-23 11:13:01,572 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 83d6969d-6073-43d4-b8e3-c182d4feea48 (type=HEARTBEAT, length=2344 bytes) from localhost:8081 in 2 millis
2017-06-23 11:13:02,296 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 6734 nanos
2017-06-23 11:13:07,300 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 5332 nanos
2017-06-23 11:13:12,305 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 6497 nanos
2017-06-23 11:13:17,308 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 5851 nanos
2017-06-23 11:13:22,314 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 5554 nanos
2017-06-23 11:13:27,318 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 5786 nanos
2017-06-23 11:13:32,045 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message to Cluster Coordinator due to: java.net.ConnectException: Operation timed out
2017-06-23 11:13:32,323 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 6357 nanos
2017-06-23 11:13:37,328 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 6451 nanos
2017-06-23 11:13:42,333 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 5380 nanos
2017-06-23 11:13:42,333 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator localhost:8081 requested disconnection from cluster due to Have not received a heartbeat from node in 40 seconds
2017-06-23 11:13:42,333 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of localhost:8081 changed from NodeConnectionStatus[nodeId=localhost:8081, state=CONNECTED, updateId=125] to NodeConnectionStatus[nodeId=localhost:8081, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 40 seconds, updateId=126]
2017-06-23 11:13:42,334 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8081 -- Node disconnected from cluster due to Have not received a heartbeat from node in 40 seconds
2017-06-23 11:13:42,334 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator localhost:8080 requested disconnection from cluster due to Have not received a heartbeat from node in 41 seconds
2017-06-23 11:13:42,334 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of localhost:8080 changed from NodeConnectionStatus[nodeId=localhost:8080, state=CONNECTED, updateId=121] to NodeConnectionStatus[nodeId=localhost:8080, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 41 seconds, updateId=127]
2017-06-23 11:13:42,335 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8080 -- Node disconnected from cluster due to Have not received a heartbeat from node in 41 seconds
2017-06-23 11:13:54,349 INFO [NiFi Web Server-183] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:13:54,349 INFO [NiFi Web Server-141] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:13:54,349 INFO [NiFi Web Server-158] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:13:54,352 INFO [NiFi Web Server-211] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:14:03,063 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message to Cluster Coordinator due to: java.net.ConnectException: Operation timed out

# Disabled port blocking

2017-06-23 11:13:42,335 WARN [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8080 -- Node disconnected from cluster due to Have not received a heartbeat from node in 41 seconds
2017-06-23 11:13:54,349 INFO [NiFi Web Server-183] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:13:54,349 INFO [NiFi Web Server-141] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:13:54,349 INFO [NiFi Web Server-158] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:13:54,352 INFO [NiFi Web Server-211] o.a.n.c.m.e.NoConnectedNodesException Cluster failed processing request: org.apache.nifi.cluster.manager.exception.NoConnectedNodesException. Returning Internal Server Error response.
2017-06-23 11:14:03,063 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message to Cluster Coordinator due to: java.net.ConnectException: Operation timed out
2017-06-23 11:14:05,343 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2017-06-23 11:14:05,523 INFO [pool-12-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@3f4b59fd checkpointed with 0 Records and 0 Swap Files in 180 milliseconds (Stop-the-world time = 70 milliseconds, Clear Edit Logs time = 93 millis), max Transaction ID -1
2017-06-23 11:14:05,523 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 180 milliseconds
2017-06-23 11:14:34,103 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message to Cluster Coordinator due to: java.net.ConnectException: Operation timed out
2017-06-23 11:14:52,374 INFO [Process Cluster Protocol Request-3] o.a.n.c.p.impl.SocketProtocolListener Finished processing request e636ce6d-32b2-43f1-b4f5-57fcf9a31f25 (type=HEARTBEAT, length=2724 bytes) from localhost:8080 in 1 millis
2017-06-23 11:14:52,376 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-06-23 11:14:39,108 and sent to localhost:10000 at 2017-06-23 11:14:52,376; send took 13268 millis
2017-06-23 11:14:52,388 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8080 -- Received heartbeat from node previously disconnected due to Lack of Heartbeat. Issuing reconnection request.
2017-06-23 11:14:52,388 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8080 -- Requesting that node connect to cluster
2017-06-23 11:14:52,388 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of localhost:8080 changed from NodeConnectionStatus[nodeId=localhost:8080, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 41 seconds, updateId=127] to NodeConnectionStatus[nodeId=localhost:8080, state=CONNECTING, updateId=128]
2017-06-23 11:14:52,390 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 1 millis
2017-06-23 11:14:52,395 INFO [Process Cluster Protocol Request-4] o.a.n.c.p.impl.SocketProtocolListener Finished processing request c9b89497-1cf4-452e-a7e3-92975222f0b7 (type=RECONNECTION_REQUEST, length=2217 bytes) from localhost:8080 in 2 millis
2017-06-23 11:14:52,395 INFO [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Processing reconnection request from manager.
2017-06-23 11:14:52,395 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {localhost:8081=NodeConnectionStatus[nodeId=localhost:8081, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 40 seconds, updateId=126], localhost:8080=NodeConnectionStatus[nodeId=localhost:8080, state=CONNECTING, updateId=128]} to {localhost:8081=NodeConnectionStatus[nodeId=localhost:8081, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 40 seconds, updateId=126], localhost:8080=NodeConnectionStatus[nodeId=localhost:8080, state=CONNECTING, updateId=128]}
2017-06-23 11:14:52,396 INFO [Reconnect localhost:8080] o.a.n.c.c.node.NodeClusterCoordinator Successfully requested that localhost:8080 join the cluster
2017-06-23 11:14:52,404 INFO [Reconnect to Cluster] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
2017-06-23 11:14:52,413 INFO [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Setting Flow Controller's Node ID: localhost:8080
2017-06-23 11:14:52,413 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator This node is now connected to the cluster. Will no longer require election of DataFlow.
2017-06-23 11:14:52,413 INFO [Reconnect to Cluster] o.apache.nifi.controller.FlowController Starting 0 processors/ports/funnels
2017-06-23 11:14:52,414 INFO [Reconnect to Cluster] o.apache.nifi.controller.FlowController Started 0 Remote Group Ports transmitting
2017-06-23 11:14:52,414 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {localhost:8081=NodeConnectionStatus[nodeId=localhost:8081, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 40 seconds, updateId=126], localhost:8080=NodeConnectionStatus[nodeId=localhost:8080, state=CONNECTING, updateId=128]} to {localhost:8081=NodeConnectionStatus[nodeId=localhost:8081, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 40 seconds, updateId=126], localhost:8080=NodeConnectionStatus[nodeId=localhost:8080, state=CONNECTING, updateId=128]}
2017-06-23 11:14:52,416 INFO [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Node reconnected.
2017-06-23 11:14:52,977 INFO [Process Cluster Protocol Request-5] o.a.n.c.p.impl.SocketProtocolListener Finished processing request c42f9b35-953d-47dc-a2f9-952307c60461 (type=HEARTBEAT, length=2344 bytes) from localhost:8081 in 1 millis
2017-06-23 11:14:57,396 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8081 -- Received heartbeat from node previously disconnected due to Lack of Heartbeat. Issuing reconnection request.
2017-06-23 11:14:57,396 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for localhost:8081 -- Requesting that node connect to cluster
2017-06-23 11:14:57,396 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.node.NodeClusterCoordinator Status of localhost:8081 changed from NodeConnectionStatus[nodeId=localhost:8081, state=DISCONNECTED, Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 40 seconds, updateId=126] to NodeConnectionStatus[nodeId=localhost:8081, state=CONNECTING, updateId=130]


[complete] RJBB, Osaka Kansai International Airport

Osaka Kansai International Airport is just one of Japan's many airports. Requested in #683 by @Exfil!
Aviation is very important in Japan - after all, they are just a load of islands.

Relocated to openscope/openscope#99, closing!


Show warnings only for open blocks

(see http://forum.biicode.com/t/show-warnings-only-for-open-blocks/334 )
I'd like to use warning flags like "-Wall -Wextra -Weffc++" on my code, but using
TARGET_COMPILE_OPTIONS(${BII_BLOCK_TARGET} INTERFACE -Wall -Wextra -Weffc++)

shows a lot of warnings for the blocks I'm including (e.g. boost).
Reasoning
I generally assume the libraries I use are working, but might not be written using good C++ style. Some libraries might be written well, but still throw warnings on -Weffc++, because they use certain constructs for performance (e.g. boost often doesn't have virtual destructors on inherited classes).
As a developer using an arbitrary library, it is not necessarily my task to remove the warnings from their code (which might sometimes not be possible like in the boost example above), but I'd like to use the warnings mechanism to be notified of style issues in my own code. Warnings in code written by others just clutters the output and makes it harder for me to find the issues in my own code.
Many languages (e.g. Java, Scala, ...) only show you warnings in your own code and I think that this generally is the better approach.
They actually can distinguish whether you're including a system header, and in this case, they also hide the warnings from there. But they can't easily distinguish 3rd party library headers from your own headers (the 3rd party library might just be in some subfolder of your project for example).
Different to plain compilers, biicode is in a position to easily decide whether a header is a 3rd party header or not (open blocks are own code, closed blocks 3rd party). So biicode might have the chance here to be better than previous systems.
Since this is a behavior C++ programmers might not be used to, an opt-in or opt-out could be offered. I'll definitely opt-in, because I think it is much better behavior.
Possible implementation
I've read that including other libraries as system libraries makes gcc ignore warnings for these library headers. This could for example be done with
include_directories(SYSTEM "${LIB_DIR}/Include")

Is it possible to include closed blocks this way and therefore make gcc only show warnings for open blocks?

I am trying to do so, and added the SYSTEM option to all our TARGET_INCLUDE_DIRECTORIES, but doesnt seem to have the desired effect.
BTW, just in case you also want to try out things, there is always the option of running from source, it is not difficult to setup, and you will manage to run patches and fixes before the next release. But for the installed/compiled version, you can also hack the biicode.cmake template file. It is located in your installation => bii\biicode\client\dev\cpp\cmake folder. You can try to modify it, it will be used in your bii commands and projects. Just make sure to backup and restore it.
What I have tried is to add the SYSTEM option to all target_include_directories there, so it is actually being used for all includes, and still the warnings are propagated, will keep investigating.

I have figured out how to do it, but only for non absolute paths, i.e. not for #include "user/block/header.h" but for #include "header.h" and adding an [includes] with header.h: user/block.
Also I am configuring it as a opt-in functionality, as it breaks arduino builds. Are you using absolute or relative paths?

For my blocks to work with simple project layout (on travis), I have to use absolute paths when including other blocks. Paths inside a block are relative, but this is my code, so I want to get the warnings.
Opt-in is fine for me.

ah I misunderstood your last post - didn't see the thing about using the [includes] section before.
Yes, this might be a workaround, although it would be better without of course.
What is the problem with absolute paths? Do relative paths in subfolders work ("folder/header.h")?
Is it then possible to use a "relative" path "user/block/header.h" and map it in [includes] to "user/block/header.h"?

The current problem with absolute path, in order to achieve this SYSTEM
functionality is that absolute paths are added to include directories in
the project root CMakeLists.txt, and thus, there is no way to opt-in,
opt-out this, as it is done before any other processsing.
Relative include paths, those defined by [includes] and [paths] are added
to the include paths after, and thus can be user changed, or configured
with opt-in opt out.
Inside the same block, includes can (and should, if you want to open them
in new layouts -L=simple), be relative, which can be achieved with the
[paths] configuration.
Includes to other blocks can indeed be relative, using the [includes]
section. Basically user/block/header.h is mapped as:
[includes]
header.h: user/block
Wildcard patterns are allowed (gtest/*.h: google/gtest/include). This is
very useful if you want to keep your code build-able without biicode, as
the user/block will not be hardwired in code. I keep working trying to fix
remaining tests for this issue. Even if I am not able to fix the absolute
path case, I might try to release the other, so at least can be partly
useful.
Diego Rodriguez-Losada
CEO Biicode Innovation SL
M: +34 656274654
2015-05-03 16:53 GMT+02:00 Sebastian Meßmer<EMAIL_ADDRESS>
ah I misunderstood your last post - didn't see the thing about using the
[includes] section before.
Yes, this might be a workaround, although it would be better without of
course.
What is the problem with absolute paths? Do relative paths in subfolders
work ("folder/header.h")?
Is it then possible to use a "relative" path "user/block/header.h" and map
it in [includes] to "user/block/header.h"?
—
Reply to this email directly or view it on GitHub
https://github.com/biicode/client/issues/18#issuecomment-98490190.

Thanks for the detailed explanation, that makes sense.
If I understood you correctly, the problem with absolute paths is that biicode.conf isn't processed yet when adding them to include_directories. Maybe a solution could be to define opt-in/opt-out behavior not in biicode.conf, but somewhere else? Say settings.bii?
I'm not sure which is the better place anyhow. I'd like to have it in biicode.conf, so I don't have to reconfigure it per build client, but on the other hand this configuration might be a client decision that is/should be independent from the actual source code.

Hi Sebastian!
I have just pushed it to develop (you can use it from source, will be in next release): https://github.com/biicode/client/commit/92d9e4dae729dae6319462b7882bc437d092c776
To summarize:

SYSTEM for deps is enabled by default (it passed all of our integration and regression tests), no need to configure in biicode.conf, handled as a CMake options, which makes sense, you can opt-out with:

$ bii configure -DBII_DEPS_SYSTEM=OFF

It works both with relative and absolute includes :)

If you try it from source, or when next release is out, could you please tell me if it worked? Thanks!

Just tested it from your develop branch. It works like a charm. This really improves my ability to keep my code clean, thanks you so much :)
One possibility for improvement: Also use this for boost libraries. When using the boost/install biicode repository with "-Wall -Wextra", I get a lot of warnings from boost (for example .biicode/boost/1.57.0/boost/spirit/home/classic/utility/functor_parser.hpp)

You are welcome :) that was a good suggestion, I also dislike so many warnings when building deps. The boost libraries are handled a bit differently, probably @Manu343726 who developed that block can have a look at it.
Cheers!


Public access mandates

Fixes #335, #338 and #372
The PR achieves three new features:

It fetches the URL to the homepage from an author's profile
Fetches an unique organization ID from the author's profile
Fills in  the public access information, and optionally marks which articles are available according to the funding mandates and which do not.

The new code has almost 100% test coverage.

Waiting on the previous PRs to be merged

@ipeirotis - The build is failing due to unrelated errors. This PR can wait, since I have at least two more coming before v1.5. The other PRs were specifically for the health of this Github repository, but is otherwise non-functional. I wanted those to be merged to main and permissions for Codecov to be given, so I could check the code coverage on this and subsequent PRs.

Marking it again as draft because this shouldn't be a part of v1.4.5

This should be good to go into develop now.


Add sources and search guide pages to the new header

Fixes
Fixes #1987 by @obulat
Description
Added source and search guide pages to the new header
Checklist

[x] My pull request has a descriptive title (not a vague title like Update index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[ ] I tried running the project locally and verified that there are no visible errors.
⬆ Currently I'm getting an error, but I don't think it relates to any of the changes I made.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Thank you for your PR, @AlexVCS!
The repository recently changed from using husky to pre-commit. This is probably causing the error you are getting. To fix it, run Undo Husky config git config --unset core.hooksPath

Thanks! I've tried writing that command different ways and I keep getting command not found. I've run ```volta install pnpm

Would you be comfortable running the Playwright tests in Docker to update the snapshots for failing tests, @AlexVCS? It's OK if not, I can add the updates if you need.

I'll try and do it, thanks!

I ran the Playwright tests and the error I'm getting throughout is this
 Error: Cannot find module '~/locales/en.json'
I know that with #1949 the file was changed to en.json5 and I'm guessing that's why this error is happening?

I ran the Playwright tests and the error I'm getting throughout is this
 Error: Cannot find module '~/locales/en.json'
I know that with #1949 the file was changed to en.json5 and I'm guessing that's why this error is happening?

Can you run pnpm dev? That should also run the necessary i18n scripts

I ran the Playwright tests and the error I'm getting throughout is this
 Error: Cannot find module '~/locales/en.json'
I know that with #1949 the file was changed to en.json5 and I'm guessing that's why this error is happening?

Can you run pnpm dev? That should also run the necessary i18n scripts

the same error I was getting initially persists

Hi @AlexVCS, I rebased your branch onto main. The problems with localse should be fixed. Could you try to run pnpm dev on updated branch? Please, let me know if you have any problems.

I've added some snapshots to your PR, @AlexVCS. Judging by the snapshots, we will need to fix https://github.com/WordPress/openverse-frontend/issues/2012 before we can merge this PR to make sure that the footer doesn't break the layout.

Blocked until #2012 is closed.

#2012 has been merged so this is unblocked. @AlexVCS there are some merge conflicts, could you update the PR to resolve the conflicts?

@WordPress/openverse-frontend anyone interested in taking over this PR and finishing it?

I've re-based this PR onto main now that the internal header PR is merged. It is ready to be reviewed.

sorry for my delays in responding. I hope to pick up other PR's for the project in the future

sorry for my delays in responding. I hope to pick up other PR's for the project in the future

Welcome back, @AlexVCS! I've noticed that your fork of openverse-frontend had a stray lock file that was blocking you from updating it. Let me know if you need help getting it fixed.

Thanks! It looks like I was able to update my fork and pull down those changes! You no longer see that stray lock file, do you?

Thanks! It looks like I was able to update my fork and pull down those changes! You no longer see that stray lock file, do you?

Great! Yes, I see that your fork is updated and without any stray files.
Thank you for your contribution in this PR! 🎉


Soul Cookies: Next-tier crafting components

This idea came from a stray comment in IRC by Temia
Soul Cookie
Recipe: "WSW", "W", Wheat, "S", Soularium
Properties: 66% Chance "+1 Hunger, -1 Health", 33% Chance "-1 Hunger, +1 Health", 1% Chance "+1 Hunger, +1 Health"
Soul Essence
VAT Recipe: 1000mB Hootch, 2*Soul Cookie => 250mB
Properties: finite, lighter-than-air gas; gives light level 10
Concentrated Soul Essence
VAT Recipe: 1000mB Soul Essence, 2*Soul Cookie => 250mB
Properties: finite, lighter-than-air gas; gives light level 12
(That's 40 Gold, 40 Soul Sand, 80 Wheat, 16B Hootch for one Bucket of Concentrated Soul Essence.)
Pure Soul Essence
VAT Recipe: 1000mB Concentrated Soul Essence, 2*Soul Cookie => 250mB
Properties: infinite, viscosity high (flows far), lighter-than-air liquid; gives light level 15
(That's 168 Gold, 168 Soul Sand, 336 Wheat, 64B Hootch for one Bucket of Pure Soul Essence.)
And now we have our next-tier crafting components.

test


Refactored vertical_slash_index.cu for performance improvement

What does this PR do?
This PR refactors the CUDA kernel in vertical_slash_index.cu to optimize the performance of vertical slash indexing for large context sizes in MInference. The changes are primarily focused on improving the efficiency of block and column indexing operations during inference.
The refactoring led to a 6-second improvement in the execution time, as shown in the attached profiling report images. The optimization was achieved by reusing calculations within the CUDA kernel and improving the memory access patterns.
We tested the refactored code with 100K tokens using NVIDIA A30 GPUs (24GB memory), which limits the scale of our tests compared to the original results with 1M tokens. The project maintainers are encouraged to rebuild and validate the changes with larger token sizes and on other GPU models like A100.
The profiling results (before and after refactor) are attached, showing a reduction in total inference time from 268.128 seconds to 262.092 seconds.
Performance Results
Below are the benchmark results comparing the original and refactored versions of MInference on NVIDIA A30 GPUs for 10K, 50K, and 100K tokens:

GPU
Version
10K Tokens (s)
50K Tokens (s)
100K Tokens (s)

A30 (2 GPUs)
MInference
3.86
12.82
26.02

A30 (2 GPUs)
MInference (Refactored)
3.796
12.88
25.98

We request that you test the refactored version with an A100 GPU using your original settings, especially for larger token counts such as 1M tokens, as we were limited by the memory capacity of the A30 GPUs.

Fixes #(issue number, if applicable)
Motivation

Performance improvement: The motivation behind this refactor is to optimize the CUDA kernel used in vertical_slash_index.cu for better inference performance. Our changes focus on minimizing redundant calculations and improving memory access patterns during the inference process.

Context size handling: The refactor improves the handling of large context sizes, particularly when working with limited GPU memory on A30 devices. This may benefit applications that need to process large-scale input data efficiently.

Dependencies

No new dependencies were introduced by this PR.

Before submitting:

[x] This PR fixes a performance bottleneck in the inference pipeline.
[ ] This was discussed/approved via a GitHub issue.
[x] The documentation was updated where necessary.
[x] Tests were performed using 100K tokens on NVIDIA A30 GPUs.
[ ] Additional tests are recommended for larger context sizes (e.g., 1M tokens) and on different hardware (e.g., A100 GPUs).

Who can review?
Tagging relevant maintainers for this PR review:

General: @iofu728, @liyucheng09, @Starmys, @mydmdm
Kernel-related: @Starmys

Attached Image:

Profiling 1: Before refactor (268.128 seconds)
Profiling 2: After refactor (262.092 seconds)

This version includes the performance benchmarks and additional requests for testing on A100 GPUs for larger contexts.
Additional Notes
While the refactor has improved performance, we observed that there are still numerous cudaStreamSynchronize points in the current pipeline. These synchronization points could be further analyzed and potentially optimized to reduce unnecessary blocking and improve the overall throughput of the system. We recommend reviewing these points to explore further optimizations.

@microsoft-github-policy-service agree company="William & Mary"

Thank you for your contribution! Could you provide your CUDA version so we can check if this is related to compiler optimization ?