Added Apple Silicon compatible dependencies to FARM, tested on M1 MacBook Pro

Pull Request Description
What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)
#1114
#2138
#2158
This version of FARM submodule can install and pass all the tests on Intel-based Windows 10, Intel-based Ubuntu 18.04, Intel-based MacBook Pro, and Apple M1 based MacBook Pro.
What are the significant changes in functionality due to this change request?
Added "pip" source for fmpy, and added "forge" source for qpsolvers
Also corrected some minor bugs in FARM-Delta

For Change Control Board: Change Request Review
The following review must be completed by an authorized member of the Change Control Board.

[ ] 1. Review all computer code.
[ ] 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
[ ] 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
[ ] 4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
[ ] 5. If significant functionality is added, there must  be tests added to check this. Tests should cover all possible options.  Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in  XML block, the node <internalParallel> to True.
[ ] 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change.  The requirements and the requirements test shall be in sync.
[ ] 7. The merge request must reference an issue.  If the issue is closed, the issue close checklist shall be done.
[ ] 8. If an analytic test is changed/added is the the analytic documentation updated/added?
[ ] 9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.

Hello, could anyone share the details of the failed tests? This commit will have 3 raven tests to fail on Mac computers with M1/M2 CPU, but should pass all the tests on Windows, Intel-based Mac and Ubuntu 18.04. I clicked on the "Details" link but it says "You are not allowed to see results. Please sign in to an appropiate user." Thanks.

@wanghy-anl The failures are caused by the library installation, it seems it may caused by the libraries that updated in dependency file:
Downloading contourpy-1.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.7/301.7 kB 80.6 MB/s eta 0:00:00
Using cached googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
Downloading tblib-2.0.0-py3-none-any.whl (11 kB)
Using cached wcwidth-0.2.8-py2.py3-none-any.whl (31 kB)
Downloading xyzservices-2023.10.0-py3-none-any.whl (57 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.2/57.2 kB 22.5 MB/s eta 0:00:00
Installing collected packages: wcwidth, sortedcontainers, py-spy, opencensus-context, nvidia-ml-py, libclang, flatbuffers, distlib, colorful, zict, xyzservices, typing-extensions, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, tblib, smart-open, rpds-py, pyyaml, pyasn1, pyarrow, protobuf, prometheus-client, opt-einsum, oauthlib, multidict, msgpack, MarkupSafe, markdown, lz4, locket, lark, keras, imageio, grpcio, google-pasta, gast, fsspec, frozenlist, filelock, elementpath, contourpy, click, cftime, cachetools, blessed, attrs, async-timeout, astunparse, absl-py, yarl, xmlschema, werkzeug, virtualenv, rsa, requests-oauthlib, referencing, pydantic, pyasn1-modules, partd, numdifftools, netcdf4, Jinja2, gpustat, googleapis-common-protos, aiosignal, python-sensors, jsonschema-specifications, google-auth, fmpy, dask, bokeh, aiohttp, jsonschema, google-auth-oauthlib, google-api-core, distributed, aiohttp-cors, tensorboard, ray, opencensus, tensorflow
  Attempting uninstall: typing-extensions
    Found existing installation: typing_extensions 4.8.0
    Uninstalling typing_extensions-4.8.0:
      Successfully uninstalled typing_extensions-4.8.0
ERROR: Exception:
Traceback (most recent call last):
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 452, in run
    installed = install_given_reqs(
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 807, in install
    install_wheel(
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 731, in install_wheel
    _install_wheel(
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 692, in _install_wheel
    rows = get_csv_rows_for_installed(
  File "/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 258, in get_csv_rows_for_installed
    old_record_path = cast("RecordPath", row[0])
IndexError: list index out of range

ERROR: exiting with code 2

@wangcj05 Congjian, could you share any insights to fix this issue?
My commit is based on commit 28a735ac (merge of PR #2189) and I only changed the dependencies.xml within FARM submodule.
By the way, for the error message you posted above, may I know what OS does this info come from? I have a spare machine and I would like to build it with this OS to do further test.
Thanks, Haoyu

The failures happened for most of the machines, Linux, Mac and Windows. I will check other PRs to see if the failures are caused by other issues. I will keep you updated.

Thank you Congjian, much appreciated.
Haoyu

All jobs on 7335eed : invalidated by @wangcj05
retesting

Tests consistently failed at library installation stage. We have tested devel branch, and we do not see the issue. The difference that I see is the cvxopt library. @joshua-cogliati-inl Do you have any suggestion for this PR?

That looks more like a pip error than anything this pull request is doing, so at the moment I don't have a good idea.

Gents, I am testing this PR on my laptop with Ubuntu 18.04 LTS, and I can re-produce this error. Let me try to fix this issue.

It also repeatably occurs on devel with our pip install: https://civet.inl.gov/job/1811703/
"    old_record_path = cast("RecordPath", row[0])
IndexError: list index out of range"
So I suspect something changed in a package we are using, and now pip installs can fail.  This probably is triggering it by installing some more packages with pip from FARM. Thank you for looking into it.

If you manage to get them to pass locally, I would recommend that you push your changes.

@wangcj05 @joshua-cogliati-inl Gents, could you share the error info on Mingw Test? I tested it within Git Bash on my Windows 10 machine yesterday afternoon, and it passed all the tests...

The failure was in fetch, so I am rerunning the tests.

Job Mingw Test on bcfc8d9 : invalidated by @joshua-cogliati-inl
failed in fetch

Thanks Joshua. How about the failed tests on qsubs sawtooth? Is there anything I can do to resolve it?

The failure was in fetch (could not access hpcgitlab), so I am rerunning the tests.

Thanks Joshua. How about the failed tests on qsubs sawtooth? Is there anything I can do to resolve it?

The failure was in fetch (could not access hpcgitlab), so I am rerunning the tests.

The sawtooth computer is currently down for maintenance and upgrades, so for that one we have to wait a few weeks.  Sorry. (But they are highly likely to not be this pull requests' fault.)

Thanks Joshua. How about the failed tests on qsubs sawtooth? Is there anything I can do to resolve it?

The failure was in fetch (could not access hpcgitlab), so I am rerunning the tests.

The sawtooth computer is currently down for maintenance and upgrades, so for that one we have to wait a few weeks. Sorry. (But they are highly likely to not be this pull requests' fault.)

Thanks for the info.
I saw the Mingw tests are "Failed but allowed", is it a good time now to merge this PR and update the FARM submodule ID?

@wanghy-anl @joshua-cogliati-inl The changes just affect FARM plugin only. It is good for me to merge it. @joshua-cogliati-inl Please go ahead and merge this PR.

@joshua-cogliati-inl @wangcj05 Thanks Gents, much appreciated. Haoyu


parameter folder

Hi,
I can't see any parameter folders on my installation.
Is the repo up to date? Or, we should create a new yml file for it?
If you have only one camera work on this param file : <your_ws>/src/arena_camera/param/lucid_vision_camera.param.yml If you have more than one camera work on this param file : <your_ws>/src/arena_camera/param/multi_camera.param.yml

I believe the repo is not well maintained at this stage, the camera_info config also included a fix path which the username is knna. You might try create your own, and change the launch to read yours.


Do not insert null indexes

We were inserting indexes even for zeroed fields. It adds considerable extra work and storage space for optional fields (for example, the user_data_* ones).
This PR also removes the support for nullable types from the derived indexes, unifying the ZERO == null semantics for all the secondary indexes.
EDIT: The benchmark shows only ~5% performance gain because effectively only pending_transfer_id is no longer inserted for the benchmark workload (we use random user_data_*).

Nice! What is the performance difference?
On Tue, Feb 6 2024 at 12:44 AM, Rafael Batiati @.***>
wrote:

Merged #1520 https://github.com/tigerbeetle/tigerbeetle/pull/1520 into
main.
—
Reply to this email directly, view it on GitHub
https://github.com/tigerbeetle/tigerbeetle/pull/1520#event-11710779221,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADE4FVDQS5T4KLDBXDU3VTYSFOERAVCNFSM6AAAAABC2ZRKOOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRG4YTANZXHEZDEMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Nice! What is the performance difference?

The benchmark had a modest ~5% of performance improvement because only pending_transfer_id and timeout are no longer inserted. The real improvement depends on each use case.
If we changed the workload to submit the user_data_* fields as ZERO, we would reach >350K TPS, which means ~25% more (280K TPS) if compared with the previous code running the same workload (submitting user_data zeroed as well) and ~40% more (250K TPS) if compared with the current workload submitting random numbers.
The difference between 25% and 40% also shows how fast sequential indexes are (e.g. prefix always the same ZERO + sequential timestamp) in comparison with no index at all 🤔.

This is great @batiati.


1.1.0-beta009 crashed Visual Studio - too much memory consumed.

We have a very old Assembly that has many files and has never had the loving attention of someone used to StyleCop.  I analyzed it using the 1.1.0-beta009 version of StyleCopAnalyzers and Visual Studio stopped responding.  Visual Studio eventually came back to life (after about 5 mins) and briefly reported > 12,000 errors, then gave up the ghost.
With the StyleCop.MSBuild version, it identified the first 1,000 errors or so and then stopped.  My suggestion would be to have a similar cap for this Roslyn version (at least until VS becomes a 64bit application).

I've verified in the past that StyleCop Analyzers can run single-pass Fix All in Solution operations effectively with 200,000+ errors shown. If you are still encountering memory-related crashes, I would recommend following the steps in https://aka.ms/reportPerf so we can investigate them directly. 😄

I'll investigate - thanks for the link.


Addition of a legendary-chance and ub-chance to specify the chances of these Pokémon being generated by the pool

This doesn't really make sense. The species is randomly selected from all species which means if there were a "legendary change" or "ub chance" it wouldn't be accurate to the value in the config


Should add a detail page or view operation to the template to view the parameters of the template, not only by clicking edit

What's it about?

What's the reason why we need it?

I believe this is an important feature for KubeClipper. There're a few use cases:

case one
case two
...

Please leave your comments below if there's anyone agrees with me. Or just give me a thumb up.
Area Suggestion

/kind feature-request

/kind/feature

/milestone v1.2.0

close


RestClient.Execute doesn't respect WebExceptionHandler

Hello,
Before I go into detail about my issue, I am aware that this version of Grapevine (4.2.0) is no longer the latest version, however it is still in maintenance mode.  Switching to the latest version is not possible at this time.
To preface this, I am attempting to discover the best way to be alerted to disconnections either from the server going down, or the client losing connection to the network, etc (any of those unforeseen issues that could cause a message send or receive failure).  So I have a little test application that creates a RestServer and RestClient communicating over localhost using port 8884.
To my issue, I am looking through the source code for RestClient, specifically the Execute method, seeing how the WebExceptionHandlers dictionary is used.  I see that in case an exception is encountered, you first search for the WebExceptionStatus key in the dictionary, and if found, you Invoke that handler and then return null.  If the key is not found, you throw that exception to the invoking method (whatever executed the RestClient.Execute(IRestRequest)).  However, in my specific case, I'm seeing that this is not the case.  Below is a sample of what I'm doing.
public class Client
{
   private RestClient _client;

   public Client(string addr, int port)
   {
       _client = new RestClient()
       {
           Host = addr,
           Port = port
       };

       _client.WebExceptionHandlers = new System.Collections.Generic.Dictionary<WebExceptionStatus, WebExceptionHandler>
       {
           { WebExceptionStatus.ConnectFailure, GenericExceptionHandler },
           { WebExceptionStatus.ConnectionClosed, GenericExceptionHandler },
           { WebExceptionStatus.Timeout, GenericExceptionHandler },
           { WebExceptionStatus.SendFailure, GenericExceptionHandler }
       };
   }

   private void GenericExceptionHandler(IRestClient client, IRestRequest request, WebException exception)
   {
       Console.WriteLine($"Client ({client.Host}) ran into exception ({exception.Status}) with message:\n" + exception.Message);
   }
}

So, the WebExceptionHandlers dictionary is filled with general coverage of failure points.  The issue I'm running into is with the ConnectFailure handler.  Here is the execution method I'm using to test sending messages to the server:
public void SendMessage(string msg)
{
    Console.WriteLine("Client sending message:\n" + msg);

    var sendRequest = new RestRequest("/send");
    sendRequest.ContentType = ContentType.TEXT;
    sendRequest.HttpMethod = HttpMethod.POST;
    sendRequest.Encoding = Encoding.UTF8;

    sendRequest.Payload = msg;
    sendRequest.ContentLength = sendRequest.Payload.Length;

    // This try shouldn't really be necessary if I've covered all the WebExceptionHandler cases
    try
    {
        var response = _client.Execute(sendRequest);

        if (response != null)
        {
            Console.WriteLine($"Client received response ({response.StatusCode}):\n" + response.GetContent());
        }
    }
    // This shouldn't really call when a ConnectFailure occurs, but it does as you'll see with my example.
    catch(WebException e)
    {
        bool contains = _client.WebExceptionHandlers.ContainsKey(e.Status);
        Console.WriteLine($"Client {(contains ? "contains" : "doesn't contain")} a handler for this error status ({e.Status})");
    }
}

With that RestClient implementation out of the way, and assuming I have a RestServer implementation that works with a Route setup to handle the POST request (I can share that code with you as well if you need to see it), here is the sample code I'm using to test disconnection handling and other failures:
class Program
{
    static void Main(string[] args)
    {
        Client theClient = new Client("localhost", 8884);

        using (Server theServer = new Server("localhost", 8884))
        {
            theClient.SendMessage("Test");
        }

        theClient.SendMessage("Test 2");

        using (Server theServer = new Server("localhost", 8884))
        {
            theClient.SendMessage("Test 3");
        }

        return;
    }
}

The output of this test is the following:
Client sending message:
Test
Server received message:
Test
Client received response (Ok):

Ok

Server is stopping.
Client sending message:
Test 2
Client contains a handler for this error status (ConnectFailure)
Client sending message:
Test 3
Server received message:
Test 3
Client received response (Ok):

Ok

Server is stopping.

As you can see, the call to theClient.SendMessage("Test 2"); results in an exception being thrown in the RestClient's Execute method rather than the WebExceptionHandler being invoked.  Looking at the source code for Grapevine.RestClient, I am lead to believe that the Client.GenericExceptionHandler should have been called instead of running into an exception.
// Excerpt from RestClient.Execute(IRestClient) method
try
{
    var httpresponse = (HttpWebResponse)request.GetResponse();
    var elapsed = stopwatch.ElapsedMilliseconds;
    response = new RestResponse(httpresponse) { ElapsedTime = elapsed };
}
catch (WebException e)
{
    var elapsed = stopwatch.ElapsedMilliseconds;

    // THIS IS THE ISSUE
    // From my own example, WebExceptionHandlers does indeed contain the key, yet it doesn't execute this conditional code
    if (WebExceptionHandlers.ContainsKey(e.Status))
    {
        WebExceptionHandlers[e.Status].Invoke(this, restRequest, e);
        return null;
    }
    if (e.Response == null) throw;

    var httpresponse = (HttpWebResponse)e.Response;
    response = new RestResponse(httpresponse)
    {
        ElapsedTime = elapsed,
        Error = e.Message,
        ErrorStatus = e.Status
    };
}

Upon further analysis, it seems that none of the WebExceptionHandlers are being used upon encountering a WebException being returned from the HttpWebRequest.GetResponse() call.  I shorted the timeout on the request to 100 milliseconds and even though there is an exception handler for the Timeout WebExceptionStatus, it also falls into the try-catch block that handles WebExceptions being returned from the RestClient.Execute(IRestRequest) method.

Upon further analysis, it seems that none of the WebExceptionHandlers are being used upon encountering a WebException being returned from the HttpWebRequest.GetResponse() call.  I shorted the timeout on the request to 100 milliseconds and even though there is an exception handler for the Timeout WebExceptionStatus, it also falls into the try-catch block that handles WebExceptions being returned from the RestClient.Execute(IRestRequest) method.

I'm happy to take a look at this, but let me preface my help with a question...
I've removed the Client feature from Grapevine 5 in favor of recommending people use IHttpClientFactory for a number of reasons, but by far the largest being that I didn't think anyone was actually using Grapevine.Client. If I'm mistaken in this, I'll happily look at adding some version of them back (especially since I'm still in prereleases with GV5), but I'll need some community guidance on what would be the most useful implementation. Do you think this would be worth spending some time on?
That being said, I'll take a look at this and see what I can come up with. As always, thanks for using Grapevine!

I'm happy to take a look at this, but let me preface my help with a question...
I've removed the Client feature from Grapevine 5 in favor of recommending people use IHttpClientFactory for a number of reasons, but by far the largest being that I didn't think anyone was actually using Grapevine.Client. If I'm mistaken in this, I'll happily look at adding some version of them back (especially since I'm still in prereleases with GV5), but I'll need some community guidance on what would be the most useful implementation. Do you think this would be worth spending some time on?
That being said, I'll take a look at this and see what I can come up with. As always, thanks for using Grapevine!

@OmnipresentObserver What version of .NET Framework are you using?
I'm trying this now using a console app targeting netcoreapp2.1, and it seems to be working fine. However, my client isn't getting a ConnectFailure error, but an UnknownError. When I register a handler for that, it works just fine.

@OmnipresentObserver What version of .NET Framework are you using?
I'm trying this now using a console app targeting netcoreapp2.1, and it seems to be working fine. However, my client isn't getting a ConnectFailure error, but an UnknownError. When I register a handler for that, it works just fine.

@scottoffen,
In this particular case, I am running on .NET Framework 4.7.2 in a console application.  I could package up the test application to see if you can reproduce what I'm seeing.
As for your question about the Client usage in Grapevine 5, I'm more partial to leaning on IHttpClientFactory like you are.  Sadly, the code I'm dealing with I adopted and it was already using Grapevine.Client and I don't have the time to re-design.  So, I will work with what I have and improve it.

@scottoffen,
In this particular case, I am running on .NET Framework 4.7.2 in a console application.  I could package up the test application to see if you can reproduce what I'm seeing.
As for your question about the Client usage in Grapevine 5, I'm more partial to leaning on IHttpClientFactory like you are.  Sadly, the code I'm dealing with I adopted and it was already using Grapevine.Client and I don't have the time to re-design.  So, I will work with what I have and improve it.

I could package up the test application to see if you can reproduce what I'm seeing.

Please do, that would help tremendously. Meanwhile, I'll pull down and install 4.7.2 in preparation.

I could package up the test application to see if you can reproduce what I'm seeing.

Please do, that would help tremendously. Meanwhile, I'll pull down and install 4.7.2 in preparation.

As mentioned earlier, here is the project that includes the Grapevine test running in .NET Framework 4.7.2.
GrapevineTest.zip

As mentioned earlier, here is the project that includes the Grapevine test running in .NET Framework 4.7.2.
GrapevineTest.zip

Thanks for the code, that made tracking down the issue a piece of cake. The reason it isn't firing the exception handler is because the exception is being thrown on the call to ToHttpWebRequest() which is (currently) outside of the the try/catch block where the exception handler would be executed. I'll adjust that try/catch block and publish a new version.

Thanks for the code, that made tracking down the issue a piece of cake. The reason it isn't firing the exception handler is because the exception is being thrown on the call to ToHttpWebRequest() which is (currently) outside of the the try/catch block where the exception handler would be executed. I'll adjust that try/catch block and publish a new version.


Upload photos/files for a spot

@jasonash The camera feature has been integrated into the app so pictures can be taken and added to spots. How should this data get sent to (and retrieved from) the db?

Are there going to be multiple pictures per spot?
If not, we could add functionality to the API to accept a multi-part
form POST on http://strabospot.org/db/pictures/12345 to upload the image
and a GET request on the same URI to request said image.
On 1/21/15 12:52 PM, Jessica Good wrote:

@jasonash https://github.com/jasonash The camera feature has been
integrated into the app so pictures can be taken and added to spots.
How should this data get sent to (and retrieved from) the db?
—
Reply to this email directly or view it on GitHub
https://github.com/StraboSpot/strabo-server/issues/11.

@zmichels should answer that but I think the answer is yes, there can be the possibility of multiple pictures per spot.

yes. Multiple pictures should be able to be associated with a spot.

On Jan 21, 2015, at 1:12 PM, jasonash<EMAIL_ADDRESS>wrote:
Are there going to be multiple pictures per spot?
If not, we could add functionality to the API to accept a multi-part
form POST on http://strabospot.org/db/pictures/12345 to upload the image
and a GET request on the same URI to request said image.
On 1/21/15 12:52 PM, Jessica Good wrote:

@jasonash https://github.com/jasonash The camera feature has been
integrated into the app so pictures can be taken and added to spots.
How should this data get sent to (and retrieved from) the db?
—
Reply to this email directly or view it on GitHub
https://github.com/StraboSpot/strabo-server/issues/11.

—
Reply to this email directly or view it on GitHub https://github.com/StraboSpot/strabo-server/issues/11#issuecomment-70901372.

OK, I will implement the functionality below, but will incorporate a
separate unique identifier for pictures allowing multiple pictures per spot.
On 1/21/15 1:18 PM, Zachary Michels wrote:

yes. Multiple pictures should be able to be associated with a spot.

On Jan 21, 2015, at 1:12 PM, jasonash<EMAIL_ADDRESS>wrote:
Are there going to be multiple pictures per spot?
If not, we could add functionality to the API to accept a multi-part
form POST on http://strabospot.org/db/pictures/12345 to upload the
image
and a GET request on the same URI to request said image.
On 1/21/15 12:52 PM, Jessica Good wrote:

@jasonash https://github.com/jasonash The camera feature has been
integrated into the app so pictures can be taken and added to spots.
How should this data get sent to (and retrieved from) the db?
—
Reply to this email directly or view it on GitHub
https://github.com/StraboSpot/strabo-server/issues/11.

—
Reply to this email directly or view it on GitHub
https://github.com/StraboSpot/strabo-server/issues/11#issuecomment-70901372.

—
Reply to this email directly or view it on GitHub
https://github.com/StraboSpot/strabo-server/issues/11#issuecomment-70902436.

There is now a service via the API for uploading pictures to the database.
There is more information about the new services here:
http://strabospot.org/api#imageFunctions
Please note that this service requires a multipart/form-data
encapsulation with POST variables named "feature_id" and "image_file"
To help with testing, I have implemented a test page for uploading
images to the service:
http://strabospot.org/imageupload
On 1/21/15 1:16 PM, Jessica Good wrote:

@zmichels https://github.com/zmichels should answer that but I think
the answer is yes, there can be the possibility of multiple pictures
per spot.
—
Reply to this email directly or view it on GitHub
https://github.com/StraboSpot/strabo-server/issues/11#issuecomment-70902058.

@jasonash I tried out the test page and got the error Error: "Feature<PHONE_NUMBER>7903408 not found." but that id is a valid id for a spot in the server db.

@jessica-azgs OK, this should be fixed now. There was a lingering error in looking for the id.

@jasonash If a spot which has an image associated with it is uploaded and then say, the next day the spot is uploaded again it looks like on the server you then get a 2 of the same image associated with the spot. Can the server just see if the image_file being uploaded for a spot is the same as one already in the spot and then not duplicate the image?

@jessica-azgs OK, this should be fixed now. I found an error which was causing images to not be deleted when a spot is deleted. Now, when the spot is deleted, all associated images are also deleted making room for the next round of uploads.

@jessica-azgs OK, the image upload controller now checks to see if an image already exists for a spot with the supplied filename. If it does, an error is given. Otherwise, the image is uploaded as normal.

@jasonash If I add a caption to an image that was previously uploaded and then upload I'm getting a message that the image already exists on the server and the caption is not updated.

@jessica-azgs OK, this should be fixed now. Now if the image already exists, it is deleted and repopulated with the new caption.

@jasonash Is there another way to check if the image already exists than by the file name, like just checking if the image file itself is the same? Every image we're upload has the same name of "image.jpeg" because that's not a property we're actually using in the app.

@jessica-azgs I think the only way to check if the image files the same would be to compare the SHA-1 hash of each. I could conceivably calculate the hash on upload and store it in the database, and then compare on subsequent uploads for an identical hash value. I'll work on this first thing tomorrow morning.

@jessica-azgs OK, I have altered the image controller to use SHA1 hashes for file comparison rather than file name. This should allow you to use the same file name for every image but still prevent duplicate images from being stored in the database.

@jasonash Thanks! Looks like it is working now.


download a bin file from cloud and store(write in a file (IDFGH-2336)

----------------------------- Delete below -----------------------------
If your issue is a general question, starts similar to "How do I..", or is related to 3rd party development kits/libs, please discuss this on our community forum at esp32.com instead.
INSTRUCTIONS
Before submitting a new issue, please follow the checklist and try to find the answer.

[x] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[ ] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[x] I have searched the issue tracker for a similar issue and not found a similar issue.

If the issue cannot be solved after the steps before, please follow these instructions so we can get the needed information to help you in a quick and effective fashion.

Fill in all the fields under Environment marked with [ ] by picking the correct option for you in each case and deleting the others.
Describe your problem.
Include debug logs on the monitor or the coredump.
Provide more items under Other items if possible can help us better locate your problem.
Use markup (buttons above) and the Preview tab to check what the issue will look like.
Delete these instructions from the above to the below marker lines before submitting this issue.

----------------------------- Delete above -----------------------------
Environment

Development Kit:      [ESP32-Wrover-Kit|ESP32-DevKitC|ESP32-PICO-Kit|ESP32-LyraT|ESP32-LyraTD-MSC|none]
Kit version (for WroverKit/PicoKit/DevKitC): [v1|v2|v3|v4]
Module or chip used:  [ESP32-WROOM-32|ESP32-WROOM-32D|ESP32-WROOM-32U|ESP32-WROVER|ESP32-WROVER-I|ESP32-WROVER-B|ESP32-WROVER-IB|ESP32-SOLO-1|ESP32-PICO-D4|ESP32]
IDF version (run git describe --tags to find it):
// v3.2-dev-1148-g96cd3b75c
Build System:         [Make|CMake]
Compiler version (run xtensa-esp32-elf-gcc --version to find it):
// 1.22.0-80-g6c4433a
Operating System:     [Windows|Linux|macOS]
Power Supply:         [USB|external 5V|external 3.3V|Battery]

Problem Description
//Detailed problem description goes here.
Expected Behavior
Actual Behavior
Steps to repropduce

step1
...

// It helps if you attach a picture of your setup/wiring here.
Code to reproduce this issue
// the code should be wrapped in the ```cpp tag so that it will be displayed better.
#include "esp_log.h"

void app_main()
{

}

// If your code is longer than 30 lines, GIST is preferred.
Debug Logs
Debug log goes here, should contain the backtrace, as well as the reset source if it is a crash.
Please copy the plain text here for us to search the error log. Or attach the complete logs but leave the main part here if the log is *too* long.

Other items if possible

[ ] sdkconfig file (attach the sdkconfig file from your project folder)
[ ] elf file in the build folder (note this may contain all the code details and symbols of your project.)
[ ] coredump (This provides stacks of tasks.)

Hai,
I am working on dfu protocol to update the nrf52840. for that i need to  use esp32 to pair nrf  through BLE . i am sending command from esp32 to nrf as per the protocol . now i am at the point of sending a bin file . downloaded a bin file through http from cloud  and stored in char buffer. and my question is how to store(write)the data  into a file ?? and how to read the data from the file ? i need to read and  split the  data  from the file ,then i have to send through  ESP32 to nrf ( in esp32 i am using gatt client to pair with nrf) . is there source code ?? can you please help me out to solve this  ??
thanks in advance.

Please have a look at storage examples at https://github.com/espressif/esp-idf/tree/master/examples/storage for above requirement. Since this is not IDF issue, I will close this ticket.


Feat ajv keywords

Information

Type
Breaking change

Feature
No

:tada: This PR is included in version 6.17.0 :tada:
The release is available on:

v6.17.0
GitHub release

Your semantic-release bot :package::rocket:


Create working directories for systemd units automatically

Create working directories for systemd units automatically if they do not exist. Existing directories will keep their permissions intact.
Includes test coverage.
Closes #6.

Looks good. Thank you!


Add documentation

LGTM :+1:

Oh sorry, there's a couple of slight issues here- the parameters in the syntax need to have angle brackets around them to be treated as proper params and to be italicised.
Also, much more minor, could you hard-wrap the description at 72 characters please?


[FLINK-14121] Update commons-compress because of CVE-2019-12402

What is the purpose of the change

Fix CVE-2019-12402

Brief change log

Update commons-compress to the latest release

Verifying this change
This change is already covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): yes
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Do you know which tests cover this change?

@GJL No, I do not which tests do that.
Perhaps I'm trust the people, the process and the projects a bit too much.
I expect that any functionality that needs a dependency is enough test covered in Flink to show it still works.
Similar I expect that the guys at commons-compress have test coverage on the working of their project.
I did a build of the project locally and in Travis and both passed.
I expect this to be good.

The following NOTICE files also need to be updated.
flink-filesystems/flink-swift-fs-hadoop/src/main/resources/META-INF/NOTICE
flink-filesystems/flink-oss-fs-hadoop/src/main/resources/META-INF/NOTICE
flink-dist/src/main/resources/META-INF/NOTICE

Do you see anything else, @zentol?

@GJL @zentol I've updated those NOTICE files.
I've also done a search in all of the code for "1.18" and I have not been able to find any additional places related to this fix.

The flink-oss-fs-hadoop NOTICE should not be changed since it isn't actually bundling commons-compress; it is set to provided since it is a transitive dependency of flink-core (which is also provided).
Beyond that this looks good.

The flink-oss-fs-hadoop NOTICE should not be changed since it isn't actually bundling commons-compress; it is set to provided since it is a transitive dependency of flink-core (which is also provided).

Good catch. Since this implies that we have to remove some other dependencies from the NOTICE file, I am suggesting to merge this PR as is and fix the NOTICE file in a separate jira issue.

I am suggesting to merge this PR as is and fix the NOTICE file in a separate jira issue.

Sounds good. 👍

The flink-oss-fs-hadoop NOTICE should not be changed since it isn't actually bundling commons-compress;

Actually, commons-compress seems to be bundled:
jar -tf flink-oss-fs-hadoop-1.10.0.jar | grep XZCompressorInputStream
org/apache/commons/compress/compressors/xz/XZCompressorInputStream.class

Do you want to have another look, @zentol?

@GJL My guess is that you aren't on the latest master. Until recently we also bundled hadoop-common in this module, which pulled in commons-compress.

I see ok.


github-util: Update FROM version

Update underlying docker images.
Issue number: #1

Replaced by https://github.com/Senzing/docker-web-app-demo/issues/61


Add a filterMethod so different filter logic can be decided as well as provide an alternate filter method

Hi, I needed this component to use different logic on the items it was returning so I forked and added some new logic via a prop. I added it in a way that it would not effect any current users of your package as the default (not including the prop) will run the current logic you have.
I have added the following:
New prop filterMethod. Accepts the following inputs: "full" anything else runs your default logic which I called partial this is not hard coded and can be changed.
Updated README.md to explain the new prop type by adding it to the props table. I also added to the notes section two examples of how the filters work.
Note in the new filter I tested both RegExp and indexOf() against a list of about 2000 items and from my tests it seemed indexOf returned results faster. I can submit a RegExp version if you'd prefer.

@crcnum4 I will test it. But honestly I feel it is a great plus for the component.

Thanks, I thought it might be useful thus why I made it universal without breaking the original logic.

Can you remove the expo files? I think it is not really necessary. @crcnum4

oh i included expo files? sure i can check for that

removed

Anything else I can do with this item to make it better to merge?

@AugustoAleGon
Anything else i can do to make this pull request better?

PR request approved


Israeli OS community

Title
Israeli OS community
Author or authors
Uriel Ofir
Description
We have a local Israeli open-source community.
We are very active on our Discord server - https://discord.gg/WztShUnu.
Our site allows you to easily find information about Israeli open-source projects.
The community is young, having only been active for a few months. But we got a lot of significant achievements.
We open-source parties where we help people find their next step in the open-source world.
Resource link
https://opensource-il.vercel.app/
Conent type
community and data about OS in Israel
Related topics
community.

Thanks @UrielOfir! This year, we'll be updating the library to feature interviews / stories from open source maintainers. Do you have any published stories we could link to here?
Otherwise, is your group running any events during the month of May? If so, we'd love to put them on the calendar. https://github.com/github/maintainermonth/issues/new?assignees=&labels=&template=add-to-calendar.yml&title=EVENT_NAME

Hi @abbycabs ,
If I will write a post on a blog about our community would it count as a story?
As for an event, I add this one- https://github.com/github/maintainermonth/issues/106. Is it a relevant event?
Uriel :)

We're looking for stories that highlight maintainers specifically. Do you have a blog post about a maintainer?

I can write a post about our community and the projects I maintain in Medium.
What is the deadline to send you the post?

Anytime during the month of May

Unfortunately, I won't be able to write a post due to my busy schedule.
My interview for the Sustain podcast went well and I had a wonderful experience. So I'll settle for that.
Thank you for the opportunity.


Set up Travis CI

Related to https://cmiclab.cs.ucl.ac.uk/CMIC/NiftyNet/issues/269
We should set up CPU-only CI tests on Travis until a long-term solution for GPU testing has been found.

[ ] would the Travis matrix work for the different scripts currently available?

https://docs.travis-ci.com/user/languages/python

For the ImageSplit project, I wanted to avoid different testing paths for travis, gitlab-ci and local testing. So I set up tox.ini to do most of the CI steps using virtualenvs, and then configured .travis.yml and .gitlab-ci.yml to run the tests using the tox command. Another advantage is I can run any of the CI stages locally; e.g. tox will run all the tests; tox -e lint will run the linting etc.

A failing test example with tox: https://cmiclab.cs.ucl.ac.uk/GIFT-Surg/ImageSplit/-/jobs/42111

[ ] configure tox: sample only!
[ ] set up Travis to use tox script
[ ] consider opening a new issue for translating the existing CI scripts to tox (or maybe multiple issues, with one for each part)

[ ] testenv:lint not run

[ ] revert above commit (tox-lint)

@tomdoel the lint testenv does not get run, unless atravis section is added (see above commit), due to the way Travis handles build matrices.

[ ] Even the "quick" tests take too long on Travis, possibly due to the lack of a GPU.

[ ] export not working on Travis

[ ] how to deal with forks, as Travis needs to be set up for each fork separately?

[ ] rebase onto dev after #114 resolved

redefined-variable-type was deprecated already in 2017, might have just been removed by now.

redefined-variable-type became an extension (unavailable by default) as of pylint 1.7

@wyli As of the above commit, Travis CI is set up and the pylintjob and quicktest sections of GitLab CI have been translated to Travis CI. You can actually click the red cross next to the commit hash above and see the result for yourself 😄
There are two issues, though:

I haven't been able to get pylint to run, mainly due to version incompatibilities (current version: 1.9.1, on Adelie we have 1.5.2)
But the actual problem is that it takes about 30 mins to run even the quicktest section, due to the absence of GPU computing on Travis.

@dzhoshkun Looks nice! I'll send you a merge request to fix the pylint tests (we could take this opportunity to disucss/test the new workflow for contributors #98 ).

There seems to be another, (given the current setup) possibly easier, way to continue testing with our custom hardware (since GitLab v10.6 on 22 March as far as I can see).

[ ] finalise the GitLab CI/CD for GitHub test: https://gitlab.com/dzhoshkun/cpp-py-example/services/github/edit (imported from https://github.com/dzhoshkun/cpp-py-example)

@wyli @tomdoel I've managed to set up GitLab CI for one of my personal projects on GitHub, running its CI pipelines on Adelie. For the result, please see this commit (click the green tick!) and also this PR showing "All checks passed".
For the record, this feature has been around for a while, but involved a manual setup, and did not transmit the CI pipeline result back to GitHub.
Also for the record, I've imported the project not from CmicLab to GitLab, but from GitHub to GitLab. Mentioning this explicitly, as currently there seems to be an issue exporting projects from CmicLab.

Cool, looks like a feasible solution for GPU tests.

Sometimes CI runner couldn't find pylint command (https://gitlab.com/NifTK/NiftyNet/-/jobs/80703023) any idea?


Allow use of enviroment variables in profiles when using profiles_yml_filepath

I'm looking to build a dag which uses environment variables in the profiles when using a yml path. Something like the below:
dbt_envs= {
    'user': 'test',
    'private_key': <lookup private key>
}

profile_config = ProfileConfig(
    profile_name='dbt_project',
    target_name='snowflake_dev',
    profiles_yml_filepath='/usr/local/airflow/dags/dbt/profiles.yml',
    env=dbt_envs
)

With the profile yml referencing it like:
    snowflake_dev:
      type: snowflake
      account: test_account.eu-west-1
      user: "{{ env_var('USER') }}"
      private_key: "{{ env_var('PRIVATE_KEY') }}"

Hi @DanMawdsleyBA, Thanks for your message!
Some questions:

Which execution_mode are you using?
Are you setting the environment variables USER and PRIVATE_KEY when running Airflow, or would the expectation be that Cosmos should set them at a process level?

If the environment variables are available for the Airflow process, a way to forward them to the underlining Cosmos/Airflow operators is to use:
`operator_args={"append_env": True}`

Thanks for your response so I would be looking to build a dbt dag that would run all models and tests so running dbt build
So in the python dag I have lookup the aws secret using service_client.get_secret_value(SecretId='secret_value') this retrieves all the connection details.
So if set the operator_args as above could I do something like this:
in the dag have:
user='user'
private_key=service_client.get_secret_value(SecretId='secret_value')

my_cosmos_dag = DbtDag(
    project_config=ProjectConfig(dbt_project_path='/usr/local/airflow/dags/oprn_crw_ops_planning/dbt'),
    profile_config=profile_config,
    operator_args={"append_env": True},
    # normal dag parameters
    schedule_interval='@daily',
    start_date=datetime(2023, 1, 1),
    catchup=False,
    dag_id=f'{Path(__file__).stem}_{env}',
)

in the profile have the below and it would be populated with the value user and private key from the dag:
    snowflake_dev:
      type: snowflake
      account: test_account.eu-west-1
      user: "{{ env_var('USER') }}"
      private_key: "{{ env_var('PRIVATE_KEY') }}"

Is that correct?

Hi @DanMawdsleyBA , sorry for the delay getting back to you.
Do you know if this latest change worked?
You may have to capitalize user for this to work:
dbt_envs{
    'USER': 'user',
    'PRIVATE_KEY': service_client.get_secret_value(SecretId='secret_value')
}

If there are errors, could you share the logs?
The problem you may be having if Cosmos is using load_method=LoadMode.DBT_LS is that the environment variable is only being set for the execution of the operators, but not at DAG processing time.
If that is the case, something that may solve the issue is to do this in your DAG:
import os
os.environ["USER"] = 'user'
os.environ["PRIVATE_KEY"]  = service_client.get_secret_value(SecretId='secret_value')

Hi @tatiana the airflow logs themselves aren't useful as they just display no tasks but the dag parses. When I run a similar version locally using the debugger I get this so I can see the environment variables are loaded for user and private key. But it hits an error later on which looks to do with the sources (This is a working dbt project both on my local and airflow using a bash command) See below error (I'm using dbt-snowflake v1.52 and astronomer-cosmos v1.0.5):

Looking at the graph file here does there need to allow for sources but not create them as tasks?

@DanMawdsleyBA I agree the current behavior is misleading.
As you noticed in the source code, at the moment, Cosmos does not create tasks for DBT sources. Even though the log message says "error", it is more like a warning since the code skips those tasks, continuing to build the DAG with other supported resource types.
I just checked the dbt Sources documentation. How could we improve Cosmos?
Would you like to see each source materialized as an Airflow task?
What would be the expected behavior of those tasks?
Perhaps, if users set TestBehavior.AFTER_ALL, we could have a similar behavior to what we do with models, having a Task Group that joins the source task and that source test.
Looping in @jlaneve who can give more ideas and suggestions.

@tatiana I guess you include sources as tasks even if it's as simple task as checking if the table exists. A future feature perhaps could add a setting like freshness to see if that table has been loaded within a set sla period.
When I upload the dag should I be able to see all of the tasks (Even if the connection isn't setup)? Currently I'm unable to get any tasks to be created and not getting any logs or errors from Airflow. Not sure if you have any tips on debugging?

I like your proposal, @DanMawdsleyBA , could you create a dedicated Github issue with this suggestion for creating a task per source, with the suggested behaviour?
Based on the code you shared before, it seems you're using a custom profile, using profiles_yml_filepath='/usr/local/airflow/dags/dbt/profiles.yml. In this scenario, Cosmos does not rely on Airflow connections. Is this file available in your remote deployment?
Where/how are you running your remote Airflow service?  The conversion between a dbt project into an Airflow DAG happens during Airflow DAG parsing. Airflow names these logs as DAG processor logs (more on Airflow logs config). The logs' location will likely depend on your remote setup.
Once you find where the logs are being stored remotely, setting the logging level to DEBUG in your remote Airflow instance can be helpful. If you're using environment variables, a way of accomplishing this is:
export AIRFLOW__CORE__LOGGING_LEVEL=DEBUG

@DanMawdsleyBA I agree the current behavior is misleading.
As you noticed in the source code, at the moment, Cosmos does not create tasks for DBT sources. Even though the log message says "error", it is more like a warning since the code skips those tasks, continuing to build the DAG with other supported resource types.
I just checked the dbt Sources documentation. How could we improve Cosmos?
Would you like to see each source materialized as an Airflow task?
What would be the expected behavior of those tasks?
Perhaps, if users set TestBehavior.AFTER_ALL, we could have a similar behavior to what we do with models, having a Task Group that joins the source task and that source test.
Looping in @jlaneve who can give more ideas and suggestions.

I've spent a lot of time thinking about this and talking to users, and I think the right approach is outlined in this issue: https://github.com/astronomer/astronomer-cosmos/issues/427. Everyone's going to want different behavior with their sources (some people don't want them, some people want them as sensors, custom operators, etc) so a generating function seems to be the most flexible.

I like your proposal, @DanMawdsleyBA , could you create a dedicated Github issue with this suggestion for creating a task per source, with the suggested behaviour?
Based on the code you shared before, it seems you're using a custom profile, using profiles_yml_filepath='/usr/local/airflow/dags/dbt/profiles.yml. In this scenario, Cosmos does not rely on Airflow connections. Is this file available in your remote deployment?
Where/how are you running your remote Airflow service? The conversion between a dbt project into an Airflow DAG happens during Airflow DAG parsing. Airflow names these logs as DAG processor logs (more on Airflow logs config). The logs' location will likely depend on your remote setup.
Once you find where the logs are being stored remotely, setting the logging level to DEBUG in your remote Airflow instance can be helpful. If you're using environment variables, a way of accomplishing this is:
export AIRFLOW__CORE__LOGGING_LEVEL=DEBUG

Thanks for your help! I was able to find the logs for when dbt parses the dag and saw that no nodes were found even though both the profile, project are all in place. When I ran it locally I saw there was 33 nodes. I think there seems to be some sort of path issue where the dbt ls is possibly not running in the correct path. In my case on airflow I set project path to: '/usr/local/airflow/dags/oprn_crw_ops_planning/dbt' but the logs show the command here
/usr/local/airflow/.local/bin/dbt ls --output json --profiles-dir /usr/local/airflow/dags/oprn_crw_ops_planning/dbt --profile dbt_crew --target snowflake . So it might be running in the wrong path. I saw you made some changes previously here: https://github.com/astronomer/astronomer-cosmos/issues/411 it might be possible that fixes this issue as you specify the project path

Hey Dan, the change you mentioned in the other ticket was released as part of 1.0.5 and 1.1.0a1 - you could try it out.
I just saw your message in the other ticket. was the issue running dbt deps before dbt ls?

Hey Dan, the change you mentioned in the other ticket was released as part of 1.0.5 and 1.1.0a1 - you could try it out.
I just saw your message in the other ticket. was the issue running dbt deps before dbt ls?

Yeah it was the issue of not running dbt deps. When I removed all of the packages I was able to get the dag to parse correctly

@DanMawdsleyBA I'm closing this ticket since we have a ticket to add dbt ls in the backlog: #445. I'm very sorry it took so long to troubleshoot this problem. We'll improve this. If you are available and could help with the project, we'd also appreciate any contributions.


[QUESTION] How to write XLSX file to disk in chunks to prevent memory issues

Versions

PHP version: 7.0.32
Laravel version: 5.5
Package version: 3.1

Description
I need to export quite a lot of data to an excel file with some sheets inside. Currently what I am doing is writing my own CSV files chunking the query and writing it directly to storage. This way I can process all the information and get the data but I have to manually import it into Excel.
I tried this example:
https://laravel-excel.maatwebsite.nl/3.1/exports/multiple-sheets.html
But the memory gets exhausted quite easy and the excel file is not filled with data.
Is there a way to write the xlsx file in chunks of data?
I do not find any example of this. The file is always fully generated in memory and then stored into disk.
Can anyone help me?
Thanks

同问，数据量一大就导出不了，如用户表的信息1万条就导不出了

同问，数据量一大就导出不了，如用户表的信息1万条就导不出了

Please reply in English.

@Kyosfonica Did you have a look at the Maatwebsite\Excel\Concerns\WithCustomQuerySize concern. It allows Exportables that implement the FromQuery concern, to provide their own custom query size. That might be helpful to you (there's no documentation about it yet, but Pull Requests are always welcome).

I'll take a look. Thanks

I'm sort of in the same boat, but the data is not presented in a way that it will be easy to use a FromQuery concern. The table is a key-value pair table and one line in the export might be up to 489 rows in the database.
Using the queue functionality, is there a way to yield this so it will prevent memory issues?

An alternative to using FromQuery is to use FromIterator, you could create an iterator that creates those rows and doesn't keep it in memory.

There is no documentation about FromIterator method. Could please someone explain me, how do you use FromIterator method with Queued Export? Thank you in advance!
PS. I've got same error when trying to export 200k+ raws in queue.

@patrickbrouwers FromIterator and queue does not work? seems like when used together an empty file is generated only with the headers


Broken on latest celery

File "celerytest/celerytest/worker.py", line 39, in run
worker.run()
AttributeError: 'Worker' object has no attribute 'run'

I'm getting this error -- has this new version been released yet?


Update usage document with latest changes in the scripts.

@mogorant : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


Graph - Multiple issues with parsing of numeric constants in the graph gremlin query

Steps to reproduce.
I have tested this with both the Document Client extensions in C# as well as the Gremlin.Net native connection integers behaved the same between both gremlin.net however floats were not consistent.
quick summary of the issues

Negative sign lost in gremlin queries in both connections for ints/floats g.V('').inject(-1)
Syntax for passing a float constant in gremlin.net is not the same as the output format and does not round trip
Unable to pass literal floating point constants in with Microsoft.Azure.Graphs when using the exponent form

with <PackageReference Include="Microsoft.Azure.Graphs" Version="0.3.1-preview" />
✔ executed in 00:00:00.0554110. 1.97 RUs. 1 results. 65 characters.
gremlin> g.V('').inject(-9223372036854775807).inject(-2147483647).inject(-1).inject(0).inject('neg ints:').fold()
[  "neg ints:",  0,  1,<PHONE_NUMBER>,<PHONE_NUMBER>854775807  ]
✔ executed in 00:00:00.0715030. 1.97 RUs. 1 results. 65 characters.

gremlin> g.V('').inject(-100.1).inject(-10000000000000000000.1).inject(-1E19D).inject(-1E308D).inject('neg float')
✖ Gremlin Query Compilation Error: Failed to interpret string "1E+19" into any data type.

with <PackageReference Include="Gremlin.Net" Version="3.2.7" />
gremlin> g.V('').inject(-9223372036854775807).inject(-2147483647).inject(-1).inject(0).inject('neg ints:')
[  "neg ints:",  0,  1,<PHONE_NUMBER>,<PHONE_NUMBER>854775807  ]
✔ executed in 00:00:00.3091340. 5 results. 0 characters.

gremlin> g.V('').inject(-100.1).inject(-10000000000000000000.1).inject(-1E19D).inject(-1E308D).inject('neg float')
[  "neg float",  1E+308,  1E+19,  1E+19,  100.1 ]
✔ executed in 00:00:00.1459430. 5 results. 0 characters.

gremlin> g.V('').inject(1E+19)
✖ ScriptEvaluationError:
ActivityId : f2678603-5fbd-4108-aee8-b5a59983eea1
ExceptionType : GraphCompileException
ExceptionMessage :
	Gremlin Query Compilation Error: Script compile error: Unexpected error encountered during query compilation
Source : Microsoft.Azure.Graphs
	GremlinRequestId : f2678603-5fbd-4108-aee8-b5a59983eea1
	Context : graphcompute
	Scope : graphparse-translate-outer
	GraphStatusCode : ScriptEvaluationError
	HResult : 0x80131500

ActivityId : f2678603-5fbd-4108-aee8-b5a59983eea1
ExceptionType : FormatException
ExceptionMessage :
	Input string was not in a correct format.
Source : mscorlib
	HResult : 0x80131537

on the azure portal data explorer the results were the same.

Confirmed that these bugs exists:

Negative signs being lost on ints/floats
Failing to parse numbers with exponents.

Fixes for these issues are in-progress. Note that the fixes will be applied to the Gremlin Server (so accessible via Gremlin.NET) and won't be available for Microsoft.Azure.Graphs.
Regarding syntax round trip, can you clarify the specific issue?
If this is referring to the following:
g.inject(10000000000000000000.1) = [1E+19]

The reason for this is that the implementation only support the core numeric types, which does not include BigDecimal. As is shown, CosmosDB Graph will parse BigDecimal values to a Double.
These are the supported constant types for CosmosDB Graph:

String
Int
Long
Float
Double
Boolean

These issues should now be resolved for the CosmosDB Gremlin server (via Gremlin.NET or other gremlin driver client).
As per my reply above, there are some cases that will still not round trip.


Signature Meta-constraint Concatenation

This PR adds functionality for concat'ing models defined in interfaces to models defined in modules

I'm going to amplify @joelburget 's concern at top-level here: it is true that the Pact design generally ignores model stuff at runtime. If it is feasible that the proving system will do the interface-model resolution instead, we should abandon this approach.


Can not convergence

Dear author, I have faced the similar problem when I train the net on davis 2016.
I used python main.py -c 0 -m train -t train -s models/vm_vgg.pth -p 416 790 -l 1e-5 -x to start training, and got the log as follow.
DEBUG [       davis.py:062]:	Number of specified sequences in davis dataset is 30
   INFO [     encoder.py:027]:	Using vgg as encoder, with upsample factor 1.0
  DEBUG [       davis.py:135]:	Number of all pairs 74602
  DEBUG [        main.py:171]:	Train set size: 74416, Validation set size: 186
  DEBUG [  preprocess.py:049]:	Using image augmentation with the following transformations: ['hflip', 'random_crop', 'random_scale']
  DEBUG [        main.py:197]:	Number of trainable parameters in VideoMatch: 10585152
  DEBUG [        main.py:211]:	Using foreground threshold 0.5
  DEBUG [        main.py:212]:	Running untrained VideoMatch on validation set...
  DEBUG [        main.py:225]:	Untrained Videomatch IOU on validation set: 0.638
  DEBUG [        main.py:236]:	Using loss function dice
  DEBUG [        main.py:237]:	Training started...
  DEBUG [        main.py:242]:	Epoch: 	[1/1]
[W TensorIterator.cpp:924] Warning: Mixed memory format inputs detected while calling the operator. The operator will output channels_last tensor even if some of the inputs are not in channels_last format. (function operator())
  DEBUG [        main.py:274]:	Iter [   50/74416]:	avg loss = 0.8048,	avg val IOU = 0.720
  DEBUG [        main.py:274]:	Iter [  100/74416]:	avg loss = 0.7860,	avg val IOU = 0.752
  DEBUG [        main.py:274]:	Iter [  150/74416]:	avg loss = 0.7890,	avg val IOU = 0.757
  DEBUG [        main.py:274]:	Iter [  200/74416]:	avg loss = 0.7743,	avg val IOU = 0.770
  DEBUG [        main.py:274]:	Iter [  250/74416]:	avg loss = 0.7817,	avg val IOU = 0.769
  DEBUG [        main.py:274]:	Iter [  300/74416]:	avg loss = 0.7906,	avg val IOU = 0.769
  DEBUG [        main.py:274]:	Iter [  350/74416]:	avg loss = 0.7737,	avg val IOU = 0.733
  DEBUG [        main.py:274]:	Iter [  400/74416]:	avg loss = 0.7693,	avg val IOU = 0.476
  DEBUG [        main.py:274]:	Iter [  450/74416]:	avg loss = 0.7676,	avg val IOU = 0.288
  DEBUG [        main.py:274]:	Iter [  500/74416]:	avg loss = 0.8078,	avg val IOU = 0.216
  DEBUG [        main.py:274]:	Iter [  550/74416]:	avg loss = 0.7770,	avg val IOU = 0.181
  DEBUG [        main.py:274]:	Iter [  600/74416]:	avg loss = 0.8047,	avg val IOU = 0.152
  DEBUG [        main.py:274]:	Iter [  650/74416]:	avg loss = 0.7901,	avg val IOU = 0.191
  DEBUG [        main.py:274]:	Iter [  700/74416]:	avg loss = 0.8225,	avg val IOU = 0.174
  DEBUG [        main.py:274]:	Iter [  750/74416]:	avg loss = 0.8217,	avg val IOU = 0.169
  DEBUG [        main.py:274]:	Iter [  800/74416]:	avg loss = 0.8020,	avg val IOU = 0.187
  DEBUG [        main.py:274]:	Iter [  850/74416]:	avg loss = 0.7928,	avg val IOU = 0.191
  DEBUG [        main.py:274]:	Iter [  900/74416]:	avg loss = 0.8316,	avg val IOU = 0.184
  DEBUG [        main.py:274]:	Iter [  950/74416]:	avg loss = 0.8386,	avg val IOU = 0.198
  DEBUG [        main.py:274]:	Iter [ 1000/74416]:	avg loss = 0.8094,	avg val IOU = 0.183
  DEBUG [        main.py:274]:	Iter [ 1050/74416]:	avg loss = 0.7938,	avg val IOU = 0.192
  DEBUG [        main.py:274]:	Iter [ 1100/74416]:	avg loss = 0.7821,	avg val IOU = 0.182
  DEBUG [        main.py:274]:	Iter [ 1150/74416]:	avg loss = 0.8171,	avg val IOU = 0.178
  DEBUG [        main.py:274]:	Iter [ 1200/74416]:	avg loss = 0.7996,	avg val IOU = 0.137
  DEBUG [        main.py:274]:	Iter [ 1250/74416]:	avg loss = 0.8253,	avg val IOU = 0.130
  DEBUG [        main.py:274]:	Iter [ 1300/74416]:	avg loss = 0.8318,	avg val IOU = 0.127
  DEBUG [        main.py:274]:	Iter [ 1350/74416]:	avg loss = 0.8340,	avg val IOU = 0.153
  DEBUG [        main.py:274]:	Iter [ 1400/74416]:	avg loss = 0.8016,	avg val IOU = 0.171
  DEBUG [        main.py:274]:	Iter [ 1450/74416]:	avg loss = 0.8026,	avg val IOU = 0.158
  DEBUG [        main.py:274]:	Iter [ 1500/74416]:	avg loss = 0.8261,	avg val IOU = 0.156
  DEBUG [        main.py:274]:	Iter [ 1550/74416]:	avg loss = 0.8373,	avg val IOU = 0.159
  DEBUG [        main.py:274]:	Iter [ 1600/74416]:	avg loss = 0.8490,	avg val IOU = 0.156
  DEBUG [        main.py:274]:	Iter [ 1650/74416]:	avg loss = 0.8260,	avg val IOU = 0.158
  DEBUG [        main.py:274]:	Iter [ 1700/74416]:	avg loss = 0.8072,	avg val IOU = 0.151
  DEBUG [        main.py:274]:	Iter [ 1750/74416]:	avg loss = 0.8316,	avg val IOU = 0.138
  DEBUG [        main.py:274]:	Iter [ 1800/74416]:	avg loss = 0.8281,	avg val IOU = 0.154
  DEBUG [        main.py:274]:	Iter [ 1850/74416]:	avg loss = 0.8112,	avg val IOU = 0.144
  DEBUG [        main.py:274]:	Iter [ 1900/74416]:	avg loss = 0.8384,	avg val IOU = 0.143
  DEBUG [        main.py:274]:	Iter [ 1950/74416]:	avg loss = 0.8355,	avg val IOU = 0.141
  DEBUG [        main.py:274]:	Iter [ 2000/74416]:	avg loss = 0.8488,	avg val IOU = 0.139
  DEBUG [        main.py:274]:	Iter [ 2050/74416]:	avg loss = 0.8240,	avg val IOU = 0.136

Hi, can you tell me your Pytorch version? I will look in to this and get back to you.

Glad to see your reply.
I used torch 1.6 to run your code.
additionally, numpy==1.19.2, opencv-python==<IP_ADDRESS>, pillow==7.2.0, python==3.7.9, torchvision==0.7.0.

Sorry for the late reply, I was a bit busy. I ran the training and I experience the same thing. Then I remembered that it's probably because the feature extraction layer (e.g VGG-19) can only improve so much before the extra training starts to ruining the learnt weights. You can avoid this by stopping the training before IOU on validation set starts to decline. You can also squeeze the extra performance by lowering the learning rate and continuing the training.
Original authors weren't really specific on how they trained the feature layer, but I suspect that there isn't a major performance boost with training because VideoMatch algorithm isn't end-to-end. Since a lot of postprocessing is done on the features (softmatching layer), gradients just don't map that well from the calculated output (segmentation mask) to the feature layer.
I might be wrong and there is a nasty bug in the code/model, but I doubt it since the IOU does improve until a certain pivoting point.
This is an output with a smaller learning rate, decrease in IOU started around the same value (approx. 0.763) as in your output
/usr/bin/python3.6 /home/stas/gits/VideoMatch/main.py -c 0 -m train -t train -s models/vm_vgg.pth -p 416 790 -l 1e-6 -x
  DEBUG [       davis.py:061]:	Number of specified sequences in davis dataset is 30
   INFO [     encoder.py:027]:	Using vgg as encoder, with upsample factor 1.0
  DEBUG [       davis.py:133]:	Number of all pairs 74683
  DEBUG [        main.py:171]:	Train set size: 74497, Validation set size: 186
  DEBUG [  preprocess.py:050]:	Using image augmentation with the following transformations: ['hflip', 'random_crop', 'random_scale']
  DEBUG [        main.py:197]:	Number of trainable parameters in VideoMatch: 10585152
  DEBUG [        main.py:211]:	Using foreground threshold 0.5
  DEBUG [        main.py:212]:	Running untrained VideoMatch on validation set...
  DEBUG [        main.py:225]:	Untrained Videomatch IOU on validation set: 0.612
  DEBUG [        main.py:236]:	Using loss function dice
  DEBUG [        main.py:237]:	Training started...
  DEBUG [        main.py:242]:	Epoch: 	[1/1]
  DEBUG [        main.py:275]:	Iter [   50/74497]:	avg loss = 0.8175,	avg val IOU = 0.629
  DEBUG [        main.py:275]:	Iter [  100/74497]:	avg loss = 0.8072,	avg val IOU = 0.644
  DEBUG [        main.py:275]:	Iter [  150/74497]:	avg loss = 0.8153,	avg val IOU = 0.656
  DEBUG [        main.py:275]:	Iter [  200/74497]:	avg loss = 0.7951,	avg val IOU = 0.666
  DEBUG [        main.py:275]:	Iter [  250/74497]:	avg loss = 0.8061,	avg val IOU = 0.676
  DEBUG [        main.py:275]:	Iter [  300/74497]:	avg loss = 0.8031,	avg val IOU = 0.685
  DEBUG [        main.py:275]:	Iter [  350/74497]:	avg loss = 0.8079,	avg val IOU = 0.685
  DEBUG [        main.py:275]:	Iter [  400/74497]:	avg loss = 0.8047,	avg val IOU = 0.690
  DEBUG [        main.py:275]:	Iter [  450/74497]:	avg loss = 0.7833,	avg val IOU = 0.700
  DEBUG [        main.py:275]:	Iter [  500/74497]:	avg loss = 0.7986,	avg val IOU = 0.703
  DEBUG [        main.py:275]:	Iter [  550/74497]:	avg loss = 0.8015,	avg val IOU = 0.707
  DEBUG [        main.py:275]:	Iter [  600/74497]:	avg loss = 0.8011,	avg val IOU = 0.712
  DEBUG [        main.py:275]:	Iter [  650/74497]:	avg loss = 0.8351,	avg val IOU = 0.708
  DEBUG [        main.py:275]:	Iter [  700/74497]:	avg loss = 0.7972,	avg val IOU = 0.716
  DEBUG [        main.py:275]:	Iter [  750/74497]:	avg loss = 0.8326,	avg val IOU = 0.718
  DEBUG [        main.py:275]:	Iter [  800/74497]:	avg loss = 0.8294,	avg val IOU = 0.721
  DEBUG [        main.py:275]:	Iter [  850/74497]:	avg loss = 0.8250,	avg val IOU = 0.717
  DEBUG [        main.py:275]:	Iter [  900/74497]:	avg loss = 0.7802,	avg val IOU = 0.726
  DEBUG [        main.py:275]:	Iter [  950/74497]:	avg loss = 0.7792,	avg val IOU = 0.724
  DEBUG [        main.py:275]:	Iter [ 1000/74497]:	avg loss = 0.8033,	avg val IOU = 0.723
  DEBUG [        main.py:275]:	Iter [ 1050/74497]:	avg loss = 0.7924,	avg val IOU = 0.728
  DEBUG [        main.py:275]:	Iter [ 1100/74497]:	avg loss = 0.8122,	avg val IOU = 0.731
  DEBUG [        main.py:275]:	Iter [ 1150/74497]:	avg loss = 0.8162,	avg val IOU = 0.733
  DEBUG [        main.py:275]:	Iter [ 1200/74497]:	avg loss = 0.8297,	avg val IOU = 0.732
  DEBUG [        main.py:275]:	Iter [ 1250/74497]:	avg loss = 0.7930,	avg val IOU = 0.734
  DEBUG [        main.py:275]:	Iter [ 1300/74497]:	avg loss = 0.8186,	avg val IOU = 0.740
  DEBUG [        main.py:275]:	Iter [ 1350/74497]:	avg loss = 0.7819,	avg val IOU = 0.746
  DEBUG [        main.py:275]:	Iter [ 1400/74497]:	avg loss = 0.7893,	avg val IOU = 0.751
  DEBUG [        main.py:275]:	Iter [ 1450/74497]:	avg loss = 0.8196,	avg val IOU = 0.746
  DEBUG [        main.py:275]:	Iter [ 1500/74497]:	avg loss = 0.7867,	avg val IOU = 0.744
  DEBUG [        main.py:275]:	Iter [ 1550/74497]:	avg loss = 0.8163,	avg val IOU = 0.744
  DEBUG [        main.py:275]:	Iter [ 1600/74497]:	avg loss = 0.8149,	avg val IOU = 0.748
  DEBUG [        main.py:275]:	Iter [ 1650/74497]:	avg loss = 0.7770,	avg val IOU = 0.745
  DEBUG [        main.py:275]:	Iter [ 1700/74497]:	avg loss = 0.8143,	avg val IOU = 0.745
  DEBUG [        main.py:275]:	Iter [ 1750/74497]:	avg loss = 0.7975,	avg val IOU = 0.746
  DEBUG [        main.py:275]:	Iter [ 1800/74497]:	avg loss = 0.7909,	avg val IOU = 0.750
  DEBUG [        main.py:275]:	Iter [ 1850/74497]:	avg loss = 0.8029,	avg val IOU = 0.745
  DEBUG [        main.py:275]:	Iter [ 1900/74497]:	avg loss = 0.7833,	avg val IOU = 0.752
  DEBUG [        main.py:275]:	Iter [ 1950/74497]:	avg loss = 0.7982,	avg val IOU = 0.748
  DEBUG [        main.py:275]:	Iter [ 2000/74497]:	avg loss = 0.8091,	avg val IOU = 0.752
  DEBUG [        main.py:275]:	Iter [ 2050/74497]:	avg loss = 0.7764,	avg val IOU = 0.753
  DEBUG [        main.py:275]:	Iter [ 2100/74497]:	avg loss = 0.7767,	avg val IOU = 0.754
  DEBUG [        main.py:275]:	Iter [ 2150/74497]:	avg loss = 0.7678,	avg val IOU = 0.756
  DEBUG [        main.py:275]:	Iter [ 2200/74497]:	avg loss = 0.7805,	avg val IOU = 0.749
  DEBUG [        main.py:275]:	Iter [ 2250/74497]:	avg loss = 0.8062,	avg val IOU = 0.749
  DEBUG [        main.py:275]:	Iter [ 2300/74497]:	avg loss = 0.7941,	avg val IOU = 0.749
  DEBUG [        main.py:275]:	Iter [ 2350/74497]:	avg loss = 0.7949,	avg val IOU = 0.751
  DEBUG [        main.py:275]:	Iter [ 2400/74497]:	avg loss = 0.8029,	avg val IOU = 0.754
  DEBUG [        main.py:275]:	Iter [ 2450/74497]:	avg loss = 0.7896,	avg val IOU = 0.753
  DEBUG [        main.py:275]:	Iter [ 2500/74497]:	avg loss = 0.7780,	avg val IOU = 0.763
  DEBUG [        main.py:275]:	Iter [ 2550/74497]:	avg loss = 0.8074,	avg val IOU = 0.752
  DEBUG [        main.py:275]:	Iter [ 2600/74497]:	avg loss = 0.7871,	avg val IOU = 0.750
  DEBUG [        main.py:275]:	Iter [ 2650/74497]:	avg loss = 0.7960,	avg val IOU = 0.752
  DEBUG [        main.py:275]:	Iter [ 2700/74497]:	avg loss = 0.7885,	avg val IOU = 0.751
  DEBUG [        main.py:275]:	Iter [ 2750/74497]:	avg loss = 0.7912,	avg val IOU = 0.746
  DEBUG [        main.py:275]:	Iter [ 2800/74497]:	avg loss = 0.7845,	avg val IOU = 0.750
  DEBUG [        main.py:275]:	Iter [ 2850/74497]:	avg loss = 0.8038,	avg val IOU = 0.743
  DEBUG [        main.py:275]:	Iter [ 2900/74497]:	avg loss = 0.7579,	avg val IOU = 0.742
  DEBUG [        main.py:275]:	Iter [ 2950/74497]:	avg loss = 0.7764,	avg val IOU = 0.741
  DEBUG [        main.py:275]:	Iter [ 3000/74497]:	avg loss = 0.7814,	avg val IOU = 0.734
  DEBUG [        main.py:275]:	Iter [ 3050/74497]:	avg loss = 0.7628,	avg val IOU = 0.729
  DEBUG [        main.py:275]:	Iter [ 3100/74497]:	avg loss = 0.7693,	avg val IOU = 0.727
  DEBUG [        main.py:275]:	Iter [ 3150/74497]:	avg loss = 0.7947,	avg val IOU = 0.722
  DEBUG [        main.py:275]:	Iter [ 3200/74497]:	avg loss = 0.7648,	avg val IOU = 0.718
  DEBUG [        main.py:275]:	Iter [ 3250/74497]:	avg loss = 0.7882,	avg val IOU = 0.707
  DEBUG [        main.py:275]:	Iter [ 3300/74497]:	avg loss = 0.7838,	avg val IOU = 0.702
  DEBUG [        main.py:275]:	Iter [ 3350/74497]:	avg loss = 0.7685,	avg val IOU = 0.693
  DEBUG [        main.py:275]:	Iter [ 3400/74497]:	avg loss = 0.7991,	avg val IOU = 0.677
  DEBUG [        main.py:275]:	Iter [ 3450/74497]:	avg loss = 0.7731,	avg val IOU = 0.676
  DEBUG [        main.py:275]:	Iter [ 3500/74497]:	avg loss = 0.8045,	avg val IOU = 0.650
  DEBUG [        main.py:275]:	Iter [ 3550/74497]:	avg loss = 0.8071,	avg val IOU = 0.649
  DEBUG [        main.py:275]:	Iter [ 3600/74497]:	avg loss = 0.7534,	avg val IOU = 0.636
  DEBUG [        main.py:275]:	Iter [ 3650/74497]:	avg loss = 0.7950,	avg val IOU = 0.617
  DEBUG [        main.py:275]:	Iter [ 3700/74497]:	avg loss = 0.7746,	avg val IOU = 0.596
  DEBUG [        main.py:275]:	Iter [ 3750/74497]:	avg loss = 0.7747,	avg val IOU = 0.578
  DEBUG [        main.py:275]:	Iter [ 3800/74497]:	avg loss = 0.7637,	avg val IOU = 0.542
  DEBUG [        main.py:275]:	Iter [ 3850/74497]:	avg loss = 0.7707,	avg val IOU = 0.530
  DEBUG [        main.py:275]:	Iter [ 3900/74497]:	avg loss = 0.7455,	avg val IOU = 0.491
  DEBUG [        main.py:275]:	Iter [ 3950/74497]:	avg loss = 0.7879,	avg val IOU = 0.492
  DEBUG [        main.py:275]:	Iter [ 4000/74497]:	avg loss = 0.7835,	avg val IOU = 0.479
  DEBUG [        main.py:275]:	Iter [ 4050/74497]:	avg loss = 0.7590,	avg val IOU = 0.460
  DEBUG [        main.py:275]:	Iter [ 4100/74497]:	avg loss = 0.7949,	avg val IOU = 0.430
  DEBUG [        main.py:275]:	Iter [ 4150/74497]:	avg loss = 0.7917,	avg val IOU = 0.412
  DEBUG [        main.py:275]:	Iter [ 4200/74497]:	avg loss = 0.7633,	avg val IOU = 0.405
  DEBUG [        main.py:275]:	Iter [ 4250/74497]:	avg loss = 0.7746,	avg val IOU = 0.381
  DEBUG [        main.py:275]:	Iter [ 4300/74497]:	avg loss = 0.7952,	avg val IOU = 0.363
  DEBUG [        main.py:275]:	Iter [ 4350/74497]:	avg loss = 0.8045,	avg val IOU = 0.344
  DEBUG [        main.py:275]:	Iter [ 4400/74497]:	avg loss = 0.8042,	avg val IOU = 0.338
  DEBUG [        main.py:275]:	Iter [ 4450/74497]:	avg loss = 0.7878,	avg val IOU = 0.331
  DEBUG [        main.py:275]:	Iter [ 4500/74497]:	avg loss = 0.7749,	avg val IOU = 0.315
  DEBUG [        main.py:275]:	Iter [ 4550/74497]:	avg loss = 0.7517,	avg val IOU = 0.297
  DEBUG [        main.py:275]:	Iter [ 4600/74497]:	avg loss = 0.7789,	avg val IOU = 0.288
  DEBUG [        main.py:275]:	Iter [ 4650/74497]:	avg loss = 0.8143,	avg val IOU = 0.273
  DEBUG [        main.py:275]:	Iter [ 4700/74497]:	avg loss = 0.7650,	avg val IOU = 0.265
  DEBUG [        main.py:275]:	Iter [ 4750/74497]:	avg loss = 0.7550,	avg val IOU = 0.259
  DEBUG [        main.py:275]:	Iter [ 4800/74497]:	avg loss = 0.8145,	avg val IOU = 0.255
  DEBUG [        main.py:275]:	Iter [ 4850/74497]:	avg loss = 0.7832,	avg val IOU = 0.247
  DEBUG [        main.py:275]:	Iter [ 4900/74497]:	avg loss = 0.7719,	avg val IOU = 0.242
  DEBUG [        main.py:275]:	Iter [ 4950/74497]:	avg loss = 0.7935,	avg val IOU = 0.232
  DEBUG [        main.py:275]:	Iter [ 5000/74497]:	avg loss = 0.8060,	avg val IOU = 0.238
  DEBUG [        main.py:275]:	Iter [ 5050/74497]:	avg loss = 0.8017,	avg val IOU = 0.231
  DEBUG [        main.py:275]:	Iter [ 5100/74497]:	avg loss = 0.8200,	avg val IOU = 0.229
  DEBUG [        main.py:275]:	Iter [ 5150/74497]:	avg loss = 0.7977,	avg val IOU = 0.225
  DEBUG [        main.py:275]:	Iter [ 5200/74497]:	avg loss = 0.8060,	avg val IOU = 0.219
  DEBUG [        main.py:275]:	Iter [ 5250/74497]:	avg loss = 0.8096,	avg val IOU = 0.215
  DEBUG [        main.py:275]:	Iter [ 5300/74497]:	avg loss = 0.8136,	avg val IOU = 0.213
  DEBUG [        main.py:275]:	Iter [ 5350/74497]:	avg loss = 0.8170,	avg val IOU = 0.203
  DEBUG [        main.py:275]:	Iter [ 5400/74497]:	avg loss = 0.7728,	avg val IOU = 0.198
  DEBUG [        main.py:275]:	Iter [ 5450/74497]:	avg loss = 0.8246,	avg val IOU = 0.197
  DEBUG [        main.py:275]:	Iter [ 5500/74497]:	avg loss = 0.7930,	avg val IOU = 0.195
  DEBUG [        main.py:275]:	Iter [ 5550/74497]:	avg loss = 0.8133,	avg val IOU = 0.194
  DEBUG [        main.py:275]:	Iter [ 5600/74497]:	avg loss = 0.8287,	avg val IOU = 0.189
  DEBUG [        main.py:275]:	Iter [ 5650/74497]:	avg loss = 0.8076,	avg val IOU = 0.185
  DEBUG [        main.py:275]:	Iter [ 5700/74497]:	avg loss = 0.8218,	avg val IOU = 0.182
  DEBUG [        main.py:275]:	Iter [ 5750/74497]:	avg loss = 0.8040,	avg val IOU = 0.174
  DEBUG [        main.py:275]:	Iter [ 5800/74497]:	avg loss = 0.8230,	avg val IOU = 0.170
  DEBUG [        main.py:275]:	Iter [ 5850/74497]:	avg loss = 0.8099,	avg val IOU = 0.169
  DEBUG [        main.py:275]:	Iter [ 5900/74497]:	avg loss = 0.8264,	avg val IOU = 0.168
  DEBUG [        main.py:275]:	Iter [ 5950/74497]:	avg loss = 0.8149,	avg val IOU = 0.164
  DEBUG [        main.py:275]:	Iter [ 6000/74497]:	avg loss = 0.8119,	avg val IOU = 0.163
   INFO [        main.py:205]:	Ctrl+c caught, stopping the training and saving the model...
   INFO [        main.py:290]:	Saving model to path models/vm_vgg.pth

Process finished with exit code 137 (interrupted by signal 9: SIGKILL)


80 separate large vocabularies use custom validation from small vocabularies and subsets use codegen

This PR refactors the mouse_anatomy.py file to use both codegen and external validation through a registry. This is a breaking change:

ONE_OF no longer exists because it would require running the registry checks for all values. Instead the MouseAnatomyModel should be used directly as the type
Because of the ^ change, types that used to use sub groups like EMG_MUSCLES.ONE_OF now have to use the base type MouseAnatomyModel, but users can use the group type MouseEmgMuscles to get easy access to the limited set of options.
ALL no longer exists for the same reason as above. In general though nobody should be looping over all 10,000 records of the index so this is probably a good thing
_MouseAnatomyModel was renamed to MouseAnatomyModel since the type now gets used in a lot of places

One of the unit tests now runs as an integration test because it validates the models. This is not ideal but I think we need the integration tests somewhere.
This is a draft PR until we decide how we want to handle this, it should not be merged into main!

What is the expected way users will use these classes? I get some pydantic errors trying this snippet:
from pydantic import BaseModel, Field
from aind_data_schema_models.mouse_anatomy import MouseBodyParts, MouseEmgMuscles

class MyomatrixContact(BaseModel):
    body_part: MouseBodyParts = Field(..., title="Body part of contact insertion")
    muscle: MouseEmgMuscles = Field(..., title="Muscle of contact insertion")

Like this?
from pydantic import BaseModel, Field
from aind_data_schema_models.mouse_anatomy import MouseBodyParts, MouseEmgMuscles, MouseAnatomyModel

class MyomatrixContact(BaseModel):
    body_part: MouseAnatomyModel = Field(..., title="Body part of contact insertion")
    muscle: MouseAnatomyModel = Field(..., title="Muscle of contact insertion")

my_info = MyomatrixContact(body_bart=MouseBodyParts.HEAD, muscle=MouseEmgMuscles.DELTOID)

Like this?
from pydantic import BaseModel, Field
from aind_data_schema_models.mouse_anatomy import MouseBodyParts, MouseEmgMuscles, MouseAnatomyModel

class MyomatrixContact(BaseModel):
    body_part: MouseAnatomyModel = Field(..., title="Body part of contact insertion")
    muscle: MouseAnatomyModel = Field(..., title="Muscle of contact insertion")

my_info = MyomatrixContact(body_bart=MouseBodyParts.HEAD, muscle=MouseEmgMuscles.DELTOID)

Exactly. It's not as clean as the old system but I don't see a clever way to get the behavior we had before because we can't pre-generate the Union[]

One quick comment - I wonder MouseBodyParts.HEAD should rather e MouseBodyParts.get('head')? Just so it's more clear that this is doing something, not just referring to a static object.

One quick comment - I wonder MouseBodyParts.HEAD should rather e MouseBodyParts.get('head')? Just so it's more clear that this is doing something, not just referring to a static object.

Because we overwrite getattribute .get will work fine, but if we remove the attributes then we lose tab completion in the editor which is the whole point of having the group objects. By having tab completion someone who only ever uses a small subset of all 10,000 anatomy parts can find just that subset easily.

I'll post an example of an alternative implementation. I'm fine moving forward with either way:
from enum import Enum
from pydantic import BaseModel, field_validator, ConfigDict, Field
from aind_data_schema_models.registries import Registry
import requests

class MouseBodyParts(str, Enum):
    FORELIMB = "forelimb"
    HEAD = "head"
    HINDLIMB = "hindlimb"
    NECK = "neck"
    TAIL = "tail"
    TRUNK = "trunk"

class MouseAnatomyModel(BaseModel):
    model_config = ConfigDict(frozen=True)
    name: str
    registry: Registry.ONE_OF
    registry_identifier: str
    @classmethod
    def get_from_emapa(cls, name: str):
        params = {
            "q": name,
            "ontology": "emapa",
            "type": "class"
        }
        base_url = "https://www.ebi.ac.uk/ols/api/search"
        response = requests.get(base_url, params=params)
        results = response.json()
        exact_matches = [{"iri": entry["iri"],"label": entry["label"],} for entry in results.get("response", {}).get("docs", []) if entry["label"].lower() == name]
        if exact_matches:
            return cls(name=name, registry=Registry.EMAPA, emapa_id=exact_matches[0]["iri"].split("_")[1])
        else:
            return cls(name=name, registry=Registry.EMAPA, emapa_id=None)

# Can also make a child class for Muscles
class MouseBodyPartModel(MouseAnatomyModel):
    @field_validator("name", mode="after")
    def validate_name(cls, v):
        if v.name not in [v.value for v in MouseBodyParts]:
            raise AttributeError("Invalid Body Part")
        else:
            return v

class MyomatrixContact(BaseModel):
    body_part: MouseBodyPartModel = Field(..., title="Body part of contact insertion")

body_part = MouseBodyPartModel.get_from_emapa(name=MouseBodyParts.HEAD)
my_info = MyomatrixContact(body_part=body_part)

# Raises a validation warning because the body part isn't in the list
MouseBodyPartModel(name='head123', registry={'name':'Edinburgh Mouse Atlas Project', 'abbreviation':'EMAPA'}, registry_identifier='31858')

I'll post an example of an alternative implementation. I'm fine moving forward with either way:
from enum import Enum
from pydantic import BaseModel, field_validator, ConfigDict, Field
from aind_data_schema_models.registries import Registry
import requests

class MouseBodyParts(str, Enum):
    FORELIMB = "forelimb"
    HEAD = "head"
    HINDLIMB = "hindlimb"
    NECK = "neck"
    TAIL = "tail"
    TRUNK = "trunk"

class MouseAnatomyModel(BaseModel):
    model_config = ConfigDict(frozen=True)
    name: str
    registry: Registry.ONE_OF
    registry_identifier: str
    @classmethod
    def get_from_emapa(cls, name: str, base_url: str = "https://www.ebi.ac.uk/ols/api/search"):
        params = {
            "q": name,
            "ontology": "emapa",
            "type": "class"
        }
        response = requests.get(base_url, params=params)
        results = response.json()
        exact_matches = [{"iri": entry["iri"],"label": entry["label"],} for entry in results.get("response", {}).get("docs", []) if entry["label"].lower() == name]
        if exact_matches:
            return cls(name=name, registry=Registry.EMAPA, emapa_id=exact_matches[0]["iri"].split("_")[1])
        else:
            return cls(name=name, registry=Registry.EMAPA, emapa_id=None)

# Can also make a child class for Muscles
class MouseBodyPartModel(MouseAnatomyModel):
    @field_validator("name", mode="after")
    def validate_name(cls, v):
        if v.name not in [v.value for v in MouseBodyParts]:
            raise AttributeError("Invalid Body Part")
        else:
            return v

class MyomatrixContact(BaseModel):
    body_part: MouseBodyPartModel = Field(..., title="Body part of contact insertion")

body_part = MouseBodyPartModel.get_from_emapa(name=MouseBodyParts.HEAD)
my_info = MyomatrixContact(body_part=body_part)

# Raises a validation warning because the body part isn't in the list
MouseBodyPartModel(name='head123', registry={'name':'Edinburgh Mouse Atlas Project', 'abbreviation':'EMAPA'}, registry_identifier='31858')

If we do this, I would anticipate users trying to use the enum classes in place of the models anywhere that we don't enforce the model type. I think that's probably bad? You also don't have the IDE tab completion in this version which I think should be required for any option we choose.


[Release] sdk/resourcemanager/support/armsupport/0.1.0

…m spec commit: b8cae07c799a07f2e779a9e301f639e9b85124b1

[ ] The purpose of this PR is explained in this or a referenced issue.
[ ] The PR does not update generated files.

These files are managed by the codegen framework at Azure/autorest.go.

[ ] Tests are included and/or updated for code changes.
[ ] Updates to CHANGELOG.md are included.
[ ] MIT license headers are included in each file.

/azp run prepare-pipelines

/azp run go - armsupport - ci

/check-enforcer override


storage: Fix mixed samples handling in sampleRing

Two issues are fixed here, that lead to the same problem:

If newSampleRing is called with an unknown ValueType including ValueNone, we have initialized the interface buffer (iBuf). However, we would still use a specialized buffer for the first sample, opportunistically assuming that we might still not encounter mixed samples and we should go down the more efficient road.

If the sampleRing is reset, we leave all buffers alone, including iBuf, which is generally fine, but not for iBuf, see below.

In both cases, iBuf already contains values, but we will fill one of the specialized buffers first. Once we then actually encounter mixed samples, the content of the specialized buffer is copied into iBuf using append. That's by itself the right idea because iBuf might be nil, and even if not, it might or might not have the right capacity. However, this approach assumes that iBuf is empty, or more precisely has a length of zero.
This commit makes sure that iBuf does not get needlessly initialized in newSampleRing and that it is emptied upon reset.
A test case is added to demonstrate both issues above.

This bug can crash Prometheus, so we better merge it into release-2.48 and don't wait for 2.49 to go out to the public.


Typo in ApiCallback.onSucess()

Should change to onSuccess()

Please clean up this PR. A lot of unrelated changes

I am sorry. I am trying to clean up. Still trying to figure out how to create PR with only certain request but it went out prematurely.

Let me close this and redo this PR


Create a PredictPytorch operator that can be configured by the user to use the correct Triton backend:

https://github.com/NVIDIA-Merlin/systems/pull/151


UX fix: text update

*Issue:*https://github.com/bcgov/entity/issues/18679
Description of changes:
update the instructional text.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the namex license (Apache 2.0).

/gcbrun

Temporary Url for review: https://bcregistry-btr-dev--pr-80-8q3azrmb.web.app


Exported Transitions Not working.

So if you export the game with the default code the transitions will not work, as the files within the shader_patterns folder have all had a ".import" file type added.
Here's the code to work around this, but it does require the adding of a build feature called "standalone" to the export.
Line 30: scene_manager.gd

if not OS.has_feature("standalone"):
if file_folder.get_extension() == "png":
_patterns[file_folder.replace("."+file_folder.get_extension(), "")] = load(root_path + file_folder)
else:
if file_folder.get_extension() == "import":
_patterns[file_folder.replace("."+file_folder.get_extension(), "").replace(".png", "")] = load(root_path + file_folder.replace("."+file_folder.get_extension(), ""))

Not sure if this helps any, but I hope it does.

Thanks for creating this issue.
Let me see if there is a better way because I really want user to just use this addon and have fun with the tool and do not worry about the additional addon installation and this solution that you provided, complexes addon setup for user just a little bit.
I think I can fix this in another way...
I'll be back and hopefully with some resolving commits.

This is now fixed.
no need to setup anything, just use the latest versions:
godot3: https://github.com/maktoobgar/scene_manager/releases/tag/2.2.1
godot4: https://github.com/maktoobgar/scene_manager/releases/tag/3.4.1

And that resolved the issue! thank you very much.


A wild bug appeared!

Run In Bash If Your Running in Replit , It will work 😄

Same here

@Takiyo0

what version of nodejs are you using

i'm sorry for being late, repl.it is using node v12 and we need at least v14. You can try installing node v14 in repl.it by doing (we will use 14.5.1):

open repl shell, type npm i<EMAIL_ADDRESS>Now, run your bot with npx node index.js


add ability to revoke refresh tokens in user API

I'm opening this in place of #329 because last time I forced a push for #329 all the inline comments disappeared.
This PR adds API endpoints for listening clients with refresh tokens and revoking refresh tokens. Work needed for #261 (note we've considered splitting the UI work from the API work needed for #261).

going to bump this guy as well

please add some tests, and consider the v1.json change. otherwise looks good

LGTM

Still need integration tests :)

@bobbyrullo integration tests added. Found another issue in my JSON file that generated an invalid client. Need to squash.

woo hoo lgTM


[Bug]: Mods successfully imported, but not showing up.

Operating System
Windows 10
BG3 Mod Manager Version
10.0.19045
BG3 Game Version
Full Release Patch 3
Bug Summary
I am VERY new to modding, but I have been sure to religiously follow tutorials.
I have been able to get the mod manager and get Mod Fixer and Improved UI in the system. (Under overrides, I hope that's where they're supposed to be.)
Then, when I try to import another mod into the game, It shows (at the bottom in green) "Successfully imported ___".
It does not show up under Active or inactive mods. I'm not sure what I'm doing wrong.
Links
No response

This fix worked for me:
Manually extracting the PAK files from the downloaded .zip into the Mod folder (C:\Users\Name\AppData\Local\Larian Studios\Baldur's Gate 3\Mods) enabled them to show up as inactive mods. You'll need a program like 7zip to extract the file, and you also might need to enable viewing hidden folders to access the Mods folder.

Are you importing zips with pak files inside? Mods like that WASD one have special instructions to extract them to the bin folder of the game, so that sort of stuff isn't supported. "Import Mod" basically means "Copy paks (even if they're in zip) to the mods folder"


Menu: fix that the SubMenu parent component is not a Menu component cause stack overflow

Please make sure these boxes are checked before submitting your PR, thank you!

[x] Make sure you follow Element's contributing guide (中文 | English | Español | Français).
[x] Make sure you are merging your commits to dev branch.
[x] Add some descriptions and refer relative issues for you PR.

When a menu needs to be generated recursively, submenu is wrapped as a recursive component, and submenu's mouseenter event can cause a stack overflow. Because the parent component of submenu is the recursive component, and its $el is submenu itself. And the mouseleave event will not trigger as expected submenu's mouseleave event.
problem link：https://codepen.io/SAMAPENG/pen/MWgjXaJ

从 demo 来看，跟官网的例子差不多。两者之间的区别是？

官网例子中所有el-submenu都是直接写在el-menu组件下面，这样el-submenu取到的$parent为el-menu组件或者另一个el-submenu，例子中el-submenu是写在一个组件submenu中，只是根元素为el-submenu，这样el-submenu取到的$parent为此submenu组件而并非el-menu组件或者el-submenu组件。即el-submenu与el-menu隔了一个组件，造成$parent取值有误
------------------ 原始邮件 ------------------
发件人<EMAIL_ADDRESS>发送时间: 2019年9月2日(星期一) 中午11:50
收件人<EMAIL_ADDRESS>抄送: "xupeng"809584068<EMAIL_ADDRESS>主题: Re: [ElemeFE/element] Menu: fix that the SubMenu parent component isnot a Menu component cause stack overflow (#17133)
从 demo 来看，跟官网的例子差不多。两者之间的区别是？
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

为啥还不merge啊？？？？？？这个问题去年九月份就存在了，都2020年了。不merge也不解决啊？？我也出现相同问题了。子element-submenu 放到body中 很常见啊，这个爆栈报错信息真的看的很难受

merge啊


Add link to Promise.catch API docs

Fixes https://github.com/rescript-association/rescript-lang.org/issues/904
The API docs are sufficient, just wanted to add additional link there so people can find it.
//cc @fhammerschmidt

Thank you!


disable OverlayFS by default when working on ZFS

OverlayFS doesn't work on top of ZFS, thus using ZFS under /var/lib/rkt requires passing --no-overlay flag every time. This code detects ZFS and disables overlay automatically.

LGTM (but detection of ext/btrfs is unused and unnecessary/against agile principals).
@alban, @iaguis: maybe we could use this btrfs distinction for similar purpose, because AFAIR there are still issues with /var/lib/rkt on it, am I correct?

Cool! Thanks for the patch! Should we move this within SupportsOverlay, though, so that decision is centralised in one place?

@jonboulle I though about it, but SupportsOverlay() doesn't have access to getDataDir(). Alternatively I could pass the data dir path as a parameter to SupportsOverlay. Thoughts?

@jellonek According to docker sources overlayfs is broken over BTRFS, AUFS, ZFS and overlayfs. I shall include all these cases.

Afair some at redhat is working on this, but I don't know current status of this. For the moment - all these case inclusion seems like the right choice.

@kdomanski ping
any progress?

I finally found the time to revisit this. I added all cases to the PR.
@jonboulle I still cannot see a way for a clean merger with SupportsOverlay().

the code LGTM, but I don't think we can do this so broadly, because some of these combinations do work provided people are using sufficiently recent kernels (e.g. btrfs should work with 4.5.2+, overlay works with 4.2+)

@jonboulle I still cannot see a way for a clean merger with SupportsOverlay().

Perhaps we could rename that method to KernelSupportsOverlay, then it fits nicely alongside FSSupportsOverlay

@kdomanski to move forward, I suggest that you remove the btrfs/overlayfs lines from the conditional (so this PR will only disable overlay-on-zfs and overlay-on-aufs), and then we can follow up with a more nuanced check for btrfs/overlay in https://github.com/coreos/rkt/issues/2619 - does that sound reasonable?

Thanks for reworking it, LGTM.


status, discoverability

Two things that would improve discoverability of this repo

[ ] add.jl-suffix to the repository name
[ ] register the package

Looking at the commit history, it was not clear if this is still active and PRs would be accepted.

Thank you @jaakkor2
Yes, I need to work towards getting this registered.  I have some things I'd like get completed first.  I'm going to make a Issue tracking that I want to get implemented.
I'd like to get some usability feedback.  I'm not sure which is the best accessor functions, whether to simply treat it like a dictionary, or have get functions, like getprop and getdata.


feature: cloud storage for data files

I'd like to be able to store data files in object storage like S3. This works better for container management systems.

Thanks for the suggestion.
To be honest, I won’t personally be spending time implementing this, as I won’t be using this, and as I want the project to remain lean, with as few dependencies as at all possible.
That being said: I would certainly be willing to consider a PR for this.


jsdocs typedef don't work if declared before conditional unreachable code.

TS Template added by @mjbvz
TypeScript Version: 4.0.0-dev.20200702
Search Terms

jsdoc
typedef

VSCode Version: 1.46.1
OS Version: OSX Mojave 10.14.6

It seems that typedef are skipped if the code has some "conditional" unreachable code.
Have a look at the code examples with screenshots.
Steps to Reproduce:
Here are two versions of the code, one where jsdocs fails, and one when it works.
a. Buggy version, when typedef is declared before conditional / unreachable code

/**
 * @typedef {{id: string}} SomeType
 */

if (true) {
  throw new Error('Unreachable')
 // could be process.exit()
}

/**
 * @param {SomeType} someType
 */
function doSomething (someType) {
}

Result: no types

b. Working version, when typedef is declared after conditional / unreachable code

if (true) {
  throw new Error('Unreachable')
 // could be process.exit()
}

/**
 * @typedef {{id: string}} SomeType
 */

/**
 * @param {SomeType} someType
 */
function doSomething (someType) {
}

Result: we have types

Does this issue occur when all extensions are disabled?: Yes/No
(I don't know how to pass arguments to vs code on OSX)

This is true of any conditional. I think the parser isn't attaching or the binder isn't binding before if.

This fix for if is trivial, but all statements might have @typedef before them in JS files. The first step to fix this bug is to add jsdoc to all statements and measure whether there's a performance slowdown. If so, then it should only happen in JS files, and there needs to be conditional logic for the newly-jsdocced statements.
I created the branch add-jsdoc-to-all-js-statements for the prototype I made.


UAT and UI Review - Cart [Product Cross-Sell]

[x] Nama product

-- figma: line height 24px

-- staging: line height 18px

checked on staging and passed