SSE Support in @typespec/http

Our HTTP library should add support for defining SSE endpoints.
SSE is a fairly simple protocol where a service responds with a content type of text/event-stream and events are encoded in plaintext. Events are separated by two line breaks. Event data is provided by lines prefixed with data:. Events can optionally have an event type, in which case the data: lines for the event are preceded by an event: line.
Clients can consume SSE Events in web browsers using the EventSource API, but naturally we want to generate clients that make this more ergonomic.
Proposal
The gist of this proposal is to enable specs like the following:
@post op createCompletionStream(): SSEStream<Completion>;

model Completion {
  text: string;
}

Or, using event types:
@post op createCompletionStream(): SSEStream<Events>;

model Completion {
  text: string;
}

@sseEventTypes union Events {
  completion: Completion
  error: string;
}

SSEEvent<T, TEventEncoding = "application/json">
Defined as the following:
@sseStreamOf(T)
@sseEventContentType(TEventEncoding)
model SSEStream<T, TEventEncoding = "application/json"> {
  @header contentType: "text/event-stream";
  @body body: string;
}

This type enables emitters that don't understand SSE to still consume a vaguely appropriate shape including the proper content type and string body. If you understand SSE, the following decorators are used to determine the API shape.
SSE event payloads are just strings, but in practice they are often JSON encoded. This template allows customizing how an emitter should encode/decode the SSE event payloads. "application/json" is probably the most common so it is default, however "text/plain" would also be another common option.
@sseStreamOf(T: unknown) (internal-ish)
Indicates the payload of SSE events. If T has the @sseEventTypes decorator, it defines the event types and payloads. Otherwise, event types are not used and events all share the same provided shape.
@sseEventContentType(T: valueof string)
Customizes the content type of the event payload, enabling emitters to encode/decode different serializations.
May be applied to union variants when the union has the @sseEventTypes decorator. In such cases, it overrides the value set on the response payload (i.e. the value passed to the SSEStream template).
@sseEventTypes
Applied to a union, this decorator instructs the emitter to treat the union as defining the event types in the API. The union variant names are the event type, and the values are the corresponding payload.

Question raised by @witemple-msft is whether this should go into the HTTP library or another library like @typespec/sse. An argument for would be that SSE is defined by the HTML spec rather than HTTP. An argument against is that it's a lot of extra complexity, but maybe that's ok because SSE is a rarely used thing anyway?

In the OpenAI SSE API, the terminal SSE payload is the literal "[DONE]" which is not application/json. So I think we need another decorator to capture that terminal event, e.g. sseTerminateWith(T: string)?

In the OpenAI SSE API, the terminal SSE payload is the literal "[DONE]" which is not application/json. So I think we need another decorator to capture that terminal event, e.g. sseTerminateWith(T: string)?

This seems to complicate things significantly if we cannot even know what the serialization format of a given logical event is. Is with no discriminant like a content-type header. So this anonymous event in OpenAI sometimes being JSON and sometimes not is really a problem for emitter implementation.
In spec space, I don't really think we need a special "terminates" decorator and think we could probably just wrap it up into the union for event types:
union OpenAICompletionEvent {
  @sseContentType("text/plain")
  string;
  @sseContentType("application/json")
  ChatDelta
}

It's just going to be really annoying to actually implement this in an abstract way.

I agree the OpenAI API feels complicated and tbh I am not sure what was their motivation behind this design decision. Perhaps sending [DONE] makes it clear that the stream actually ended and the client is safe to assume so? We actually have a bug instance in one of those APIs where [DONE] is not sent eventually even though the stream closed. And the stream feels it has abruptly ended.
Generally speaking, the content-type value of text/event-stream lends to the fact that the data payload is an arbitrary textual value and having that terminal decorator designate a particular value as terminal should be enough in the OpenAI case.
Your proposal trades precision for simpler spec but I am not sure how an implementation should use it? If a payload is JSON, parse it and return it but if it is a string, return it as is? I don't think this is what we would want, at least in the OpenAI case.

Question raised by @witemple-msft is whether this should go into the HTTP library or another library like @typespec/sse. An argument for would be that SSE is defined by the HTML spec rather than HTTP. An argument against is that it's a lot of extra complexity, but maybe that's ok because SSE is a rarely used thing anyway?

I see both as valid arguments not to add it to the HTTP library. This is a pretty niche feature, isn't part of the HTTP spec, and we may want to break/version it without impacts to the more fundamental HTTP library.
@deyaaeldeen yes, I think the only logical interpretation of my proposal is that the emitters would have to write logic to test if the string value returned by the SSE endpoint satisfies the given sseContentType by checking them in order, so maybe this is an even more accurate representation of the OpenAI API:
union OpenAICompletionEvent {
  @sseContentType("text/plain")
 "[DONE]";
  @sseContentType("application/json")
  ChatDelta;
}

I think the implication here in TypeScript is that you have two event types: "[DONE]" and ChatDelta and the stream logic is that you check if the payload is the literal string "[DONE]" and then if not tries to parse it as JSON. If it fails then you have an error of some kind, and you end up with a natural representation of this stream that is returned to the customer, which is an AsyncIterable<"[DONE]" | ChatDelta>.
This isn't so much about simplifying the specification, but avoiding having must-understand metadata items that emitters have to check for in order to implement SSE correctly in general, especially one that I suspect only exists because of a quirk in OpenAI's implementation. Here we just say that [DONE] with a MIME type of text/plain is a possible event. We lose the information that OpenAI always produces it at the end (redundantly, since the fact that the stream is done is implied by the connection being closed), but I don't think we really want to open the door to encoding information about how events are sequenced in APIs. That is some kind of information about the protocol embedded within the SSE stream and not about the SSE stream itself.

Don't the LRO stuff in typespec encode information about when an operation terminates? It also feels annoying to have to deal with AsyncIterable<"[DONE]" | ChatDelta> when we know that [DONE] is nothing but a termination marker. Passing that up to the client doesn't seem to provide any real value. It is like putting the standard HTTP status code in the output model when we know for sure at this point the request succeeded.
I am not necessarily opposed to be less precise, I just wonder if it is worth it. We need to differentiate between what can be expressed and what is considered a best practice. If something is not a best practice, we can just lint against it.

I'm not sure I like the proposed solution above because I think we want sseEventContentType to only be applied to a model in the case where event types are not used, and otherwise to the event type union variants, where they are discriminated by the event type. Things feel too complex otherwise. Multiple content types for the same event payload just feels bad, and in the OpenAI case is due only to a weird quirk. I am not motivated to support it when there are reasonable workarounds.
Another issue is that allowing such specs invites a lot of ambiguity - it's super easy to specify events that cannot be discriminated by examining the payload - is data: [false] intended to be a string "[false]" or an array containing a boolean? The best we can do is check validity against the content type in the order they are declared, but again that feels not so good to me.
As to the reasonable workarounds, @deyaaeldeen suggested maybe we can silently ignore events whose payload cannot be understood with the given content type, allowing the spec to simply say that the event content type is application/json and the terminal marker is dropped on the floor. This feels borderline reasonable to me. Thoughts?

I worry a little bit that approach will make it impossible to distinguish intended spec violations from unintended spec violations. If the client sends something that doesn't parse because of some server memory error or whatever (however unlikely that may be), we'll just drop it rather than make the error observable.

Another issue is that allowing such specs invites a lot of ambiguity - it's super easy to specify events that cannot be discriminated by examining the payload - is data: [false] intended to be a string "[false]" or an array containing a boolean? The best we can do is check validity against the content type in the order they are declared, but again that feels not so good to me.

This is a much more general problem. If any API returns heterogenous data without a discriminant, it's a problem for emitters to figure out which one was actually returned without constructing complex runtime tests for type identity. And even with such tests, there are ambiguities. For example say an API returns some JSON model that looks like this:
model Foo {
  data: Bar | Baz;
}

model Bar {
  a?: string;
  b: safeint;
}

model Baz {
  a: string;
  b?: safeint;
}

Well, now I don't know how to parse data that has both a and b. It could be either, so we have to just pick one when deserializing at runtime or reject this case outright. For TypeScript specifically, we don't care, because the weakly-typed JSON object form is our native data format, and we just say it satisfies Bar | Baz, but in languages where you have to choose a representation of data where there is real type identity, that isn't possible. You have to actually construct the data by picking a representation, and we have to do that in some order or just reject APIs with undiscriminated ambiguities.

I've read the SSE spec now and while I wish the openai done message was JSON, its presence is more or less required for finite streams. According to the SSE spec:

Clients will reconnect if the connection is closed; a client can be told to stop reconnecting using the HTTP 204 No Content response code.

In other words, to prevent a client from attempting a reconnect only to be told to go away, the server must send some kind of terminal message. As such, we need to support that. Now ideally we would support this by requiring a dedicated event type, which would allow something like:
@post op createCompletionStream(): SSEStream<Events>;

model Completion {
  text: string;
}

@sseEventTypes union Events {
  completion: Completion
  @sseTerminalEvent
  done: void;
}

But of course this doesn't help with the OpenAI case. If we take Will's suggestion that allows multiple content types for a given event type (by allowing @sseEventContentType on union variants for a particular event type), then we can apply @sseTerminalEvent there as well:
@post op createCompletionStream(): SSEStream<Event>;

union Event {
  @sseContentType("text/plain")
  @sseTerminalEvent
  done: "[done]";

  completion: Completion; // default application/json I suppose
}

Perhaps this is the best option. The issue I have with @deyaaeldeen's sseTerminateWith is that it feels too special-casey for this one particular kind of terminal message. You could imagine doing something like this which doesn't feel super awful:
@post op createCompletionStream(): SSEStream<Event>;

union Event {
  @sseTerminalEvent
  done: { done: true }

  completion: { done: false, value: Completion };
}

Incidentally, another aspect of the specification we may want to support is lastEventId - SSE messages can have an id: attached which allows clients to resume where they left off when the connection drops. Not sure how to model this as yet.

For discriminant in union, the best an emitter can do is to check all the required properties at every level to speculate whether this object belongs to one of the union variants. However, we cannot prevent the service backend from returning extra properties, including those required properties that belong to other union variants, even if they were defined as discriminable by required properties.
It occurs to me whether it matters to which union variant those properties belong. As long as those properties share the same name and type, the deserialization process will remain the same. Furthermore, since these union variants are part of the union, any instance where the union is used can be replaced by any one of the union variants.
There may be cases where we have to narrow down the type? not sure how the typespec would look like in that cases ?
what I am worried about is properties have the same name and type but differ in format, one example would be a union of bytes | utcDateTime.  As suggested here https://github.com/microsoft/typespec/issues/2345
Also, FYI, other languages emitters currently don't have plans to support unions, they will simply generate protocol methods for it, and will have to add manual customization if they decide to support it in DPG. Their premise is that they will not take any action to verify if an object satisfies the types of other union variants as long as they find one match.

Notes:

LastEventId: how to support?
Library: @typespec/server-sent-events
Ok not supporting general streaming base layer for now, hope that we will layer on top later.
Drop sse prefix.
Need to discuss streaming all-up, folding in multipart and chunked encoding, AMQP, CloudEvents/eventgrid/etc.

I like the idea of anonymous and named union to express event payloads. It's simple and intuitive. But the concern is for named union, it seems that different places mean different interpretation. For example the name would be an discriminator for discriminated union, sometimes it could be just an information for normal union. Here the named union is more like record, it is key-value pairs, key would be event type. Named union is not union?
Also I was wondering how the breaking change evolution would be, taking the above example

In v1, we have event types as Completion
In v2, we will add an event type for that Completion, would this be a breaking?
In v3, we will add a 2nd event type e.g. Foo, would this be a breaking?
In v4, we will add the 3ird event type e.g. Bar, would this be a breaking?

hey guys, chiming in support for this. Is there a recommend workaround in the meantime?


Normalize heroes' names

Make all heroes' names normalized (as accepted on the icy-veins' links)

Fixed in #9


IOU - IOU details are broken

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to URL https://staging.new.expensify.com/
Login with any account
Click to FAB button -> Request money -> Submit a request to any user
Open IOU of money request

Expected Result:
IOU details visible
Actual Result:
IOU details are broken
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android / native
[x] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] Windows / Chrome
[x] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/78819774/7b445995-2834-42ac-997e-b53028ae402f

Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

This seems to be a dupe of https://github.com/Expensify/App/issues/23195, just manifested for IOU Reports, which are a different type of child report.

This seems to have been caused by https://github.com/Expensify/App/pull/21406

Can't seem to reproduce this on the latest staging build.

spoke too soon, looks like it's reproducible, you have to wait a bit for the API request to respond.

I'm working on reproducing too

Reproduced:

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

@jasperhuangg you were right about that PR causing the deploy blocker: https://github.com/Expensify/App/pull/21406/files#r1268723347
I'm preparing a fix.

I need to confirm if this looks fine:

The list of items is at the bottom, was that changed lately? I can see that in production it looks like this:

I'll remove the DeployBlockerCash label once we confirm this is fixed in staging.

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

Confirmed that this is working in staging now:


App must set security headers to protect against clickjacking

Hello @osiset,
My App was rejected by Shopify due to the bellow reason. please help.

App must set security headers to protect against clickjacking.
Your app does not request installation on the shop immediately after clicking "add app". Apps must ask a shop for access when being installed on a shop for the first time, as well as when they are being reinstalled after having been removed. During install or reinstall we expected OAuth to be initiated at https://cambridgetestshop.myshopify.com/admin/oauth/request_grant but was redirected to https://cambridgetestshop.myshopify.com/admin/apps/0dc7ae777517adc333ebd1aeb6f5a556/authenticate/token?shop=cambridgetestshop.myshopify.com&target=%2F. Learn more about authentication in our developer documentation

I installed the package on fresh laravel and tried to install it on Shopify, but the URL changes.
Please see below screenshot
screenshot
network
Laravel version   - 8.75.0
Package version - 17.1.0
php version        - 8.0.11

Same issue. We got rejected because of the same reason. Any workaround to solve this?

Try phpclassic/php-shopify which is easier and better.
You can have more control over installation process.

Hi.
First off, let me tell you something. I use a translation app because I'm not very good at English. sorry.
The following header must be set for security purposes.
https://github.com/osiset/laravel-shopify/wiki/Installation#appbridge
You will also need to set the following header to allow iframes.
https://shopify.dev/apps/store/security/iframe-protection
The solution to the Content-Security-Policy header was discussed in this issue.
Migrate app from Shopify App Bridge 1.x to Shopify App Bridge 2.0
However, I used a different method to create the process and resubmit the application.

Hi, my app also got rejected because of the same reason. Please help.

[sarmadmakhdoom] is your app working with this solution ?

Thanks @sarmadmakhdoom this really worked

[sarmadmakhdoom] is your app working with this solution ?

Yeah it’s working fine after this. Do you see any issues after adding this middleware?

Hi @sarmadmakhdoom. Are you using this middleware globally or as a group middleware. I used this in group middleware, still my app got rejected for the same reason. Though I can see that the content-security-policy has been set in response headers for all the routes. What can be possibly wrong? Please help.

Hi @sarmadmakhdoom below is my middleware
<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;

class FrameHeadersMiddleware
{
    /**
     * Handle an incoming request.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  \Closure(\Illuminate\Http\Request): (\Illuminate\Http\Response|\Illuminate\Http\RedirectResponse)  $next
     * @return \Illuminate\Http\Response|\Illuminate\Http\RedirectResponse
     */
    public function handle(Request $request, Closure $next)
    {
        $response = $next($request);
        $user = \Auth::user();
        if($user){
            $response->header('Content-Security-Policy', "frame-ancestors https://{$user->name} https://admin.shopify.com");
        }
        return $response;
    }
}

Add below is my kernel.php
<?php

namespace App\Http;

use Illuminate\Foundation\Http\Kernel as HttpKernel;

class Kernel extends HttpKernel
{
    /**
     * The application's global HTTP middleware stack.
     *
     * These middleware are run during every request to your application.
     *
     * @var array<int, class-string|string>
     */
    protected $middleware = [
        // \App\Http\Middleware\TrustHosts::class,
        \App\Http\Middleware\TrustProxies::class,
        \Fruitcake\Cors\HandleCors::class,
        \App\Http\Middleware\PreventRequestsDuringMaintenance::class,
        \Illuminate\Foundation\Http\Middleware\ValidatePostSize::class,
        \App\Http\Middleware\TrimStrings::class,
        \Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull::class,
        \App\Http\Middleware\FrameHeadersMiddleware::class,
    ];

I can see by inspecting the page that content-security-policy is being set on all the http requests. Still my app is getting auto rejected. I am lost here and have no idea of what to do.
Please help.

Hi @pangiras
Do you have X-Frame-Options set in your header?

Hi @inaryu
No I haven't set X-Frame_Options, and my application runs in App Bridge.

Hi @pangiras
Currently the laravel-shopify library has a bug when installing the app.
https://github.com/osiset/laravel-shopify/issues/1071
This may be the reason why your application is rejected.

Hi @pangiras
Currently the laravel-shopify library has a bug when installing the app. #1071 This may be the reason why your application is rejected.

Hi @inaryu
I can confirm that this is not the problem, as I have implemented the solution mentioned by you in #1071  and that has fixed the bug.

@pangiras If so, there may be another cause.
I also submitted my app after making this fix, and so far it has not been rejected.
Do you have any special app settings?

Hi @inaryu  and @sarmadmakhdoom. Thank you for the support provided, I really appreciate your effort and time.
I was able to get past the auto rejection using two fixes.

I added the middleware as mentioned by @sarmadmakhdoom.
I added the following header to this directory in my case /vendor/osiset/laravel-shopify/src/resources/views/layouts/default.blade.php , as I am extending default.blade.php in all the views.

<?php
  $domain=$shopDomain ?? Auth::user()->name ;
  header("Content-Security-Policy: frame-ancestors https://".$domain." "."https://admin.shopify.com");
?>

Hi @pangiras Glad we could work it out!
By the way, I submitted an application with this response.

Create middleware.
Add middleware to the web middleware group

Processing in middleware (app/Http/Middleware/AddContentSecurityPolicyHeader.php)

<?php

namespace App\Http\Middleware;

use App\Models\User;
use Closure;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
use Illuminate\Http\Response;
use Osiset\ShopifyApp\Objects\Values\ShopDomain;

class AddContentSecurityPolicyHeader
{
    protected const ADMIN_SHOPIFY_URL = 'https://admin.shopify.com';

    protected const HEADER_FORMAT = 'frame-ancestors %s %s';

    public function handle(Request $request, Closure $next)
    {
        /** @var Response|RedirectResponse $response */
        $response = $next($request);

        // If it's not a redirect response and it's not Ajax
        if ($response instanceof Response && !$request->ajax()) {
            // If the request contains shop data, get the shop data
            if ($request->has('shop')) {
                $shopDomain = ShopDomain::fromNative($request->get('shop'));
            } elseif ($request->user() instanceof User) {
                // Get from user data
                $shopDomain = $request->user()->getDomain();
            } else {
                // If you still can't get it, get it from the request data.
                $shopDomain = ShopDomain::fromRequest($request);
            }

            if ($shopDomain instanceof ShopDomain) {
                $response->header('Content-Security-Policy', sprintf(self::HEADER_FORMAT, 'https://' . $shopDomain->toNative(), self::ADMIN_SHOPIFY_URL));
            }
        }

        return $response;
    }
}

Add middleware to the web middleware group in app/Http/Kernel.php

    protected $middlewareGroups = [
        'web' => [
            \App\Http\Middleware\EncryptCookies::class,
            \Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse::class,
            \Illuminate\Session\Middleware\StartSession::class,
            // \Illuminate\Session\Middleware\AuthenticateSession::class,
            \Illuminate\View\Middleware\ShareErrorsFromSession::class,
            \App\Http\Middleware\VerifyCsrfToken::class,
            \Illuminate\Routing\Middleware\SubstituteBindings::class,
            // add middleware
            \App\Http\Middleware\AddContentSecurityPolicyHeader::class,

        ],

        'api' => [
            'throttle:api',
            \Illuminate\Routing\Middleware\SubstituteBindings::class,
        ],
    ];

I think adding it to the web middleware group will handle most of the requests.
This also means that you probably don't need to add it to your layout file.

Hi @pangiras Glad we could work it out! By the way, I submitted an application with this response.

Create middleware.
Add middleware to the web middleware group

Processing in middleware (app/Http/Middleware/AddContentSecurityPolicyHeader.php)

<?php

namespace App\Http\Middleware;

use App\Models\User;
use Closure;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
use Illuminate\Http\Response;
use Osiset\ShopifyApp\Objects\Values\ShopDomain;

class AddContentSecurityPolicyHeader
{
    protected const ADMIN_SHOPIFY_URL = 'https://admin.shopify.com';

    protected const HEADER_FORMAT = 'frame-ancestors %s %s';

    public function handle(Request $request, Closure $next)
    {
        /** @var Response|RedirectResponse $response */
        $response = $next($request);

        // If it's not a redirect response and it's not Ajax
        if ($response instanceof Response && !$request->ajax()) {
            // If the request contains shop data, get the shop data
            if ($request->has('shop')) {
                $shopDomain = ShopDomain::fromNative($request->get('shop'));
            } elseif ($request->user() instanceof User) {
                // Get from user data
                $shopDomain = $request->user()->getDomain();
            } else {
                // If you still can't get it, get it from the request data.
                $shopDomain = ShopDomain::fromRequest($request);
            }

            if ($shopDomain instanceof ShopDomain) {
                $response->header('Content-Security-Policy', sprintf(self::HEADER_FORMAT, 'https://' . $shopDomain->toNative(), self::ADMIN_SHOPIFY_URL));
            }
        }

        return $response;
    }
}

Add middleware to the web middleware group in app/Http/Kernel.php

    protected $middlewareGroups = [
        'web' => [
            \App\Http\Middleware\EncryptCookies::class,
            \Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse::class,
            \Illuminate\Session\Middleware\StartSession::class,
            // \Illuminate\Session\Middleware\AuthenticateSession::class,
            \Illuminate\View\Middleware\ShareErrorsFromSession::class,
            \App\Http\Middleware\VerifyCsrfToken::class,
            \Illuminate\Routing\Middleware\SubstituteBindings::class,
            // add middleware
            \App\Http\Middleware\AddContentSecurityPolicyHeader::class,

        ],

        'api' => [
            'throttle:api',
            \Illuminate\Routing\Middleware\SubstituteBindings::class,
        ],
    ];

I think adding it to the web middleware group will handle most of the requests. This also means that you probably don't need to add it to your layout file.

Hi @inaryu ,
I also first started out by creating the group middleware and did add into web middleware group and I could see that all the requests made had content-security-policy set, still the app got auto rejected. And then I changed it to global middleware to eliminate any doubts, still the app got auto rejected. And then I tried the fix that solved my problem. Though it isn't the best practice as I can see that in some requests the csp is repeated, but that won't do any harm I suppose. I like the way you have implemented the middleware, may it also help others facing the same problem.

Hi @inaryu ,
I also first started out by creating the group middleware and did add into web middleware group and I could see that all the requests made had content-security-policy set, still the app got auto rejected. And then I changed it to global middleware to eliminate any doubts, still the app got auto rejected. And then I tried the fix that solved my problem. Though it isn't the best practice as I can see that in some requests the csp is repeated, but that won't do any harm I suppose. I like the way you have implemented the middleware, may it also help others facing the same problem.

Hi @inaryu and @sarmadmakhdoom. Thank you for the support provided, I really appreciate your effort and time.
I was able to get past the auto rejection using two fixes.

I added the middleware as mentioned by @sarmadmakhdoom.
I added the following header to this directory in my case /vendor/osiset/laravel-shopify/src/resources/views/layouts/default.blade.php , as I am extending default.blade.php in all the views.

<?php
  $domain=$shopDomain ?? Auth::user()->name ;
  header("Content-Security-Policy: frame-ancestors https://".$domain." "."https://admin.shopify.com");
?>

You’re welcome. I’m glad I was able to help

@inaryu @sarmadmakhdoom - Feel free to submit a PR to help fix this issue for the wider package.

I followed the as mentioned by @sarmadmakhdoom. But still app gets rejected.
Anyone know what is the issue ?

Rejection can be based on a number of things - Shopify will tell you why in their response to you. So you would need to look at that and see what you need to do.

The everytime getting same feedback-
App must set security headers to protect against clickjacking.
Your app must set the proper frame-ancestors content security policy directive to avoid clickjacking attacks. The 'content-security-policy' header should set frame-ancestors https://[shop].myshopify.com/ https://admin.shopify.com/, where [shop] is the shop domain the app is embedded on.

@ingalesachin7 did you find any solution? I am getting the same response in every rejection.

Released in v17.2.0


Adding new Feature/pavd

Some cleaning and testing needed, but eventually this draft PR will close #66 and add the ability of canopy-app to read 3D inputs of observed GEDI Plant Area Volume Density (PAVD) profiles for a user-set range of latitude and longitude bounds.  Previously canopy-app could only be driven by prescribed plant shape distribution functions.
Note:  The PAVD input data example file is based on preliminary GEDI gridded/raster data provided from Scott Goetz and Pat Burns of NAU.  The following is needed to move this PR forward:

[x] Use the updated version of their gridded PAVD dataset on the NAU HPC Portal
[x]  Verification and comparisons of  PAVD highest height bins against updated 1 km GEDI FCH (also used in canopy dataset blended with Potapov FCH for gap filling and in high latitudes) from NAU HPC Portal.  May need to establish an altitude cutoff threshold for real vs negligible PAVD.
[x]  Integrated PAVD (i.e., PAI) comparison against updated, gridded 1 km VIIRS PAI is needed to validate spatial distribution and column values.

@angehung5   Thanks for the updated daily LAI in SE files.  Can you remove the 'time' column from all updated txt files?  canopy-app doesn't use the timestamp in txt files currently.
Also, can you update these .NC files here for new PAVD product too?

@angehung5   Have you updated these SE files for PAVD projection?  Seems from quick ncview they are still shifted:

How about the global hourly canopy-app files?  Thank you!

@drnimbusrain Sorry I grabbed the wrong files. Just fixed. They should work now.

Great Wei-Ting, thanks!  I will continue development here for canopy-app.
Let's move forward with the updated PAVD max bin height vs FCH (at least
with the Potapov dataset until we hear back from Pat) and the updated
integrated PAVD (PAI) vs. VIIRS PAI regression comparison.
Thank you!
On Fri, Oct 27, 2023, 11:11 PM Wei-Ting Hung @.***>
wrote:

@drnimbusrain https://github.com/drnimbusrain Sorry I grabbed the wrong
files. Just fixed. They should work now.
—
Reply to this email directly, view it on GitHub
https://github.com/noaa-oar-arl/canopy-app/pull/91#issuecomment-1783678801,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGLFYNRFRBAXRQS4ANDGCALYBRZVNAVCNFSM6AAAAAA4T3STROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTGY3TQOBQGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

@drnimbusrain Comparison using updated GEDI files with coordinates corrected.

Maxh is significantly lower than CH over tropical regions, and generally higher than CH by less than 5m. Since PAVD has a vertical interval of 5m, I think difference < 5m is acceptable.

Simply plotted everything here. We definitely need to look into this...maybe separate into different veg type. Any suggestions?

Yes, all results are based on the most updated NAU products.

Yes, maxh << CH at tall forest regions. Also maxh is relatively smooth since PAVD has an interval of 5m. But again, we should check with Pat about how they define maxh.

VIIRS LAI 2020 annual average is used in this comparison while PAVD is a multi-year average across 2019-2023.

@angehung5 Maybe you can plot up some of the updated PAVD profiles for points across the SE test domain.  What do these profiles look like, and are there any real shape?  From our proposal, seems the PAVD may be able to capture some taller forests shape distributions:

@drnimbusrain
It works fine on my end.

Are you using the SE files in the feature/pavd branch? Seems those files are not updated, while the files in the develop branch are up to date. I tried to update the files in the feature/pavd branch but somehow the system does not take the change.

@angehung5 Its strange, I updated the files again from develop branch, but the canopy-app output results are the same I was getting before with i = 25 and j= 25, and FCH = 6.4 m.  You can run the current feature/pavd branch and it prints out the following PAVD values for the grid cell:
Lat =    32.157498906278789
Lon =<PHONE_NUMBER>0000000
VIIRS PAI =    3.4859482822898071
FCH =    6.3751093731480140
pavd_orig =    7.5710922479629517E-002   6.4598932862281799E-002   4.3507631868124008E-002   2.5484789162874222E-002   1.1120960116386414E-002   3.6568134091794491E-003   1.5977938892319798E-003   6.2849238747730851E-004   3.1753323855809867E-004   3.0350771339726634E-005   0.0000000000000000        0.0000000000000000        0.0000000000000000        0.0000000000000000
...
Can you double check the files in develop and feature/pavd,.

After discussion with @angehung5 we will move this forward with the preliminary evaluations and since we have not heard back from NAU about the comparisons.
Few things left to do:

[ ] Update the CH (from Potapov to new RH-98 NAU product) in all example .nc, .txt, and point_file*.txt consistently in the repos three example SE domains.
[ ] Add additional supplementary 3D text files (consistent with the three times) with only a single point column for pavd variable and associated 14 mid-point lev.  These new point files should be consistent with time and intial lat/lon row in three SE example text files.  We can name them, point_file_3d20220630.sfcf023.txt,  point_file3d20220701.sfcf000.txt, and point_file3d_20220701.sfcf001.txt
[ ] Add capability of canopy-app to read in new 3D text files for PAVD when correct options in NL and conditions (single point text examples etc.) are met in canopy-app.

I would suggest sticking to RH-95 for now, which is generally used for canopy height estimation. We can update to RH-98 later if needed (data is already on Hopper).

@angehung5 Thanks for the updates, but there is some issue with the changes and CI...See the details here.

Fixed. Now they should work.
Below shows the difference between Potapov and NAU product (~ +- 2m). Just for a record.

@angehung5 Thank you for fixing that, as it likely had to do with the CI python example run for the point based files.  Seems when you uploaded the new point files with updated CH you had another vtype that wouldn't run through the options set in the python CI.
So I checked the first item based on now updated CH from NAU for all SE .nc and .txt files, correct?  If so, I will continue to test these in canopy-app.
Just the other two items regarding new supplementary pavd .txt files for being able to drive canopy-app with point/column data as well.
Thanks again,

@drnimbusrain Point PAVD test file added. Please go ahead and check the second item if they work.

@angehung5 If you could provide a review of this as well, when you have the time.  (If OK, do you think we should add a sensitivity test to your paper for using the observed PAVD max from GEDI as well?)   Thanks!

@angehung5 If you could provide a review of this as well, when you have the time. If OK, do you think we should add a sensitivity test to your paper for using the observed PAVD max from GEDI as well? You can see that it has an affect on the wind speed profile compared to default profile from Massman. Thanks!

@angehung5 Can you please give this a review ASAP?

@drnimbusrain A quick comparison of Massman shape function and GEDI PAVD. Seems the wind reduction based on PAVD is generally lower compared with Massman. The foliage vertical distribution is different as well, especially at Lower Teakettle, probably because PAVD values are actually averages over certain area (0.01 degree resol) and maybe affected by surrounding vegetations. It should also be noted that the vertical resolution of PAVD profile is much coarser.

@drnimbusrain A quick comparison of Massman shape function and GEDI PAVD. Seems the wind reduction based on PAVD is generally lower compared with Massman. The foliage vertical distribution is different as well, especially at Lower Teakettle, probably because PAVD values are actually averages over certain area (0.01 degree resol) and maybe affected by surrounding vegetations. It should also be noted that the vertical resolution of PAVD profile is much coarser.

Thank you @angehung5 for doing the review and test!
First, to clarify, I am only using the height of PAVD maximum from GEDI, and not the actual entire PAVD profile.  So, the GEDI PAVD is used to only set the height of maximum foliage density and uses the vegtype set standard deviations above and below the maximum based on Massman, so it really is a hybrid approach right now.  With the major issues of the gridded PAVD observations (coarse vertical res, inconsistencies with FCH we have right now from NAU, with no response back from them whatsoever, this is the best we have right now.  Either way, using Massman, or hybrid PAVD/Massman approach has uncertainty in where the peak foliage density is set, but seems Massman approach is better right now.
Thus, can you also plot the foliage profiles for the different methods for each site to see how they vary?  I wonder how you are setting the PAVD for these plots.

The shapes at Chestnut and GOA look ok, although the height of maximum density is lower than Massman. Willow Creek and  Lower Teakettle have the same issue that the maximum height of PAVD is comparable to the canopy height but PAVD values peak near the surface and are relatively low near the canopy top.
PAVD at Lower Teakettle for example, FH=32m
lat,lon,lev01,pavd01,lev02,pavd02,lev03,pavd03,lev04,pavd04,lev05,pavd05,lev06,pavd06,lev07,pavd07,lev08,pavd08,lev09,pavd09,lev10,pavd10,lev11,pavd11,lev12,pavd12,lev13,pavd13,lev14,pavd14
37.00,240.99,2.50,0.0682,7.50,0.0632,12.50,0.0449,17.50,0.0274,22.50,0.0210,27.50,0.0166,32.50,0.0120,37.50,0.0076,42.50,0.0043,47.50,0.0020,52.50,0.0007,57.50,0.0002,62.50,0.0000,67.50,0.0000

Agreed gridding should be the main reason here. Since the maximum heights of PAVD are reasonable (consistent with canopy height), satellite does see the forests. However, the surrounding shorter vegetations introduce a much lower height of maximum PAVD, which we can see at Lower Teakettle.


SPIN - Configurable feature set

Make the feature set in SPIN configurable so the user can enable or disable:

validation
construct
????

Limiting thing to just configurable validation.


feat: support loading hudi global configs

Description
Load hudi-defaults.conf from HUDI_CONF_DIR when creating table instance , for #111  .
How are the changes test-covered

[x] Automated tests: I have added a unit test for this feature

@zzhpro thanks. i've made some refactoring based on your version.


Allow ID to use field.Other with custom types

When using field.Other the generated code would cause problems on two instances:
https://github.com/crossworth/entissue2290/blob/042ee692b298775220878f08082e6592575eef58/ent/user_create.go#L231

https://github.com/crossworth/entissue2290/blob/042ee692b298775220878f08082e6592575eef58/ent/user.go#L56

Line 56: conversion from int64 to ID (string) yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
I'm not sure if this is the best way to solve the problem, but I think we can assume that when the user use a field.Other it would not be ~int64.
This WILL NOT fix the problem when using GoType. The same problem happens, but I don't know how to solve, maybe a better approach would be to check the type.
Partial fix for: #2290

@all-contributors please add @crossworth for code


Weird Asian characters not fixed

I have a couple of filenames that are encoded and decoded on various operating systems without any respect to Unicode, and what I got is something like this:
U+9DED  鷭      [Lo] CJK UNIFIED IDEOGRAPH-9DED

that should be
U+00F8  ø       [Ll] LATIN SMALL LETTER O WITH STROKE
U+0073  s       [Ll] LATIN SMALL LETTER S

U+6DEA  淪      [Lo] CJK UNIFIED IDEOGRAPH-6DEA

↓
????????????????????????????? (maybe a base 10 Arabic digit)
U+005F  _       [Pc] LOW LINE

U+7FB9  羹      [Lo] CJK UNIFIED IDEOGRAPH-7FB9

↓
U+00FC  ü       [Ll] LATIN SMALL LETTER U WITH DIAERESIS

U+E15D  \ue15d  [Co] <unknown>

↓
U+00FC  ü       [Ll] LATIN SMALL LETTER U WITH DIAERESIS

Any ideas would be appreciated.
I'm especially interested in the ??? character. :)

I wouldn't get your hopes up too much -- as I look into supporting more cases of mojibake, I continually notice that CJK mojibake is the worst. (I guess there's a reason they invented the word.)
The major Japanese and Chinese encodings seem to have been designed to be as confusable with each other as possible, and on top of that you can sometimes mix up Japanese in Shift-JIS with UTF-8 and get valid Chinese characters out.
I'm still interested to know what your particular case of mojibake is, but I would need to see the entire strings and what they turned into, not just single characters.


i can't run it in ubunutu

it always show problem get cmd when i run it in ubunutu

could you please send the error some more info.


feat: Import an template inside an existing Application

Description
This PR adds the feature to import a template inside an application.
Fixes #12336
Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

JUnit test
Manually

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

/ok-to-test sha=4f97635

@AnaghHegde Definitely it would be better if import application method is used for this purpose. I tried using that earlier but faced some issues - gitSyncId was just one of it. Another issue was - existing actions were being delete from pages. I'll try again.

@nayan-rafiq Here is what i am thinking, let's not touch the ImportApplication method. It is already designed to handle merging two applications(for git sync) where it can create/delete/update resources. For templates what we can do is set the the git sync id as null after exporting the json file - this way one can use the same template in the same application multiple times. Second, for application name getting updated, we can update the appJson so that the application name is same as where it is being imported. Let me know your thoughts on this.

/ok-to-test sha=6db0553

/ok-to-test sha=033013c

/ok-to-test sha=22a261

/ok-to-test sha=a9a6372

/ok-to-test sha=af366ba4a2992f9c5652a34dcd3f2671a7138772

/ok-to-test sha=af366ba

/ok-to-test sha=db10426

/ok-to-test sha=95a5531

/ok-to-test sha=9b57bd4

/ok-to-test sha=9b57bd4

/ok-to-test sha=e84be56

/ok-to-test sha=b59665b

/ok-to-test sha=10f0d3f

/ok-to-test sha=10f0d3f

/ok-to-test sha=cba82e8

/ok-to-test sha=16ec8337b2024d691aabc5a963da670893e86791

/ok-to-test sha=2a3108285ab6b148da23ad0d883ab09af9d61626

/ok-to-test sha=2a31082

/ok-to-test sha=da2629f

/ok-to-test sha=96a0a4d

/ok-to-test sha=2d8f3e7

/ok-to-test sha=e62d831

/ok-to-test sha=89cabb9

/ok-to-test sha=89cabb9

/ok-to-test sha=5436558

/ok-to-test sha=adffbcb

/ok-to-test sha=6275ccb

/ok-to-test sha=5a53280


camel case style properties

This patch fixes the display of camel-cased style properties showing up in the html (https://github.com/Matt-Esch/virtual-dom/issues/359).
I had to update some tests to check for properties using the dashed property names instead of camel-cased properties, but the changes are relatively minor.

I'm not sure I understand why virtual-dom would need to change the casing of the properties. I understand that the html rendering is producing the wrong thing, but this isn't the job of virtual-dom to accommodate. virtual-dom is based purely on properties of the dom nodes in a browser agnostic fashion. min-document should be patched to generate correct html from the virtual dom node.

Agreed.
Doing elem.style.className = x; and elem.style['class-name'] = x; are perfectly valid DOM expression.
It's min-document that does the wrong thing and we should fix that instead.

closing since this issue is handled upstream in min-document


Deterministic, parallel data iteration

The parallel eachobs implementation is not deterministic in that observations are returned as soon as they are loaded, so they may be returned out of order. This is very performant, and fine for some use cases like training, where data should be shuffled anyway.
To give the option to have a deterministic iteration would be helpful in many use cases, though.
This could be implemented as a wrapper around an existing iterator that does the following:

instead of iterating over data with the wrapped iterator, iterate over (1:nobs(data), data) to preserve ordering information
collect returned observations, stripping the index
return an observation only if all previous (by index) observations have been returned

I am unsure by how much this will affect performance and memory usage and how the interplay is with buffersize. Are there alternative approaches to this implementation?

I believe FFCV has the notion of a traversal order which we might want to look into. Apparently the quasi-random variant increases performance too, so there may be a third option between random and deterministic that we want to include here.

Do you know what they are doing there specifically? I thought that was just pre-shuffling and then storing into contiguous memory, but from that page it seems like that is only relevant when not in memory.


Skip LTO on try builders

Needs https://flutter-review.googlesource.com/c/recipes/+/5640
/cc @chinmaygarde @jason-simmons

Here's an example of a build where LTO added about 8 minutes: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket.appspot.com/8872059819503845344/+/steps/build_ios_debug/0/stdout


Create unit tests for utils

Create unit tests for the helper functions in utils.py

Tracking: https://anaconda.atlassian.net/browse/PAT-192
Note that we are already hitting well above 80% coverage.


Delegate hotfix 81408

Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the CODEOWNERS.
What Type of PR is This?

[ ] Issue
[ ] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[ ] Tested Locally
[ ] Optional Screen Shoot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://655dfee46e85231ea441ee23--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://655e0b75b2a3c42b1774d763--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://655e0c379646552b7c1a6ee6--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://655e0fea20c21334180f72c1--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is:

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://655e2f8296465546bd1a72eb--harness-developer.netlify.app


Cannot load images with more than 10Kb

Images are not loading if the image size is more than 10Kb
The image stops working if the file size is more than 10Kb.

If i change  name: "[name].[contenthash:8].[ext]" to name: "[name].[hash:8].[ext]" for images section in hooks.js then actual hash is used but then i get the following error

@Satish-Gupta Sorry for the late, have you synced the name between hooks.js and config.babel.js?

Hi,
I have synced the hooks.js and config.babel.js and still having the same result.

hooks.js

// Images
  require("asset-require-hook")({
    // Must use the same option with webpack's configuration
    extensions: ["gif", "jpg", "jpeg", "png", "webp", "svg"],
    publicPath: "/assets/",
    limit: 10 * 1024,
    name: "[name].[hash:8].[ext]",
  });

config.babel.js

{
        test: /\.(gif|png|jpe?g|webp|svg)$/,
        use: [
          {
            // Any image below or equal to 10K will be converted to inline base64 instead
            loader: "url-loader",
            options: { limit: 10 * 1024, name: "[name].[hash:8].[ext]" },
          },
          {
            loader: "image-webpack-loader",
            // For each optimizer you wish to configure
            // Plz check https://github.com/tcoopman/image-webpack-loader#usage
            options: { disable: true },
          },
        ],
      },

If i remove the hash section in both of the above it works.

@Satish-Gupta Please solve it by your way now. I am going to implement the server-side bundling once the starter has been upgraded to webpack v5.

@wellyshen, Sure. I will do so until the upgrade. Thank you.


CARDS-2056: PREMs Survey updates

Specs:

CARDS-2056 - PREMs Survey updates

Testing:

Test all prems surveys as patient

Reviewing:

commit by commit is recommended

CPESIC - all changes are done as per request

This change just does not make sense:

Review page does not look good with question numbers being added:

Rehab survey - all changes done

Question: In JIRA task it said: Module 3 in Rehab should be labeled “Your Rehab Experience”. In the survey it is "Your rehabilitation experience"

ED & IP

This change just does not make sense. It will just confuse the patient.

It is based on the paper template. Isn't it more confusing to see 12 come after 10?

The Review page does not look good with question numbers being added:

One of the reasons numbering was requested was to make the review page more readable. Can you be more specific re what does not look good and do you have any suggestions of improvement?

In JIRA task it said: Module 3 in Rehab should be labeled “Your Rehab Experience”. In the survey it is "Your rehabilitation experience"

I was wrong on jira, implementation is right.

One of the reasons numbering was requested was to make the review page more readable. Can you be more specific re what does not look good and do you have any suggestions of improvement?

This looks like some questions were skipped. We know why (the conditional questions), but the patient may think they did something wrong.

One of the reasons numbering was requested was to make the review page more readable. Can you be more specific re what does not look good and do you have any suggestions of improvement?

This looks like some questions were skipped. We know why (the conditional questions), but the patient may think they did something wrong. However, troubleshooting will be much easier with numbers.

And that's why there were indications while they were filling out the form that the next question would be n+m, not n+1

Question numbers have been removed in the last commit.

The email templates for CPESIC have been updated in 88ba79e.

Pagination variants updated per latest request. This PR is ready to be re-tested @sashaandjic


[HybridApp] iOS InternalShadowView crash

Problem
We are rolling out HybridApp and iOS is currently at 1%, jumping to 5/10% over the weekend. A new crash is occurring, seemingly in OldDot experience. This could be problematic, even if for now reports are mostly older devices.
- Multiple OSs: iOS 16/17
- Many devices: iPhone 15 Pro/iPhone SE 2nd gen/iPhone 11

StackTrace
Crashed: com.apple.main-thread
0  Expensify                      0x152d34 UIShadowView.InternalShadowView.draw(_:) + 54 (UIShadowView.swift:54)
1  Expensify                      0x152d80 @objc UIShadowView.InternalShadowView.draw(_:) +<PHONE_NUMBER> (<compiler-generated>:4302171520)
2  UIKitCore                      0xc2270 <redacted> + 508
3  FullStory                      0xdeaac bcmap_addDelta(std::__1::vector<unsigned short, std::__1::allocator<unsigned short>>&, long long) + 59332
4  FullStory                      0xd0fa4 bcmap_addDelta(std::__1::vector<unsigned short, std::__1::allocator<unsigned short>>&, long long) + 3260
5  FullStory                      0xde88c bcmap_addDelta(std::__1::vector<unsigned short, std::__1::allocator<unsigned short>>&, long long) + 58788
6  FullStory                      0xe33f4 PopulateInclusiveFramesForLayer(id<FSShared>, CALayer*) + 17676
7  libdispatch.dylib              0x3dd4 <redacted> + 20
8  libdispatch.dylib              0x1339c <redacted> + 92
9  libdispatch.dylib              0x3dd4 <redacted> + 20
10 libdispatch.dylib              0x125a4 <redacted> + 988
11 libdispatch.dylib              0x121b8 _dispatch_main_queue_callback_4CF + 44
12 CoreFoundation                 0x56710 <redacted> + 16
13 CoreFoundation                 0x53914 <redacted> + 1996
14 CoreFoundation                 0x52cd8 CFRunLoopRunSpecific + 608
15 GraphicsServices               0x11a8 GSEventRunModal + 164
16 UIKitCore                      0x40aae8 <redacted> + 888
17 UIKitCore                      0x4bed98 UIApplicationMain + 340
18 Expensify                      0x416cc main + 16 (main.swift:16)
19 ???                            0x1adb9b154 (Missing)

Logs
com.expensify.expensifylite_issue_754e3902437176e57ff794418fe4f3fb_crash_session_a44bd6907de04c90b2c4279be7235af0_DNE_0_v2.log
Solution
Investigate 🕵

I'm not seeing a rise in reports as we rollout. Looking good.

This is still occuring but only about ~5 crashes per day. Ignoring for now.

Looked into this and Firebase reports have increased. Though it's a small number relative to the number of users... I suggest we leave this open as a first pick and wait until contributors have access to the Mobile repo.

Lets close. This is linked to in firebase, so we can re-open later when this is a high priority crash.


butterknife + data binding

No actual issue here, I was just curious to know what your plans are if any in regards to databinding with butterknife. They don't do exactly the same thing but they are similar and I think they could possibly work well together. I don't have any suggestions for api additions or anything at this time.

No plans. The data binding might be useful in the most trivial cases, but there was no innovation done so it'll be equally as painful as similar solutions on other platforms when complexity increases (e.g., XAML). This library scales to the advanced cases since it forces you to keep binding logic in code where it more correctly belongs.


update moon heightmap

Signed-off-by: Ian Chen<EMAIL_ADDRESS>Requires https://github.com/gazebosim/gz-common/pull/439
Updated to use the 4k heightmap from https://app.gazebosim.org/Hyundai/fuel/models/Moon
I also tweaked lighting a little bit to prevent the interior spot lights from 'leaking' out to the exterior heightmap surface
Note that this large high resolution heightmap increases VRAM usage. For me it now takes almost 8GB to run the demo world. You can run nvidia-smi to see this info. If this is too memory intensive, we should probably just stick to the original heightmap.

@iche033 @nkoenig  Can I ask how did you generate 16bit .png from .tif?
When I do:
gdal_translate -outsize 4096 4096 -of PNG -ot UInt16 -scale -1913.92 -1859.52  0 65536 input.tif output.png

Then I get these:
[Err] [Image.cc:606] Image: Coordinates out of range[2413 65]

The  8bit Byte option works fine, but obviously heightmap is jagged:

I'm on Harmonic, and the moon demo looks fine with your heightmap, It just doesn't parse mine 16bit png from gdal.

so I think your tif file contains DEM data (geo tiff) which is different from a png file with grayscale color data. They are different formats and loaded differently by gazebo. I don't know think you can directly convert between the two.

@iche033 I'm using png. I took DEM from LRO/LOLA which is in geotiff and sliced it to be squared size 2^k  (4096x4096) same as your texture.
Then converted .tiff to .png with gdal, and here is the problem. When I use 8bit - grayscale it works, but when I try to output 16bit I get these errors. These are .png files, and you implemented the 16 bit .png loading.
I get same errors as you describe in your PR.
I'm on 8.6.0 Harmonic, so this 16bit png loading should be there, and it also works on your file.
Did you create png directly from qgis? I presume you also had used NASA's tiff moon data to get the heightmap.


Add IAM cloudformation

Add iam.yml with permissions for RDS to import from S3:
Importing Amazon S3 Data into an RDS for PostgreSQL DB Instance

All IAM permissions are in rds.yml ☑

All IAM permissions are in rds.yml ☑


cleanup: removes Crate class prefix, moves settings to where they are used

Codecov Report

Merging #6191 into ingestion_mqtt will decrease coverage by 0.06%.
The diff coverage is 91.17%.

@@                 Coverage Diff                  @@
##             ingestion_mqtt    #6191      +/-   ##
====================================================
- Coverage              83.3%   83.24%   -0.07%
+ Complexity            12757    12746      -11
====================================================
  Files                  1477     1476       -1
  Lines                 48242    48239       -3
  Branches               5904     5904
====================================================
- Hits                  40188    40156      -32
- Misses                 5315     5339      +24
- Partials               2739     2744       +5

Impacted Files
Coverage Δ
Complexity Δ

...e/mqtt/src/main/java/io/crate/mqtt/MqttModule.java
100% <100%> (ø)
5 <2> (?)

.../crate/mqtt/netty/MqttNettyIdleTimeoutHandler.java
11.11% <100%> (ø)
1 <1> (?)

...ain/java/io/crate/mqtt/netty/MqttNettyHandler.java
51.61% <100%> (ø)
5 <1> (?)

...io/crate/mqtt/netty/Netty4MqttServerTransport.java
77.58% <100%> (+1.22%)
14 <1> (+1)
:arrow_up:

...va/io/crate/mqtt/operations/MqttIngestService.java
87.87% <100%> (ø)
7 <2> (?)

...ain/java/io/crate/mqtt/protocol/MqttProcessor.java
88% <100%> (ø)
17 <4> (?)

...va/io/crate/mqtt/protocol/MqttMessageBuilders.java
92.3% <50%> (ø)
7 <3> (?)

...n/java/io/crate/exceptions/JobKilledException.java
0% <0%> (-66.67%)
0% <0%> (-1%)

...io/crate/protocols/postgres/types/VarCharType.java
70.83% <0%> (-20.84%)
9% <0%> (-1%)

...in/java/io/crate/data/CollectingBatchConsumer.java
69.23% <0%> (-15.39%)
4% <0%> (-1%)

... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 950625e...b28834a. Read the comment docs.


Revert "Add visual editor configuration"

Reverts gerardag/person-entity-card#22

Why you revert the PR?


Podspec: pod lib lint doesn't work (v1.2.0)

Unfortunately the Cocoapods installation method won't install v1.2.0 right now since the Podspec seems to be broken. The issue is that I'm using AppKit on the OS X target which doesn't seem to be recognized by Cocoapods even after explicitly listing it.
Output:
$ pod lib lint

 -> HandySwift (1.2.0)
    - ERROR | xcodebuild: Returned an unsuccessful exit code. You can use `--verbose` for more information.
    - ERROR | xcodebuild:  /Users/Dschee/Code/Flinesoft/ApplePlatforms/HandySwift/Sources/Extensions/CoreGraphicsExtensions.swift:75:12: error: no such module 'AppKit'
    - NOTE  | xcodebuild:  clang: error: linker command failed with exit code 1 (use -v to see invocation)
    - NOTE  | [iOS] xcodebuild:  fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can't open input file: /Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Intermediates/Pods.build/Release-iphonesimulator/HandySwift.build/Objects-normal/i386/HandySwift (No such file or directory)
    - NOTE  | [iOS] xcodebuild:  error: cannot parse the debug map for "/Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Products/Release-iphonesimulator/HandySwift/HandySwift.framework/HandySwift": No such file or directory
    - ERROR | xcodebuild:  /Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Products/Release-iphonesimulator/HandySwift/HandySwift.framework/Modules/module.modulemap:9:12: error: header 'HandySwift-Swift.h' not found
    - ERROR | xcodebuild:  /var/folders/c1/slwc_7ln1wq8kq619bcc33h40000gn/T/CocoaPods/Lint/App/main.swift:1:8: error: could not build Objective-C module 'HandySwift'
    - NOTE  | [iOS] xcodebuild:  fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can't open input file: /Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Intermediates/App.build/Release-iphonesimulator/App.build/Objects-normal/i386/App (No such file or directory)
    - NOTE  | [iOS] xcodebuild:  error: cannot parse the debug map for "/Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Products/Release-iphonesimulator/App.app/App": No such file or directory
    - NOTE  | [tvOS] xcodebuild:  error: cannot parse the debug map for "/Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Products/Release-appletvsimulator/HandySwift/HandySwift.framework/HandySwift": No such file or directory
    - ERROR | [tvOS] xcodebuild:  /Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Products/Release-appletvsimulator/HandySwift/HandySwift.framework/Modules/module.modulemap:9:12: error: header 'HandySwift-Swift.h' not found
    - NOTE  | [tvOS] xcodebuild:  error: cannot parse the debug map for "/Users/Dschee/Library/Developer/Xcode/DerivedData/App-criesqgevvftrmciwougseuwlluo/Build/Products/Release-appletvsimulator/App.app/App": No such file or directory

There is now a 1.3.0 release, see #6 for additional details. Closing this for now, I hope no one needs the 1.2.0 features soon with Swift 2 – long term there is a 1.3.0+ release with Swift 3 support that works on CocoaPods.


.NET 6 or 7 and Minimal Hosting

It would be great if you could update this docs about connecting to App Config Service and Managed Identity to use .NET 6 or 7 and minimal hosting config. It would make things a lot easier.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e0b5a566-084c-d77f-9d78-3bfd72b01080
Version Independent ID: b22f2729-2f31-a0cd-fc61-0ead183b0aab
Content: Use managed identities to access App Configuration - Azure App Configuration
Content Source: articles/azure-app-configuration/howto-integrate-azure-managed-service-identity.md
Service: azure-app-configuration
GitHub Login: @maud-lv
Microsoft Alias: malev

@arambazamba
It looks like your issue is resolved since you closed this thread.
If there are further questions regarding this matter, please tag me in a comment. I will reopen it and we will gladly continue the discussion.


Make project.get_default_index() populate a default name

The issue
Running pipenv with a Pipfile using a source without a name, like
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
"flake8" = "*"
mypy = "*"
pydocstyle = "*"
pylint = "*"
twine = "*"

[packages]
aiodns = "*"
aiohttp = "*"
async-timeout = "*"

results in a missing key error:
Traceback (most recent call last):
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/resolver.py", line 675, in <module>
    main()
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/resolver.py", line 661, in main
    _main(
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/resolver.py", line 645, in _main
    resolve_packages(
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
                        ^^^^^^^^
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/resolver.py", line 592, in resolve
    return resolve_deps(
           ^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 910, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 672, in actually_resolve_deps
    resolver = Resolver.create(
               ^^^^^^^^^^^^^^^^
  File "/usr/local/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 222, in create
    index_lookup[package_name] = project.get_default_index()["name"]
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'name'

The fix
My fix is to populate the source with a default name. My initial take was raising a proper error but then I saw that providing a default name was already handled in other places so I figured this was the right solution.
The checklist

[ ] Associated issue
[x] A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

Thanks!


Heroku deployment error

My django application deployed successfully but displays the following error when viewing
"Application error
An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command
heroku logs --tail"

2020-10-24T18:34:53.484850+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=the-best-writers-zone.herokuapp.com request_id=1557dedb-30b2-42dc-a48d-fb57d9a2f315 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T19:20:57.085676+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=the-best-writers-zone.herokuapp.com request_id=f9f55d48-8466-4e17-abba-45990a85c39f fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T19:20:58.248463+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/favicon.ico" host=the-best-writers-zone.herokuapp.com request_id=870b730d-40e7-4233-a6c2-9225c24f0bb7 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T19:35:35.461309+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=the-best-writers-zone.herokuapp.com request_id=fe4277c6-64b3-491e-9f53-11de9f72ac2c fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T19:35:35.908531+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/favicon.ico" host=the-best-writers-zone.herokuapp.com request_id=9e49fe23-74d8-4c9b-b294-ee4725d2e7b0 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T19:46:37.908815+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=the-best-writers-zone.herokuapp.com request_id=e6844ad9-91b4-4c54-933f-14dbae15ad9f fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T20:13:25.024832+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=the-best-writers-zone.herokuapp.com request_id=fc9f5894-0d17-4ac1-a5f7-818254fc23d3 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-24T20:13:25.523527+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/favicon.ico" host=the-best-writers-zone.herokuapp.com request_id=eaed62df-0870-4645-8015-5184315e0abc fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-25T01:42:15.515460+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=the-best-writers-zone.herokuapp.com request_id=f79386b1-c373-4453-bdde-96ab08f53017 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-25T01:42:16.107145+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/favicon.ico" host=the-best-writers-zone.herokuapp.com request_id=27f19fa7-644b-4846-9b12-03c933a33363 fwd="<IP_ADDRESS>" dyno= connect= service= status=503 bytes= protocol=https
2020-10-25T02:14:40.831724+00:00 app[api]: Starting process with command npm install cool-ascii-faces by user<EMAIL_ADDRESS>2020-10-25T02:14:52.730072+00:00 heroku[run.5255]: State changed from starting to up
2020-10-25T02:14:52.865351+00:00 heroku[run.5255]: Awaiting client
2020-10-25T02:14:52.892754+00:00 heroku[run.5255]: Starting process with command npm install cool-ascii-faces
2020-10-25T02:14:57.031281+00:00 heroku[run.5255]: Process exited with status 127
2020-10-25T02:14:57.063832+00:00 heroku[run.5255]: State changed from up to complete

Since this isn't a Heroku app this is a error on the development side which doesn't relate to the base template


OpenCV build fails on ubuntu with contrib modules

Please state the information for your system

OpenCV version: latest master branch for both repos
Host OS: Linux (Ubuntu 14.04)
Compiler & CMake: GCC 4.8.5 & CMake 3.4.0-rc1

In which part of the OpenCV library you got the issue?
Building the repository with extra contrib modules
Expected behaviour
The nightly buildbot succeeded in building just fine: http://pullrequest.opencv.org/buildbot/builders/master-contrib_coverage-lin64-debug
Actual behaviour
It is breaking down at the waldboost code, exactly what I am aiming to build
[ 20%] Built target pch_Generate_opencv_perf_videoio
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:77:0:
/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src/waldboost.hpp:53:7: error: redefinition of ‘class cv::xobjdetect::WaldBoost’
 class WaldBoost {
       ^
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:48:0:
/usr/local/include/opencv2/xobjdetect.hpp:136:18: error: previous definition of ‘class cv::xobjdetect::WaldBoost’
 class CV_EXPORTS WaldBoost : public Algorithm
                  ^
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:78:0:
/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src/wbdetector.hpp:56:42: error: expected class-name before ‘{’ token
 class WBDetectorImpl : public WBDetector {
                                          ^
/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src/wbdetector.hpp:71:15: error: cannot declare field ‘cv::xobjdetect::WBDetectorImpl::boost_’ to be of abstract type ‘cv::xobjdetect::WaldBoost’
     WaldBoost boost_;
               ^
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:48:0:
/usr/local/include/opencv2/xobjdetect.hpp:136:18: note:   because the following virtual functions are pure within ‘cv::xobjdetect::WaldBoost’:
 class CV_EXPORTS WaldBoost : public Algorithm
                  ^
/usr/local/include/opencv2/xobjdetect.hpp:146:30: note: 	virtual std::vector<int> cv::xobjdetect::WaldBoost::train(cv::Mat&, const cv::Mat&, bool)
     virtual std::vector<int> train(Mat& data,
                              ^
/usr/local/include/opencv2/xobjdetect.hpp:154:19: note: 	virtual float cv::xobjdetect::WaldBoost::predict(const cv::Ptr<cv::xobjdetect::FeatureEvaluator>&) const
     virtual float predict(

@alalek could you maybe take a look why it runs fine on the buildbot but not here? I am building with make -j 24 but also tried single core and it is not a difference, same error.

/usr/local/include/opencv2/xobjdetect.hpp

You have local installation of OpenCV. Try to remove it to eliminate conflict.

Will try so! :)

@alalek havent tried yet because i wanna solve some things before moving on, but could you explain why this generates problems when building opencv_contrib modules and NOT when building a new opencv version?

Because "xobjdetect" is "opencv_contrib" module.
Could you add the last command from make -j1 VERBOSE=1? I will take a look on passed include dirs.

Will do so! But to be correct, I am still thinking that opencv_contrib modules should behave the same as normal modules and thus I should not run into this issue?

Here you go!
[ 10%] Building CXX object modules/videoio/CMakeFiles/opencv_videoio.dir/src/cap_gstreamer.cpp.o
cd /home/spu/Documents/github/opencv_compl/build/modules/videoio && /usr/bin/c++   -DCVAPI_EXPORTS -DVIDEOIO_EXPORTS -D__OPENCV_BUILD=1 -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mssse3 -msse4.1 -msse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -Wno-deprecated-declarations -O3 -DNDEBUG  -DNDEBUG -fPIC -I/home/spu/Documents/github/opencv_compl/3rdparty/ippicv/unpack/ippicv_lnx/include -I/home/spu/Documents/github/opencv_compl/build -isystem /usr/local/include -isystem /usr/include/gtk-2.0 -isystem /usr/lib/x86_64-linux-gnu/gtk-2.0/include -isystem /usr/include/atk-1.0 -isystem /usr/include/cairo -isystem /usr/include/gdk-pixbuf-2.0 -isystem /usr/include/pango-1.0 -isystem /usr/include/gio-unix-2.0 -isystem /usr/include/freetype2 -isystem /usr/include/glib-2.0 -isystem /usr/lib/x86_64-linux-gnu/glib-2.0/include -isystem /usr/include/pixman-1 -isystem /usr/include/libpng12 -isystem /usr/include/harfbuzz -isystem /usr/include/gstreamer-0.10 -isystem /usr/include/libxml2 -I/home/spu/Documents/github/opencv_compl/modules/videoio/include -I/home/spu/Documents/github/opencv_compl/modules/videoio/src -I/home/spu/Documents/github/opencv_compl/build/modules/videoio -I/home/spu/Documents/github/opencv_compl/modules/cudev/include -I/home/spu/Documents/github/opencv_compl/modules/core/include -I/home/spu/Documents/github/opencv_compl/modules/imgproc/include -I/home/spu/Documents/github/opencv_compl/modules/imgcodecs/include -I/home/spu/Documents/github/opencv_compl/modules/ts/include -I/home/spu/Documents/github/opencv_compl/modules/highgui/include -I/home/spu/Documents/github/opencv_compl/modules/videoio/test -I/home/spu/Documents/github/opencv_compl/modules/videoio/perf     -Winvalid-pch  -o CMakeFiles/opencv_videoio.dir/src/cap_gstreamer.cpp.o -c /home/spu/Documents/github/opencv_compl/modules/videoio/src/cap_gstreamer.cpp
/home/spu/Documents/github/opencv_compl/modules/videoio/src/cap_gstreamer.cpp: In member function ‘virtual bool CvVideoWriter_GStreamer::open(const char*, int, double, CvSize, bool)’:
/home/spu/Documents/github/opencv_compl/modules/videoio/src/cap_gstreamer.cpp:1419:25: error: ‘GST_VIDEO_FORMAT_ENCODED’ was not declared in this scope
         input_pix_fmt = GST_VIDEO_FORMAT_ENCODED;
                         ^
/home/spu/Documents/github/opencv_compl/modules/videoio/src/cap_gstreamer.cpp: In member function ‘virtual bool CvVideoWriter_GStreamer::writeFrame(const IplImage*)’:
/home/spu/Documents/github/opencv_compl/modules/videoio/src/cap_gstreamer.cpp:1590:26: error: ‘GST_VIDEO_FORMAT_ENCODED’ was not declared in this scope
     if (input_pix_fmt == GST_VIDEO_FORMAT_ENCODED) {
                          ^
make[2]: *** [modules/videoio/CMakeFiles/opencv_videoio.dir/src/cap_gstreamer.cpp.o] Error 1
make[2]: Leaving directory `/home/spu/Documents/github/opencv_compl/build'
make[1]: *** [modules/videoio/CMakeFiles/opencv_videoio.dir/all] Error 2
make[1]: Leaving directory `/home/spu/Documents/github/opencv_compl/build'
make: *** [all] Error 2

This log is about another issue actually

Yeah I just updated and deleted it :D I am running the correct code now :P Gimme a second :dog:

Ah damn, the other error comes sooner :D gimme a second to fix it here locally :P

Here you go!
[ 51%] Generating precomp.hpp.gch/opencv_xobjdetect_Release.gch
cd /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect && /usr/bin/cmake -E make_directory /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp.gch
cd /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect && /usr/bin/c++ -O3 -DNDEBUG -DNDEBUG -fPIC "-D__OPENCV_BUILD=1" "-DCVAPI_EXPORTS" -isystem"/home/spu/Documents/github/opencv_compl/3rdparty/ippicv/unpack/ippicv_lnx/include" -isystem"/home/spu/Documents/github/opencv_compl/build" -isystem"/usr/local/include" -isystem"/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/include" -isystem"/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src" -isystem"/home/spu/Documents/github/opencv_compl/build/modules/xobjdetect" -I"/home/spu/Documents/github/opencv_compl/modules/cudev/include" -I"/home/spu/Documents/github/opencv_compl/modules/core/include" -I"/home/spu/Documents/github/opencv_compl/modules/imgproc/include" -I"/home/spu/Documents/github/opencv_compl/modules/ml/include" -I"/home/spu/Documents/github/opencv_compl/modules/imgcodecs/include" -I"/home/spu/Documents/github/opencv_compl/modules/videoio/include" -I"/home/spu/Documents/github/opencv_compl/modules/highgui/include" -I"/home/spu/Documents/github/opencv_compl/modules/objdetect/include" -isystem"/home/spu/Documents/github/opencv_compl/3rdparty/ippicv/unpack/ippicv_lnx/include" -isystem"/home/spu/Documents/github/opencv_compl/build" -isystem"/usr/local/include" -isystem"/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/include" -isystem"/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src" -isystem"/home/spu/Documents/github/opencv_compl/build/modules/xobjdetect" -I"/home/spu/Documents/github/opencv_compl/modules/cudev/include" -I"/home/spu/Documents/github/opencv_compl/modules/core/include" -I"/home/spu/Documents/github/opencv_compl/modules/imgproc/include" -I"/home/spu/Documents/github/opencv_compl/modules/ml/include" -I"/home/spu/Documents/github/opencv_compl/modules/imgcodecs/include" -I"/home/spu/Documents/github/opencv_compl/modules/videoio/include" -I"/home/spu/Documents/github/opencv_compl/modules/highgui/include" -I"/home/spu/Documents/github/opencv_compl/modules/objdetect/include" -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mssse3 -msse4.1 -msse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -DCVAPI_EXPORTS -x c++-header -o /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp.gch/opencv_xobjdetect_Release.gch /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:77:0:
/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src/waldboost.hpp:53:7: error: redefinition of ‘class cv::xobjdetect::WaldBoost’
 class WaldBoost {
       ^
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:48:0:
/usr/local/include/opencv2/xobjdetect.hpp:136:18: error: previous definition of ‘class cv::xobjdetect::WaldBoost’
 class CV_EXPORTS WaldBoost : public Algorithm
                  ^
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:78:0:
/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src/wbdetector.hpp:56:42: error: expected class-name before ‘{’ token
 class WBDetectorImpl : public WBDetector {
                                          ^
/home/spu/Documents/github/opencv_contrib/modules/xobjdetect/src/wbdetector.hpp:71:15: error: cannot declare field ‘cv::xobjdetect::WBDetectorImpl::boost_’ to be of abstract type ‘cv::xobjdetect::WaldBoost’
     WaldBoost boost_;
               ^
In file included from /home/spu/Documents/github/opencv_compl/build/modules/xobjdetect/precomp.hpp:48:0:
/usr/local/include/opencv2/xobjdetect.hpp:136:18: note:   because the following virtual functions are pure within ‘cv::xobjdetect::WaldBoost’:
 class CV_EXPORTS WaldBoost : public Algorithm
                  ^
/usr/local/include/opencv2/xobjdetect.hpp:146:30: note: 	virtual std::vector<int> cv::xobjdetect::WaldBoost::train(cv::Mat&, const cv::Mat&, bool)
     virtual std::vector<int> train(Mat& data,
                              ^
/usr/local/include/opencv2/xobjdetect.hpp:154:19: note: 	virtual float cv::xobjdetect::WaldBoost::predict(const cv::Ptr<cv::xobjdetect::FeatureEvaluator>&) const
     virtual float predict(
                   ^
make[2]: *** [modules/xobjdetect/precomp.hpp.gch/opencv_xobjdetect_Release.gch] Error 1
make[2]: Leaving directory `/home/spu/Documents/github/opencv_compl/build'
make[1]: *** [modules/xobjdetect/CMakeFiles/pch_Generate_opencv_xobjdetect.dir/all] Error 2
make[1]: Leaving directory `/home/spu/Documents/github/opencv_compl/build'
make: *** [all] Error 2

Manually deleting the /usr/local/include/opencv and /usr/local/include/opencv2 did the trick! However I am not convinced that this is what I should be doing. Normally I execute make -j 24 to first build OpenCV then I do a sudo make install to allow OpenCV to overwrite the /usr/local/include directories using sudo rights. So in my opinion it is weird that the contrib modules are looking for existing OpenCV installations, while the normal modules are not.

Thanks!
For some reason some directories are passed via "-isystem" parameter.

All directories named by -isystem are searched after all directories named by -I, no matter what their order was on the command line.

https://gcc.gnu.org/onlinedocs/cpp/System-Headers.html
At this moment I have no idea about isystem usages here.

Okay, then lets keep this open and hope that we find a solution someday :)

/usr/local/include is in default GCC include path list
$ gcc -xc++ -E -v - <<< ""
...
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/backward
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/include
 /usr/local/include
 /usr/include
End of search list.
...

But both -I and -isystem paths goes before any defaults (like /use/local/include) and works as expected:
$ gcc -xc++ -E -v -I/home/alalek/projects/opencv_dev/modules/core/include - <<< ""
#include "..." search starts here:
#include <...> search starts here:
 /home/alalek/projects/opencv_dev/modules/core/include
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/backward
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/include
 /usr/local/include
 /usr/include
End of search list.

and
$ gcc -xc++ -E -v -isystem/home/alalek/projects/opencv_dev/modules/core/include - <<< ""
#include "..." search starts here:
#include <...> search starts here:
 /home/alalek/projects/opencv_dev/modules/core/include
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/backward
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/include
 /usr/local/include
 /usr/include
End of search list.

In attached command line there is -isystem"/usr/local/include" entries and it is bad.
Could you "grep" CMakeCache.txt and CMakeVars.txt files for "/usr/local/include"?

That returns me
spu@TOBCAT:~/Documents/github/opencv_compl/build$ grep '/usr/local/include' CMakeCache.txt CMakeVars.txt
CMakeCache.txt:EIGEN_INCLUDE_PATH:PATH=/usr/local/include
CMakeVars.txt:EIGEN_INCLUDE_PATH=/usr/local/include
CMakeVars.txt:JAVA_AWT_INCLUDE_DIRECTORIES=[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Development Kit\1.4;JavaHome]/include;[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Development Kit\1.3;JavaHome]/include;[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Development Kit\registry;JavaHome]/include;/include;/usr/include;/usr/local/include;/usr/lib/java/include;/usr/local/lib/java/include;/usr/lib/jvm/java/include;/usr/lib/jvm/java-6-sun/include;/usr/lib/jvm/java-1.5.0-sun/include;/usr/lib/jvm/java-6-sun-<IP_ADDRESS>/include;/usr/lib/jvm/java-6-openjdk/include;/usr/local/share/java/include;/usr/lib/j2sdk1.4-sun/include;/usr/lib/j2sdk1.5-sun/include;/opt/sun-jdk-<IP_ADDRESS>/include;/usr/lib/jvm/default-java/include;/usr/local/jdk-1.7.0/include;/usr/local/jdk-1.6.0/include
CMakeVars.txt:EIGEN_INCLUDE_PATH=/usr/local/include
CMakeVars.txt:JAVA_AWT_INCLUDE_DIRECTORIES=[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Development Kit\1.4;JavaHome]/include;[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Development Kit\1.3;JavaHome]/include;[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Development Kit\registry;JavaHome]/include;/include;/usr/include;/usr/local/include;/usr/lib/java/include;/usr/local/lib/java/include;/usr/lib/jvm/java/include;/usr/lib/jvm/java-6-sun/include;/usr/lib/jvm/java-1.5.0-sun/include;/usr/lib/jvm/java-6-sun-<IP_ADDRESS>/include;/usr/lib/jvm/java-6-openjdk/include;/usr/local/share/java/include;/usr/lib/j2sdk1.4-sun/include;/usr/lib/j2sdk1.5-sun/include;/opt/sun-jdk-<IP_ADDRESS>/include;/usr/lib/jvm/default-java/include;/usr/local/jdk-1.7.0/include;/usr/local/jdk-1.6.0/include

Oh and BTW, once I deleted those folders manually, I finished building it to 100% complete!

Good to know, but deleting directories is just a workaround.
The real problem is still here.

True ^_^

Great,
I have this (not /usr/local/include):
EIGEN_INCLUDE_PATH=/usr/include/eigen3

And this in the command like:
-isystem /usr/include/eigen3

So it is enough to trying to fix something on my side.

True but I would actually prefer, and it seems way more logical to me, that if I decide to install Eigen on a shared system, that it is only installed under the local user repository, and not under the global user repository right?

Problem is not with eigen, it is with contrib modules directories which are passed via -isystem.
Could you take a look on this fix: https://github.com/alalek/opencv/commits/issue_6403

@alalek I will be testing it tomorrow at work! I will keep you posted!

Yep that did the trick! Just tested it and building with the existing folders works perfectly fine now!


will thunder include kalman filter-smoother at some point?

we have some new stuff on parallelize high-dimensional KFS, which perhaps we should implement in your framework?

That definitely sounds interesting! Potentially something to add as an operation on the TimeSeries object. Is this intended for massively univariate or multivariate time series operations? Can you tell us more about the use cases / algorithm / implementation?

massively multivariate :)
uses case 1 was: whole brain fMRI
use case 2 is about to be: whole fish brain!
in brief, the algorithm is a penalized expectation maximization algorithm,
which uses fista for the m-step.
everything factorizes quite nicely, so i think spark might be ideal or
overkill :)
i could send you a pre-print describing the algorithm,
we hope to post the manuscript to arxiv in a couple weeks,
we are just finalizing wordsmithing now....
On Wed, May 13, 2015 at 2:12 PM, Jeremy Freeman<EMAIL_ADDRESS>wrote:

That definitely sounds interesting! Potentially something to add as an
operation on the TimeSeries object. Is this intended for massively
univariate or multivariate time series operations? Can you tell us more
about the use cases / algorithm / implementation?
—
Reply to this email directly or view it on GitHub
https://github.com/thunder-project/thunder/issues/180#issuecomment-101762773
.

--
the glass is all full:  half water, half air.
openconnecto.me, jovo.me, office hours
https://www.google.com/calendar/embed?src=e2ktu4lrgul8anp8hclrcminp8%40group.calendar.google.com&ctz=America/New_York


kvstore Entry exposes embedded io.Reader but consumes its contents in String

Code review comment from @peterbourgon:

Embedding an io.Reader in this way allows callers to treat Entry as an io.Reader directly, which doesn't seem to be correct, as methods like String consume from that Reader and mutate internal state in a way that callers aren't able to do.

I suspect he's right but I don't know the intention here.  @dgryski, do you know?
Un-embedding io.Reader would mean a breaking API change, so it'd at least need a bump to the next 0.x "major" version.  (0.2.0 as of this writing)

The idea was so that you could stream a response from the KV store out to the client without having to hold the entire value in memory.

A single type that implements both io.Reader and fmt.Stringer is very tricky. A correct implementation must allow a caller to call Read with a buffer of size 3 and get the first 3 bytes of the value, and then call String and get the entire value, and then call Read again with a huge buffer and get all of the bytes from the 4th byte forward. I'm not sure how you'd do this without caching the value in memory.
My suggestion would be to provide two separate types, constructors, etc. for values: one which implements io.Reader, and one which implements fmt.Stringer (or is just a string directly).

Another option could be to replace String() with a method that makes it more clear that it consumes the reader.  Essentially io.ReadAll but to a string rather than byte slice.

It was too late for us to change this behavior, so I've documented it instead.  In practice I think few people will run into it, since they'll be using KV store values in one of two rather different ways (small data where strings are nice, or larger files they probably want to stream to clients or other backends)


master branch does not build cleanly (Streams)

A clean checkout of math-classes fails to build because two files seem to mistakenly import MathClasses.theory.Streams instead of Coq.Lists.Streams.
The tail end of the build fails with:
Warning: in file ./interfaces/canonical_names.v, library MathClasses.theory.Streams is required and has not been found in the loadpath!
COQC interfaces/canonical_names.v
File "./interfaces/canonical_names.v", line 2, characters 0-34:
Warning: Option Automatic Introduction is deprecated
[deprecated-option,deprecated]
File "./interfaces/canonical_names.v", line 3, characters 0-38:
Warning: Option Automatic Coercions Import is deprecated
[deprecated-option,deprecated]
File "./interfaces/canonical_names.v", line 5, characters 15-41:
Error: Unable to locate library MathClasses.theory.Streams.

I was able to get past this by changing the imports of interfaces/canonical_names.v and theory/streams.v.

That was introduced by this PR:
https://github.com/math-classes/math-classes/pull/52
This is strange because the CI passed. Perhaps @ppedrot knows more.

theory/streams is included in
https://github.com/math-classes/math-classes/blob/master/Make

Is  the problem that you have two files with different capitalization? Streams.v and streams.v both in the theory directory.  At least on OS X, the second overwrites the first.  Perhaps rename the latter?

Thanks! Done. I did it directly from the github interface, so hopefully did not create any new problems.

It looks like you also need to update the import in interfaces/canonical_names.v to be CoqStreams.

Thanks. Done.

@spitters A piece of advice: when you are not sure that the change will compile, you can do it in another branch and open a PR. This way, Travis will run and will confirm that there are no errors.