[Question] A question towards HugeCTR::concurrent_unordered_map::get_insert

I'm studying hashtable implementation and find that it might hang in some condition theoriotically. However, although I can reproduce with a similar code snippet, I can't repduce when launching HugeCTR training. I guess there is "magic" trick behind the scenes.
https://github.com/NVIDIA-Merlin/HugeCTR/blob/main/HugeCTR/include/hashtable/cudf/concurrent_unordered_map.cuh#L642
It is clearly a branch divergence case. Line 649 depends on Line 644 to move on.
Assuming two threads in a same warp call this function with same key(this key doesn't exist in hashtable before) and the compiler compiles Line 648 ahead of Line 642, HugeCTR would hang theoriotically, but it seems not happening.
A small case which will hang on A100, cuda 11.7.

hi @heroes999 thanks for trying hugectr.
I believe the reason why hugectr's code can work well is because in hugectr we set the writing address as volatile so that the compilers make sure the writing can be viewed in other threads. For more detailed information, you can check cuda programing guide

Thanks for reply. @shijieliu
But I think this case might relate to branch divergence -- all threads in a warp should execute the same instructions so that if Line649 is executed first, other threads in the same warp are inactive(sleep). Even with the volatile qualifier, I'm not sure it will do any help if the needed thread(to change existing_value) hangs.

@heroes999 Thanks for your questions.
With the volatile qualifier, your test case will not hang because the compiler can handle warp divergence from volatile variable properly. Saying thread16-31 is executed first, and it finds that dst[tid%16] != src[tid%16], it will NOT loop back immediately. Instead, thread0-15 gets executed and sets the volatile variable then. After that, thread16-31 reloads the volatile variable and gets away from the loop. Without the volatile qualifier, this warp divergence can cause hanging because thread16~31 will loop back forever if it gets executed first.
You can use Nsight Compute  to view the compiled instructions to understand this mechanism.
#include <cuda_runtime.h>
#include <iostream>

__global__ void test_kernel(int *src, volatile int *dst) {
  int tid = threadIdx.x + blockIdx.x * blockDim.x;
  if (tid < 16) {
    dst[tid] = 0;
  } else {
    while (dst[tid%16] != src[tid%16]) {}
  }
}

int main() {
  int h_src[16] = {0};
  int h_dst[16] = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
  int *d_src, *d_dst;
  cudaStream_t stream;
  cudaStreamCreateWithPriority(&stream, cudaStreamNonBlocking, 0);
  cudaMalloc((void**)&d_src, 16*sizeof(int));
  cudaMalloc((void**)&d_dst, 16*sizeof(int));
  cudaMemcpyAsync(d_src, h_src, 16 * sizeof(int), cudaMemcpyHostToDevice, stream);
  cudaMemcpyAsync(d_dst, h_dst, 16 * sizeof(int), cudaMemcpyHostToDevice, stream);
  test_kernel<<<1, 64, 0, stream>>>(d_src, d_dst);
  cudaDeviceSynchronize();
  cudaStreamDestroy(stream);
  cudaFree(d_dst);
  cudaFree(d_src);
  return 0;
}

It still hangs both on A100/cuda 11.7 RTX3090/cuda 11.7，any additional compiler params should be added? @KingsleyLiu-NV

@heroes999
I still cannot reproduce the hang on A100/cuda11.7, and the order of conditional branch does not matter. Here are the code and log files:
test_warp_divergence.cu.txt
test_warp_divergence.log
Can you please do cuobjdump -sass {YOUR_EXECUTABLE} to check the instructions for your hanging case?

Thanks @KingsleyLiu-NV, I've located the root cause - the default arch of nvcc is compute_52... which I assume is at least 70.
This case will no longer hang with compute capacity over 70.


Live content being deleted on page publish

Just creating an issue for the moment because I want someone to double check it before I PR anything - basically, on the publication of a page containing an element area, other elements will be deleted. I've traced it as best as possible to https://github.com/dnadesign/silverstripe-elemental/blob/master/code/extensions/ElementPublishChildren.php#L20  - the element set that it looks up for potential deletion are those that belong to either to $this->owner->Elements() (ie where ListID = $this->owner->ID) OR where the ParentID = $this->owner->ID - which in fact is looking up elements that are attached to an ElementArea with that ID. These elements subsequently get deleted from live.
So while changing this fixes the issue, I'm just wondering if there's some other logic being expected, as in, whether the ParentID comparison has a valid reason for existing.

I wrote the original code, can't see why I went for ParentID, as the extension within the module is purely used for ElementList.. I think perhaps at some time ElementPublishChildren was also added to SiteTree or even ElementArea which used the widget ParentID. Hence why ElementPublishChildren is an extension rather than just on ElementList.
I think it might be good to simply use ListID, remove ParentID and allow the name of any additional filters be passed through the extension if needed.
private static $extensions = array(
'ElementPublishChildren("ParentID")'
);
By default that can be left off for ListID.


The 'reference' attribute for image parameters is not described in the XSD yet

Apparently, a 'reference' attribute has been added to <image> parameters in the execution model, but
Libs/CommandLineModules/Core/Resources/ctkCmdLineModule.xsd has not been updated yet accordingly.
I am seeing 38 such occurrences in yesterday's nightly build.

I just noticed that the 'reference' attribute is also found in the description of transform, geometry, and table parameters.

Also, for transform, I wonder whether we would need two attributes (one for the fixed and one for the moving image)?
According to @jcfr , the attribute is not used anywhere yet.  It would make sense to collect input on how people intend to use it.  (I guess @fedorov could want to use it for provenance in DICOM exports, for instance?)

Yes, for a dicom registration object you would want the two frames of
reference that are connected by the transform, so having the fixed
reference and the moving reference would make a lot of sense.
On Thu, Jan 7, 2016 at 5:14 PM, Hans Meine<EMAIL_ADDRESS>wrote:

Also, for transform, I wonder whether we would need two attributes (one
for the fixed and one for the moving image)?
According to @jcfr https://github.com/jcfr , the attribute is not used
anywhere yet. It would make sense to collect input on how people intend to
use it. (I guess @fedorov https://github.com/fedorov could want to use
it for provenance in DICOM exports, for instance?)
—
Reply to this email directly or view it on GitHub
https://github.com/commontk/CTK/issues/623#issuecomment-169823245.

I am not familiar deeply enough with this specific topic (i.e., what is the relationship between XSD mentioned here and Slicer execution model), but "reference" is an ambiguous term. As an example, Slicer does not have a concept to parallel DICOM Frame of reference, and "reference" attribute you mention here probably refers to the image node, which may or may not have a DICOM representation in the database. In Slicer, "reference" attribute is used in quite a few CLIs here: https://github.com/Slicer/Slicer/tree/master/Modules/CLI, but I believe its purpose is to initialize things like how the output should look like (window level, lookup table), and its semantics will probably depend on the object type.

I think it would be helpful to think of what is needed to support proper
tracking so that all inputs and outputs of CLIs can be used to make
well-formed DICOM objects.  I've looked at this problem in the past and it
seems tractable, but in some cases it may not be identical to the semantics
of current flags.  That is to say that making CLIs fully DICOM compatible
will take more than this one change.
On Fri, Jan 8, 2016 at 5:25 PM, Andrey Fedorov<EMAIL_ADDRESS>wrote:

I am not familiar deeply enough with this specific topic (i.e., what is
the relationship between XSD mentioned here and Slicer execution model),
but "reference" is an ambiguous term. As an example, Slicer does not have a
concept to parallel DICOM Frame of reference, and "reference" attribute you
mention here probably refers to the image node, which may or may not have a
DICOM representation in the database. In Slicer, "reference" attribute is
used in quite a few CLIs here:
https://github.com/Slicer/Slicer/tree/master/Modules/CLI, but I believe
its purpose is to initialize things like how the output should look like
(window level, lookup table), and its semantics will probably depend on the
object type.
—
Reply to this email directly or view it on GitHub
https://github.com/commontk/CTK/issues/623#issuecomment-170144720.

I agree with both of you; this issue has already brought up the following questions:

What was the intention behind the reference attribute?

I think @jcfr said that it was not used in Slicer at all.
@fedorov suspected it to be intended used for LUT settings.

It could be used for provenance tracking (which I assumed it was meant for), in which case we may want to make sure that it has "enough provenance" for later DICOM exports.
All of these questions need to be separately looked at for the four mentioned parameter types (starting with image).

I'd be interested in looking at the problem from a different perspective:
if we consider CLIs to be mappings from one set of DICOM instances to
another set, then what bookkeeping is needed to track the provenance.  I
would then think about how to wrap and augment existing CLIs so they can be
used in the DICOM context.  I'd contend that if the DICOM bookkeeping is
done well, then it will also allow more sophisticated behavior at the
application level, such as associating the correct frames of references for
transforms etc.
On Sun, Jan 10, 2016 at 3:39 AM, Hans Meine<EMAIL_ADDRESS>wrote:

I agree with both of you; this issue has already brought up the following
questions:

What was the intention behind the reference attribute?

I think @jcfr https://github.com/jcfr said that it was not used
in Slicer at all.
@fedorov https://github.com/fedorov suspected it to be intended
used for LUT settings.

It could be used for provenance tracking (which I assumed it was
meant for), in which case we may want to make sure that it has "enough
provenance" for later DICOM exports.
All of these questions need to be separately looked at for the four
mentioned parameter types (starting with image).

—
Reply to this email directly or view it on GitHub
https://github.com/commontk/CTK/issues/623#issuecomment-170326233.

Yes, I would be very interested to see proposals on how this perspective should be approached. But I think this perspective is too broad for me to suggest one myself. It is not even clear what would be the mapping from the XSD-supported element types into DICOM IODs, forget operations on those.
From my point, I want first to see tools in place to handle DICOM conversions explicitly via converter CLIs for now. It will not be easy (and maybe even counterproductive) to generalize semantics of an arbitrary CLI into XSD attributes for the purposes of DICOM attribute mapping/transformation.

My suggestion is to ignore the XSD and current CLI support and instead
focus on the best fit for operations on DICOM instances.  Then go back and
adapt for legacy slicer execution module CLIs either by rewriting or by
providing an adaptor.
On Sun, Jan 10, 2016 at 3:28 PM, Andrey Fedorov<EMAIL_ADDRESS>wrote:

Yes, I would be very interested to see proposals on how this perspective
should be approached. But I think this perspective is too broad for me to
suggest one myself. It is not even clear what would be the mapping from the
XSD-supported element types into DICOM IODs, forget operations on those.
From my point, I want first to see tools in place to handle DICOM
conversions explicitly via converter CLIs for now. It will not be easy (and
maybe even counterproductive) to generalize semantics of an arbitrary CLI
into XSD attributes for the purposes of DICOM attribute
mapping/transformation.
—
Reply to this email directly or view it on GitHub
https://github.com/commontk/CTK/issues/623#issuecomment-170391093.


Cdr 1882 remove deprecated modules

You can actually go ahead and delete the contents of the three modules that you removed as well.  They don't automatically go away because of being ignored since they are already registered


Add automatic log file rotation

Remove or create new log file after some time.

Will we rotate logs based on file size or time?
The amount of logs already rotated kept should be also passed as a parameter?

Will we rotate logs based on file size or time?

I think we should support both log rotation variants.

The amount of logs already rotated kept should be also passed as a parameter?

What do you mean? I think that on logger creation should be passed rotation type (size and/or time), and how many compressed logs to store. On log rotation event close current file, compress it and rename to current date and time or increment id, then open new log file.

Added log file rotation base on time. https://github.com/cfnptr/logy/commit/562c9522bcd18a2eecef6ad2a3eec68a552964c5


is DataProductOwner mandatory or optional?

In the data product specification the dataProductOwner field is set to be optional.
I expected it to be mandatory, since a data product must have an owner
Is there anything I am missing?

mandatory is better


feat: upgrade Lightning version

Description
In this PR, I have upgraded the Lightning version from 2.9.0 to 2.11.0. This upgrade resolves the inline content spacing issue that was present in Lightning 2.10.0. Now, the inline content is correctly displayed as expected.
References
LUI-889
Testing
It would be great if we can verify that this upgrade did not introduce any new bugs or issues before we merge this.

Test Execution Passed.

:tada: This PR is included in version @lightningjs/ui-components-test-utils-v1.4.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Check and ignore ANP and ACNP in agent when AntreaPolicy is not enabled

If the AntreaPolicy feature gate is not enabled on the Antrea Agent but
is enabled on the Antrea Controller (e.g. due to misconfiguration, or
Controller/Agents have not been all restarted after a configuration
change), Agent might crash with a segment fault when receiving and
processing ANP or ACNP messages from controller, so this commit adds
a check in Agent NetworkPolicyController to ingnore ANP and ACNP
messages when AntreaPolicy is not enabled.

Codecov Report

Merging #1336 into master will decrease coverage by 9.67%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1336      +/-   ##
==========================================
- Coverage   54.47%   44.80%   -9.68%
==========================================
  Files         119       75      -44
  Lines       11240     5598    -5642
==========================================
- Hits         6123     2508    -3615
+ Misses       4526     2810    -1716
+ Partials      591      280     -311

Flag
Coverage Δ

#integration-tests
44.80% <ø> (ø)

#unit-tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/agent/proxy/service.go
0.00% <0.00%> (-100.00%)
:arrow_down:

pkg/ovs/ovsconfig/default.go
0.00% <0.00%> (-100.00%)
:arrow_down:

pkg/agent/util/iptables/lock.go
0.00% <0.00%> (-81.82%)
:arrow_down:

pkg/agent/proxy/endpoints.go
0.00% <0.00%> (-81.43%)
:arrow_down:

pkg/cni/client.go
0.00% <0.00%> (-75.52%)
:arrow_down:

...kg/agent/flowexporter/connections/conntrack_ovs.go
0.00% <0.00%> (-73.15%)
:arrow_down:

pkg/agent/flowexporter/connections/conntrack.go
0.00% <0.00%> (-68.43%)
:arrow_down:

pkg/agent/querier/querier.go
0.00% <0.00%> (-66.67%)
:arrow_down:

pkg/agent/config/traffic_encap_mode.go
35.71% <0.00%> (-64.29%)
:arrow_down:

pkg/agent/proxy/proxier.go
0.00% <0.00%> (-38.94%)
:arrow_down:

... and 59 more

The commit message and log messages mention the AntreaProxy feature gate instead of AntreaPolicy :)

Thanks. Fixed it.

/test-all

/test-all

/test-all


Dropbox moving away from long-lived access tokens

I got the following from Dropbox via email:

The Dropbox API now supports OAuth scopes, PKCE, refresh tokens, and short-lived access tokens. We’re asking developers to review their app’s permission settings and ensure their apps support short-lived access tokens. Apps that require background access will need to update to use refresh tokens, which is made easier by using our updated SDKs.

Starting September 30th, 2021, the Dropbox OAuth flow will no longer return long-lived access tokens. It will instead return short-lived access tokens, and optionally return refresh tokens. Please be sure to review, test, and move to the new permission model before then.

In the code I see that refresh tokens are missing from the dropboxfs API (e.g. in the opener). From the email, it looks as though they will be needed from Sep 30, 2021.
https://dropbox.tech/developers/now-available--scoped-apps-and-enhanced-permissions

I'm still using this package. Is anyone here still working on it? It seems like it's going to stop working in 30 days.


Package can't be imported with version (only using branch)

Describe the bug
A package can be imported through SPM using a branch or a version number.
The tags in the repository are currently of the form vx.y.z, so adding the package is impossible using a specific version.
SDK Version

Client: Xcode/Swift pacakge

To Reproduce
Steps to reproduce the behavior:

Create new Xcode project
Add a package dependency in the UI
Enter the repo url
See the last available version is 0.6.0

Same step to reproduce if building a package. In package.swift:
dependencies: [
.package(url: "https://github.com/WalletConnect/WalletConnectSwiftV2", .exact("1.0.4"))
]

Expected behavior
Should be able to import package by version number
Screenshots

Device (please complete the following information):
Xcode
Additional context
N/A

hey, @DarthMike it is fixed now, you can try last exact version 1.0.4

Thanks @llbartekll


feat(kubeconfig): add mcp controlplane kubeconfig

Description of your changes
add mcp controlplane kubeconfig

Fixes #
I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable to ensure this PR is ready for review.
[ ] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested
kubectl apply -f examples/iam/robot.yaml
kubectl apply -f examples/iam/robotteammembership.yaml
kubectl apply -f examples/iam/team.yaml
kubectl apply -f examples/iam/token.yaml
kubectl apply -f examples/mcp/controlplanepermission.yaml
kubectl apply -f examples/mcp/controlplanekubeconfig.yaml

kubectl get secret mycp-kubeconfig -n upbound-system -o jsonpath="{.data.kubeconfig}" | base64 -d >> mycp-kubeconfig.yaml

KUBECONFIG=mycp-kubeconfig.yaml kubectl get pods -A
Error from server (Forbidden): pods is forbidden: User "upbound-cloud-impersonator" cannot list resource "pods" in API group "" at the cluster scope

KUBECONFIG=mycp-kubeconfig.yaml kubectl get crds
NAME                                                         CREATED AT
[...]
bucketaccelerateconfigurations.s3.aws.upbound.io             2023-07-10T14:03:17Z
[...]

lint: level=error msg="Timeout exceeded: try increasing it by passing --timeout option"

@haarchri Would it make sense to implement ControlPlane as MR and have the kubeconfig in the connection secret? When user wants only the kubeconfig, they can set the management policy to Observe.

yes currently on it - will update the PR shortly
kubectl get controlplane.mcp
NAME               READY   SYNCED   EXTERNAL-NAME      ID                                     DEPLOYED-CONFIGURATION      CONFIGURATION-STATUS   AGE
haarchri-test-02   True    True     haarchri-test-02   72b0e134-e1e6-404a-9086-350e4543fc91   testing-configuration-rds   ready                  47m

@muvaf  done


Compilation error

Just trying this out as it seems to be great to test fragments in storybook, but it keeps crashing with this error :
ModuleParseError: Module parse failed: Unexpected token (56:5)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| } = RelayConcreteNode;
|
> type ValueResolver = (
|   typeName: string | null | undefined,
|   context: MockResolverContext,
...

Storybook install is pretty recent, component written in TS

Hi @maxaggedon, I'm not actively maintaining (nor using) the package anymore. It was implemented when Relay v11 and Storybook v6.3 were out, and both of these have changed drastically in their newest versions.
My recommendation would be to fork the project, and try to upgrade packages like the relay-test-utils package (used in this file) to match your version of Relay.


Changed from node12 to node16

I'm interested in testing this before merging it.

Yeah it's why I haven't merged it, just thinking how to go about testing it, maybe a branch on the vz-orchestration repo that uses this branch's runner?


Allow PageView to set a cacheExtent

It would be nice to have a PageView that has a cacheExtent specified, so that pages could be pre-built.
This is currently blocked because  the iOS accessibility bridge does not look for an allowImplicitScrolling flag on the parent scrollable - it only looks to see if there are additional scroll children in the tree.  Setting a cache extent would then automatically enable implicit scrolling for PageViews in Voiceover mode, which we do not want.  If we can figure out a good way to address the iOS bridge issue, it should be fairly trivial to expose a cacheExtent parameter on the PageView.
/cc @lukepighetti who has done some work on this on the framework side.
Unfortunately, the iOS bug is not trivial to fix.

What if we extend Viewport and override describeSemanticsClip to only include its cacheExtent if allowImplicitScrolling is false?

The problem isn't the clip, it's about whether swiping right or left should allow the page view to advance or not.

Sorry but I fail to see what the difference is when it comes to the implicitscrolling
According to tge documentation of the sematics clip method it seems to determine whether or not children of the viewport are exposed to implicit scrolling for voiceover:
https://github.com/flutter/flutter/blob/707aa49e42db746a8f4528e4700fb9017c5969d8/packages/flutter/lib/src/rendering/object.dart#L2408-L2433
So in my interpretation if this it should allow us to circumvent the cache extent issue for PageView without enabling implicit scrolling

I'm pretty sure that if you don't include the nodes, the iOS bridge won't be able to scroll properly. But it's been a little while since I dug into this, so I may be wrong. If you're able to use sematnics clip to make this work without breaking iOS that sgtm.

Adding customer: money to this since they have a PageView that's showing some jankiness in transition, and being able to cache the next page might help that.

Hello @dnfield
May be this is related issue
https://github.com/flutter/flutter/issues/49126

Before Flutter add allowImplicitScrolling, I create a widget to solve the set cacheExtent problem. I add a parameter preloadPagesCount to page_view.dart to help people preload specific pages, here is the widget
The pseudocode of preloadPagesCount logic is:
if (preloadPagesCount < 1) {
  cacheExtent = 0
}
else if (preloadPagesCount == 1) {
  cacheExtent = 1
} else if (widget.scrollDirection == Axis.horizontal) { // horizontal scroll, calculate preload base on page width
  MediaQuery.of(context).size.width * preloadPagesCount - 1 // currentPagerWidth * preloadPagesCount = the extent which need to preload
} else {
MediaQuery.of(context).size.height * preloadPagesCount - 1 // `-1` is a magic number I wrote 3 years ago, I think I add it because we need `Math.floor()`(for example height=10.9, count = 1, height * count = 10.9, without -1 the cacheExtent value will set to 11 which means two pages)
}

I could use Math.floor(MediaQuery.of(context).size.width) * preloadPagesCount to replace the magic number -1. If you think the logic is reasonable I'm happy to take this issue ticket and create a PR to the Flutter repo :)

Any updates?

I'm very surprised that such a necessary component of PageView has not been implemented after several years. Could we please receive some sort of update on this?

Any updates?

update please

so?

still no update on this

Use react native

Use react native

yes and have a slower app 👍🏾

A not so good method: if our app does not need to consider auxiliary features, the problem is easy to solve, copy the source code of PageView, and then pass cacheExtent


error event sometimes emitted

From what I understand, needle should always pass an error to the callback and never emit an error event.
However, I've processed in production crash with an ETIMEOUT error. I've been able to reproduce it locally.
It seems to happen when the host is unreachable. It's a simple GET request and I pass the following options:
var defaultOptions = {
  timeout: 5000,
  read_timeout: 5000,
  rejectUnauthorized: false
};

The extended stack trace I get:
message connect ETIMEDOUT
stack Error: connect ETIMEDOUT
    at errnoException (net.js:905:11)
    at afterConnect (net.js:896:19)
---------------------------------------------
    at Readable.on (_stream_readable.js:708:33)
    at http.js:1754:12
    at process._tickDomainCallback (node.js:492:13)
---------------------------------------------
    at ClientRequest.onSocket (http.js:1731:11)
    at new ClientRequest (http.js:1433:10)
    at exports.request (https.js:123:10)
    at Needle.send_request (/Users/laurent/dev/front/node_modules/needle/lib/needle.js:373:26)
    at Needle.start (/Users/laurent/dev/front/node_modules/needle/lib/needle.js:291:15)
    at exports.(anonymous function) (/Users/laurent/dev/front/node_modules/needle/lib/needle.js:581:61)
    at exports.listMessagesSince (/Users/laurent/dev/front/server/connectors/facebook/facebook.js:134:11)
    at FacebookPump._poll (/Users/laurent/dev/front/server/pumps/facebook/facebook_pump.js:75:12)
    at listOnTimeout (timers.js:121:15)
---------------------------------------------

Can you share the code you're using so I can reproduce the error?

Not easily, but I've added logs everywhere in my local copy of needle, so I'll be able to provide more info soon.

OK, I've been able to reproduce the problem once (but it took several hours). From what I see, I get a socket timeout after you remove the 'error' handler.
I get first an ECONNRESET which is correctly reported by needle, followed by a ETIMEDOUT which is not caught.
I should also have mentioned that I'm still on node.js 0.10.40, so it might be a node.js bug. Would you accept a PR that would basically do request.on('error', function() {}); after you remove the real handler?
set error handler
remove error handler
set error handler
remove error handler
set error handler
needle timeout
had_error
remove error handler
{ [Error: socket hang up] code: 'ECONNRESET' }
Fri Oct 09 2015 19:13:50 GMT-0700 (PDT) #error# facebook - Poll error with inbox: 2, company: front
data { status: 'internal_error',
  reason: 'socket hang up',
  source: 'facebook',
  accountStatus: 'offline',
  errorType: 'facebook_unreachable' }
reason Poll error with inbox: 2, company: front
Fri Oct 09 2015 19:13:50 GMT-0700 (PDT) #info# facebook-pump - Account status set to offline (errorType: facebook_unreachable) (inbox 2).
Fri Oct 09 2015 19:14:04 GMT-0700 (PDT) #critical# domain-pump - error performing inbox (2) action bump_last_sync
message connect ETIMEDOUT
stack Error: connect ETIMEDOUT
    at errnoException (net.js:905:11)
    at afterConnect (net.js:896:19)
---------------------------------------------
    at Readable.on (_stream_readable.js:708:33)
    at http.js:1754:12
    at process._tickDomainCallback (node.js:492:13)
---------------------------------------------
    at ClientRequest.onSocket (http.js:1731:11)
    at new ClientRequest (http.js:1433:10)
    at exports.request (https.js:123:10)
    at Needle.send_request (/Users/laurent/dev/front/node_modules/needle/lib/needle.js:378:26)
    at Needle.start (/Users/laurent/dev/front/node_modules/needle/lib/needle.js:291:15)
    at exports.(anonymous function) (/Users/laurent/dev/front/node_modules/needle/lib/needle.js:587:61)
    at exports.listMessagesSince (/Users/laurent/dev/front/server/connectors/facebook/facebook.js:134:11)
    at FacebookPump._poll (/Users/laurent/dev/front/server/pumps/facebook/facebook_pump.js:75:12)
    at listOnTimeout (timers.js:121:15)
---------------------------------------------
    at /Users/laurent/dev/front/server/pumps/facebook/facebook_pump.js:63:27

Hmm, that's odd.  Once the request is done the listener is removed on this line.
If indeed the request object is able to emit more than one error then I we should definitely remove or change that behaviour. However I do need to be able to reproduce the error, so it would greatly help to see an example bit of code that provokes it. :)

Sorry, but it's not that easy to isolate this part of the code. It's a very large codebase and the issue only happens ~once a day. Basically, my code polls a Facebook endpoint every 10s.
I've added more logs to track exactly which request is emitting the error.
I had a crash today that looks like this. It shows that request 125 is emitting an error after you remove the error handler.
remove error handler:  124
{ [Error: socket hang up] code: 'ECONNRESET' }
Mon Oct 12 2015 13:44:35 GMT-0700 (PDT) #error# facebook - Poll error with inbox: 2, company: front
data { status: 'internal_error',
  reason: 'socket hang up',
  source: 'facebook',
  accountStatus: 'offline',
  errorType: 'facebook_unreachable' }
reason Poll error with inbox: 2, company: front
Mon Oct 12 2015 13:44:35 GMT-0700 (PDT) #info# facebook-pump - Account status set to offline (errorType: facebook_unreachable) (inbox 2).
set error handler 125
needle timeout 125
had_error socket hang up 125
remove error handler:  125
{ [Error: socket hang up] code: 'ECONNRESET' }
Mon Oct 12 2015 13:45:00 GMT-0700 (PDT) #error# facebook - Poll error with inbox: 2, company: front
data { status: 'internal_error',
  reason: 'socket hang up',
  source: 'facebook',
  accountStatus: 'offline',
  errorType: 'facebook_unreachable' }
reason Poll error with inbox: 2, company: front
spurious error from 125
message connect EHOSTUNREACH
stack Error: connect EHOSTUNREACH

Here's the relevant part in your code:
var requestCount = 0;

Needle.prototype.send_request = function(count, method, uri, config, post_data, out, callback) {

  var timer,
      returned     = 0,
      self         = this,
      request_opts = this.get_request_opts(method, uri, config),
      protocol     = request_opts.protocol == 'https:' ? https : http;

  var requestId = requestCount++;

  function done(err, resp, body) {
    if (returned++ > 0) return;

    if (timer) clearTimeout(timer);
    console.log('remove error handler: ', requestId);
    request.removeListener('error', had_error);

    request.on('error', function (err) {
      console.log('spurious error from', requestId);
      throw err;
    });

    if (callback)
      callback(err, resp, body);
    else
      out.emit('end', err, resp, body);
  }

  function had_error(err) {
    console.log('had_error', err.message, requestId);
    debug('Request error', err);
    done(err || new Error('Unknown error when making request.'));
  }

To be more accurate: we simply poll the same URL every 10s on a laptop with an unstable connection (limit of range Wi-Fi).

Have you ran your code with NODE_DEBUG=http? That will surely help debug what's going on behind the curtains. This is the relevant section in Node.js's http_client.js -- the line that triggers the timeout error.
I've been adding a few tests under errors_spec.js, trying to replicate the behaviour but so far no luck. Are you triggering requests using setInterval or setTimeout after each one finishes?

I'm now thinking that we might be better by removing the line that removes the error handler. I just did a test trying to replicate #126 and I'm getting a EPIPE error, even though the request has already finished and the node process is not really piping to anywhere.

Ok I think I have it. Take a look at the errors branch.

OK, I'm running the new branch. I'll tell you tomorrow if the crash still occurs.

I think I still have the crash even in the errors branch. I've restarted everything with debug activated.

Any news about this?

Ok, closing. I think the recent changes to needle.js should take care of this.


✏ Typos on v10.10 release notes

Minor Typos on Release Notes
The v10.10 release notes are missing the var keyword in two places in the 2nd bullet point which shows the new @Persisted attribute syntax. Current text:

No more overriding class methods like primaryKey(), indexedProperties() or ignoredProperties(). The primary key and indexed flags are set directly in the property declaration with @Persisted(primaryKey: true) _id: ObjectId or @Persisted(indexed: true) indexedProperty: Int. If any @Persisted properties are present, all other properties are implicitly ignored.

There should be a var after the @Persisted attribute and before the property name. Correct examples below:

@Persisted(primaryKey: true) var _id: ObjectId
@Persisted(indexed: true) var indexedProperty: Int

Goals

Copy and paste the example syntax into Xcode to build my Swift model.
Expected Results

Pasting @Persisted(primaryKey: true) _id: ObjectId into a Swift class source file will not generate an error.
Actual Results

Xcode generates a compiler error: "Expected 'var' keyword in property declaration" with a fix-it of "Insert 'var '"
Steps for others to Reproduce

Code Sample

import RealmSwift
class Project: Object, ObjectKeyIdentifiable {
    @Persisted(primaryKey: true) _id: ObjectId
    @Persisted(indexed: true) indexedProperty: Int
}

Version of Realm and Tooling

Realm framework version: v10.10 to v10.12
Xcode version: 12.5.1 or 13 beta 4
iOS/OSX version: iOS version 14.7 or 15 beta 4
Dependency manager + version: Swift Package Manager 5.4-5.5

Hi @phatblat . Thank you for reporting the issue. I'll update the example.


Win10 Key Grabber

From SFA Hits

Don't know how to do that but great idea

<3 paid checkers have that

Concept: on a FA account it checks if the Windows 10 ediditon is reddeemed and if it isn't it gives the code

and this offer ended in 21 april 2020 rip now peole selling these codes for 10 $ each xD


[OpenAPI Tags]

Would be nice to be able to add tags.
The Rust Docs are hard to read for Macros / Don't show anything.
For example, everything is 'default' right now.

You can add tags, salvo supported it.
See this example: https://github.com/salvo-rs/salvo/blob/main/examples/oapi-todos/src/main.rs

Thanks for the super quick response.
It doesn't seem to work, I have
#[endpoint(tags("test"))]
pub async fn test_input_data(
    res: &mut Response,
    some_request: JsonBody<TestInput>,
) -> ApiResponse<ApiResponseData> {
    // set_ok_data(res, some_request.into_inner());
    ApiResponse {
        status: "status".to_string(),
        message: "message".to_string(),
        data: ApiResponseData {
            some_field: "some value".to_string(),
        },
    }
}

But it still says "default"

I have tested exampel-opi-todos, it works.

@chrislearn thanks, indeed I copy pasted the endpoint macro definition and it doesn't show.
I'm using merge router, so perhaps that might break it? I don't know; but it still says 'default' 🙈

@chrislearn sorry my bad. Rust wasn't reporting that it was a duplicate function (not being used)


Services Gateway Issue

Important notices
Our forum is located at https://forum.opnsense.org , please consider joining discussions there in stead of using GitHub for these matters.
Before you ask a new question, we ask you kindly to acknowledge the following:
[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[X] I have searched the existing issues and I'm convinced that mine is new.
I'm trying to delete some dpinger services manually.  It seems that if you go to your gateways > New > create a new gateway, it won't show the gateway in the gateways but it still makes a dpinger for it.  What is the best way to do this?

can you post a screenshot of your gateways (without addresses)? It might be a bug, since dpinger reads the config directly there might be items that are not valid anymore which we now hide (since 19.7). If that's the case and you're not in a huge hurry, I'll try to look into it in the following days.
The work around is to remove the item(s) from your config.xml manually.

Please see attached.  I believe I have all of these settings correct for Verizon to be the failover too.
From: Ad Schellevis<EMAIL_ADDRESS>Sent: Saturday, August 3, 2019 3:26 AM
To: opnsense/core<EMAIL_ADDRESS>Cc: cmb991<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [opnsense/core] Services Gateway Issue (#3625)
can you post a screenshot of your gateways (without addresses)? It might be a bug, since dpinger reads the config directly there might be items that are not valid anymore which we now hide (since 19.7). If that's the case and you're not in a huge hurry, I'll try to look into it in the following days.
The work around is to remove the item(s) from your config.xml manually.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/opnsense/core/issues/3625?email_source=notifications&email_token=AHL7ELWZIMQGI4SKVRJBMFTQCUXKBA5CNFSM4IJCF6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PJBIQ#issuecomment-517902498 , or mute the thread https://github.com/notifications/unsubscribe-auth/AHL7ELQT6NRTH3WPJ7DBLF3QCUXKBANCNFSM4IJCF6OA .  https://github.com/notifications/beacon/AHL7ELSTNMVWBX6VPBDXIU3QCUXKBA5CNFSM4IJCF6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PJBIQ.gif

@cmb991 it looks like your attachments lost,  can you try pasting them online?

Sorry.  Here are the pictures.  The setup is Xfinity as the primary connection and Verizon as the failover.  Although, I’m noticing Verizon is taking the primary spot when I plug it in, so I’m not sure what that is about.

@cmb991 https://github.com/opnsense/core/commit/93bbe1ee168bbe0297746c2a2f68a5a4a4cd0972 should fix the visibility issue. It's probably best to wait for 19.7.2 tomorrow and apply this patch on top (there are some other changes stashed in the same files).

I missed a spot (https://github.com/opnsense/core/commit/74ff64787cf78654595231a359b47cad6a88511e) and found another small presentation issue while working on this item https://github.com/opnsense/core/commit/bb6833d5013facc058a70376362ad0b45409ff03.
To install all changes on OPNsense 19.7.2:
opnsense-patch 93bbe1e 74ff647 bb6833d

Everything seems to be working now, I just had a question about this setup now; the firewall rules, I only have the one LAN rule in a gateway group called failover.  Does that local DNS route firewall rule that the guide says to put in need to have the failover gateway or no since its locally doing the DNS reroute?  Screenshot below

@cmb991 thanks for confirming, let's close the issue then.
The port 53 rule prevents dns traffic being send to the gateway, which is needed if you are using a local forwarder (unbound for example)

opnsense-patch 93bbe1e 74ff647 bb6833d

works! The Gateways are now displayed. But i've found a litte Problem with the Checkbox in front of the entries. If i mark all entries and klick on the trash. Some will be present. I have to purge every single item in the list.

@skl283 I haven't seen the select all issue yet, but I'll try it out on a box over here.

No Problem.Had just issue while delete all "testing"entries before applying the patches ;-)

I had a similar issue since upgrading to 19.7.2. One issue remains though: after applying your patches I cannot modify the automatically generated gateway of a DHCP WAN connection! See attached screenshot. How can I set upstream gateway or priority now? Or do I need a gateway group by default if one WAN connection is DHCP based?

@alexanderharm the buttons aren't shown on the right? can you resize your screen a bit? when it's a bit smaller it should resize to xs and show the buttons again.  You should also be able to scroll to the right (the buttons should be there, there just don't seem to be enough space there).

Thanks, resizing the screen did the job.

we might use "table-condensed" to strip some of the margins, same as we did recently on the firewall pages.

@alexanderharm https://github.com/opnsense/core/commit/1aad01cc92c42f3a24ea61f231c3692b6f07dd3c should be better. (although it can still break out in some situations)


Closing CVE-2023-28155, CVE-2023-26136

Hey, I just made a Pull Request!
This pull request updates the @kubernetes/client-node dependency from version 0.20.0 to 1.0.0-rc4. This update is critical as it addresses two known security vulnerabilities related to the request package, a dependency of @kubernetes/client-node, and its transitive dependency tough-cookie:

CVE-2023-28155 - A vulnerability in the request package, which could potentially allow attackers to perform various attacks due to improper input validation.

CVE-2023-26136 - A vulnerability in the tough-cookie package, a transitive dependency of request, which could allow attackers to craft cookies that may bypass intended security restrictions, leading to potential information disclosure or session hijacking.

By updating to @kubernetes/client-node version 1.0.0-rc4, these vulnerabilities are mitigated through the inclusion of patched versions of the affected packages. It is highly recommended to merge this pull request at the earliest to ensure the security of our application and protect against potential exploits leveraging these vulnerabilities.
Changes:

Updated @kubernetes/client-node from 0.20.0 to 1.0.0-rc4 in package.json.

Impact:

Addresses critical security vulnerabilities CVE-2023-28155 and CVE-2023-26136 related to the request and tough-cookie packages.
Enhances the security and stability of the application by using the latest, patched versions of dependencies.

Please review the changes and merge this pull request to enhance our application's security posture.
Thank you.
Contribution is Signed-off-by: Daniel Meyer (external expert on behalf of DB InfraGO<EMAIL_ADDRESS>:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[x] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Any update on this? It looks like it was almost over the line!

I'm currently not working on this project anymore so any help to get it over the line would be appreciated.

Can't let this auto-close

Still needs fixing

How much longer will this take to fix?

It's seemingly been abandoned. Anyone is free to make a replacement.

I have opened https://github.com/backstage/backstage/pull/25385 to supersede this pull request, fix the conflicts and push it to resolution.

This shouldn't have been allowed to close until https://github.com/backstage/backstage/pull/25385 gets merged

The above branch is still awaiting a merge


While assigning devices to hierarchy, clicking on device name breaks the flow

Try to create a hierarchy. While adding assets, try to assign device and click on any device name. You will notice that the whole flow breaks as soon as you click on the device name and there is no way to go back
One workaround could be clicking on device name should open the device page in next tab.

It is as per the Cumulocity design followed while creating the new groups through Cockpit. While assigning to the devices, if user wants to know more about the device, then clicking on the device name, a message is displayed with a link and on clicking the link, page navigates to Device Management application in new tab.
If user wants to navigate back to the default assets page, then click on the assets in the left navigation pane else click on go back (<-), to navigate to create assets page. In case if user wants to assign the devices, then he/she has to click on the checkbox and click on assign button.
While this is an existing behavior since the initial design, this can be considered as an enhancement for better user experience and same will be notified to the Cumulocity core team


Issues with sse_channel

When using Server Sent Events transport there are at least 2 issues caused by underlying implementation of sse_channel

When access token is used it is getting overriden along with any other query parameters causing 401 error. On signalr_client side this error is not handled properly and nothing happens until handshake timeout is triggered in HubConnection.
There are no stop mechanism in sse_channel so when .stop() method is called on instance of HubConnection the underlying sse connection continues to exist and receiving server sent events leading to unwanted behavior.

If anyone encounters same issues I've made a workaround for them in here.
@sefidgaran I'd be glad to help resolving these if you can suggest a solution on how to deal with sse_channel. The package itself seems to be abandoned so I'm not sure if opening PR there would work.

Save problem

@milten21
Please feel free to open a PR.
Thanks


Add journal component

Add journal component, which is a grouping of properties for an associated journal entry.  Adds test coverage for journal component.

Codecov Report

Merging #56 (a2af8e4) into main (6322059) will increase coverage by 0.24%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #56      +/-   ##
==========================================
+ Coverage   95.98%   96.23%   +0.24%
==========================================
  Files          13       14       +1
  Lines        1095     1141      +46
==========================================
+ Hits         1051     1098      +47
+ Misses         44       43       -1

Impacted Files
Coverage Δ

ical/event.py
95.58% <ø> (+0.51%)
:arrow_up:

ical/todo.py
97.22% <ø> (ø)

ical/calendar.py
100.00% <100.00%> (ø)

ical/journal.py
100.00% <100.00%> (ø)

ical/types.py
95.78% <100.00%> (+0.03%)
:arrow_up:

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


Not integrate well with asyncrun.vim

That's because the file entry in the quickfix list is empty

Well, I think that's not the key point.

Should be fixed now. Let me know if you have any more issues

This works and don't yield errors, but I still can't see the quickfix messages. Is this because the other side(asyncrun.vim) is doing some dirty hacks?

This works and don't yield errors, but I still can't see the quickfix messages. Is this because the other side(asyncrun.vim) is doing some dirty hacks?


Require Django >= 2.1 and Python >= 3.6

Dropped support for older versions.
Went back to tox-travis.
Prepare release 12.0.0

I'll check for Python 2 specific code such as super calls.


Add CI for preview deployment

This PR fixes #56.
I will rebase and undraft this PR after #65 is merged.

Ah, it would be better to create the PR not from a fork.


PropertyChanged not working

Hello, I used your project (the navigation) as a template for me and changed a few things up. Now I have the problem that ImplementPropertyChanged (Nuget: PropertyChanged.Fody) is not working correctly. Is there a known problem with this Nuget and the MasterDetailPage?
Regards,
Jan

I have never used Fody sorry.

Yeah, by mistake I deleted : INotifyPropertyChanged. Adding it again and delete Fody was the solution for my problem. Great Project and Template. @jamesmontemagno


Warning BaseWithApp

I am receiving this error after upgrading to the latest heroku. It is killing my deployment scripts.
/Library/Ruby/Gems/1.8/gems/heroku-2.16.1/lib/heroku/command/base.rb:209: warning: already initialized constant BaseWithApp
!     !    Resource not found
/Library/Ruby/Gems/1.8/gems/heroku-2.16.1/lib/heroku/command/base.rb:209: warning: already initialized constant BaseWithApp

After downgrading to 2.15.3 everything worked again.

@sjonnet19 Hmm. I don't believe that the warning should actually be causing the problem in and of itself. There may be some other issues that were introduced during the cleanup.  Could you  give me some context around what you are trying to do and what command is actually failing so I can try to get it fixed for you?  Thanks!

I'm attempt with my scripts to destroy apps and/or create apps then add add ons and start a worker.
I think the warning does f with my script though. I am removing my err checks to see if that fixes it.

Any reason for the warnings?
By removing my catches it fixed it...

No good reason for the warning no, I just made a mistake in how I did it.  The file must be getting required/loaded more than once some how.  In any event, I just pushed a commit that should prevent it here:
https://github.com/heroku/heroku/commit/3e133703df54bd22f55942bbd3681022bb888170
Sorry for the confusion and difficulty, glad you found a quick workaround.

Thank you for the help and quick turn around as well.


Allow kube-proxy to run regardless of any node taint

What this PR does / why we need it:
at the moment a user can taint a node and inadvertently prevent kube-proxy from running on that node. this allows kube-proxy to run on any node of the cluster regardless of the taint.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
If applicable:

[ ] documentation
[ ] unit tests
[ ] tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

@lachie83 can I get a 👍 from you on this? This applies tolerations to kube-proxy so it'll run anywhere, or at least that is the intention. Grabbed from the official daemonset.

/lgtm


Sys - Upgrade gallery_saver

Unfortunately, when installing visual-editor, it might run into versioning issues because it's dependent on gallery_saver, which in turn depends on an outdated version of http 0.13.0. This creates issues where we can't upgrade google_fonts because of this. Additionally, if I try to install this package, I get the following error:
Because gallery_saver 2.3.2 depends on http ^0.13.3 and no versions of gallery_saver match >2.3.2 <3.0.0, gallery_saver ^2.3.2 requires http ^0.13.3.
So, because my_app depends on both gallery_saver ^2.3.2 and http ^1.0.0, version solving failed.
exit code 1

Because gallery_saver is not maintained anymore, it makes sense to replace it with gal, which is maintained and serves the same purpose.
After (hopefully) merging #237 , this will be the only last "outdated" dependency, which will make it possible to install visual-studio without a hinch (fingers crossed! 🤞).
I'll open a PR on this.

I'll have a look at these libs tomorrow evening. I've had some major overflow at the day job. Won't keep you hanging too much. I'd do it today but my brain is zapped.


[Bug] The copied Modal is not functional

Steps to reproduce:

Add a Modal
Click on the Property pane
Copy the modal
Now try to open the modal via Entity explorer
and observe
Now connect the Copy modal to a button
and observe

Observation:
it is observed that by clicking on the modal via Entity explorer the modal does not appear.
Clicking on the connected button to "On click" it is observed that the modal does not appear
Expectation:
It is expected that the modal must open
Video:
https://www.loom.com/share/d05428bdb76745ebbe8ebb3943d719e3

Closing this in favour of #5128


Option to Download Table

Hello,
The new addition to download tables as csv is great. However, can we make this optional and not a default?

upgrade  to 0.4.12. For backward compatability, allowd_download will true by default


exercism: 3.3.0 -> 3.4.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for exercism is: A Go based command line tool for exercism.io
meta.homepage for exercism is: https://github.com/exercism/cli
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 3.4.0 with grep in /nix/store/ggdkk21nwdn8lhqhsj2c34xh5qcj1gfd-exercism-3.4.0
found 3.4.0 in filename of file in /nix/store/ggdkk21nwdn8lhqhsj2c34xh5qcj1gfd-exercism-3.4.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

exercism

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/ggdkk21nwdn8lhqhsj2c34xh5qcj1gfd-exercism-3.4.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A exercism https://github.com/r-ryantm/nixpkgs/archive/135a2bee38ab6693af6be0b30a8e0a6fdbc4efdb.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/135a2bee38ab6693af6be0b30a8e0a6fdbc4efdb#exercism

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/ggdkk21nwdn8lhqhsj2c34xh5qcj1gfd-exercism-3.4.0
ls -la /nix/store/ggdkk21nwdn8lhqhsj2c34xh5qcj1gfd-exercism-3.4.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

exercism

Maintainer pings
cc @rbasso @NobbZ for testing.


1 (quasi)issue + a few Q's

Hi :grin: THANK YOU for copyparty 🎉 it's really great!
First, the issue. This very probably is not your issue, which is precisely why I didn't create a Bug Report to bring it to your attention and elected to go with "Something Else," but about 2 weeks or so ago, completely out of nowhere, copyparty "broke." I run a script at boot which invokes copyparty so there's very little handling on my part once arguments have been settled on, so I'm not really sure what I did but I would not be the only instance of someone mentioning this if it were on your end so I'm pretty confident it was something that either me or my machine did.
Anyway, from this point onward (and still to this day), any copyparty commands are met with:
Traceback (most recent call last):
   File "/usr/bin/copyparty", line 5, in <module>
     from copyparty.__main__ import main
ModuleNotFoundError: No module named 'copyparty'

so the binary is still present. I alluded to this only being a quasi-issue because I was able to install a second (functioning, thankfully) instance of copyparty via pip (or rather, pipx). I honestly don't recall how or why I ended up doing this but assume it must've been the shortest path to a working copyparty server (fully functioning, anyway. I know of the emergency backups and nearly had to go that route). The strange thing is that when looking at the binary file which runs without a hitch, located at /~/.local/bin/ and comparing it with the one that fails, located at /usr/bin/, they're the same, verbatim. The only difference is that the former executes as expected and the latter fails with the above output Re: missing module. I wish I knew enough about Linux binaries to provide more info but that's about all I've got on hand. However, I'm happy to provide anything that might be helpful, as needed. Anyway, as mentioned I'm up and running again but I don't love having a broken instance installed or needing to include /home/user_name_here/.local/bin before my copyparty commands, even if it is the case that they're issued via script 9x out of 10.
Any clue what I might've done and/or more importantly, what I can do to right the ship and get the party going again?
Now the couple of questions, if I may. They should be pretty easy compared to the mess above. First, is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior? I poured over the README and supplemental documentation and couldn't find any plugins or anything that appeared to extend video codec support. I guess I just assumed that copyparty would support TS files if the browser in use did but that doesn't seem to be the case.
Thing 2: I know I saw this in the documentation somewhere, but I can't for the life of me find it now so please forgive me for asking but how do I essentially give myself "A" permissions and anonymous users no perms, like not even read? Would I need to resort to some of the trickery you mention, like nesting within a hidden folder or some such?
Last thing: I run copyparty with both -e2dsa and -e2ts (this isn't the question, but is that necessary or redundant? I couldn't tell from the documentation, but it seems the latter is for metadata only so I think it's :ok_hand: also, iirc I only did this as a means to try to account for the issue I'm about to explain) but when I attempt to use the search function, nothing is ever returned. On a few occasions, I've noticed upon clicking the :mag_right:, suddenly all of my many files disappear and only a handful of files are left, which also disappear as soon as I enter my query (assuming the query doesn't match with one of the 5 or so files, which it wouldn't because they're right there )  , which leads me to believe that something is off with indexing. Possibly relevant, in the early days before I added a crap ton of arguments (I've copy-pasta'd the command in its entirety below, in case a poorly chosen/conflicting argument, etc. could be causing any of my troubles), there was A LOT of indexing going on, which always prevented uploads even hours and iirc, possibly even after a full day or more. So, since every time I actually looked at what was being indexed it seemed to be indexing dot files, none of which I need to be indexed, I appended the no index argument, like so --no-idx \..+
This seemed to do the job but maybe too well, as now I see no indexing happening. Did I eff up? I don't recall if I was able to search prior to adding that argument, though I suppose I could easily find out but as I'm already this far and nearly finished, I'm going to hit submit rather than wait to test it out only to inevitably get distracted and fail to finish this monstrosity of an issue.
/home/user_name_here/.local/bin/copyparty -e2dsa -e2ts --no-robots -a user_name_here:pw -v .::r:A,user_name_here --no-idx ..+ --dbd=wal -i <IP_ADDRESS> -p 3923 --no-dupe --xlink --db-act 30 --no-athumb --no-acode --nos-dup --grid --favico 💾 --theme 6
(I know I should probably go the config file route...)
PS - sorry for my long-windedness, but really - thanks again for diligently maintaining the project!

Hey, and thanks for using my software! Sounds like giving it a proper workout too :grin:
And awesome work digging through that issue -- welcome to the world of fun that is python packaging :sweat_smile: :+1:
When you install a python package with pip install --user, it'll install to ~/.local like you've already seen, but it doesn't plonk everything into one single place -- instead it kinda sprays all over in there. You get the binaries in ~/.local/bin/, maybe some docs in ~/.local/share, and then the actual python code goes in ~/.local/lib/pythonX.Y/site-packages/copyparty. That's how it works for most python software in general too
And here's the kicker; this sounds a lot like the kind of thing that would happen if you install the package locally, and then later something like this happens:

python is upgraded, from say 3.11 to 3.12 or such, so it goes looking in ~/.local/lib/python3.12/ instead of ~/.local/lib/python3.11/
the packages in ~/.local/lib/pythonX.Y/site-packages are deleted

As for why the launcher binary that's in /usr/bin works while the one in ~/.local/bin doesn't, I would have guessed that the very first line, the #! one would be different. That line (called a shebang) tells Linux what interpreter to run the script with, and if one of them says /usr/bin/python3 and the other is like /usr/bin/python3.11 then that's it. But if those are the same too, then we'll have to keep digging :>
one thing to note is that the launcher binary is pretty much just a fancy alternative to python3 -m copyparty, or more specifically /usr/bin/python3 -m copyparty if that's what the shebang says. So that's another way to check.
but I'm curious if the following command would fix it? and if nothing else, it might throw us a hint in the output: python3 -m pip install --user -U copyparty

as for the questions,

is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior?

if they do appear as text in your browser, then my first hunch is that either copyparty is sending the wrong mimetype to the browser, or the browser doesn't understand it... But maybe I'm missing some details here; is this something that happens if you click the filename to download it, or are you clicking the txt link next to the filename, or maybe something else? Oh and, I'm guessing your copyparty is running on a Linux distro, but what browser/OS are you accessing it from? At any rate, we can at least verify the mimetype suspicion with curl: curl -vo /dev/null http://<IP_ADDRESS>:3923/some.ts should mention Content-Type: video/mp2t near the end, if that's working as intended.

how do I essentially give myself "A" permissions and anonymous users no perms, like not even read?

that comes down to this part of your command, -v .::r:A,user_name_here -- to make it slightly less confusing, this is exactly the same as -v ./:/:r:A,user_name_here, or in other words, share the folder you're launching copyparty from (./) at the server root (/), with read-access for everyone (since there's no usernames before the next :), and A for yourself. So you'd get what you want with -v .::A,user_name_here instead :>
but just to mention what that could look like in a config file instead,
[accounts]
  user_name_here: the_pw

[/]
  /home/dave/Music
  accs:
    A: user_name_here

both -e2dsa and -e2ts (this isn't the question, but is that necessary or redundant?

almost, but not quite -- -e2ts does enable -e2ds, but the a in -e2dsa makes it scan through absolutely all volumes, and not just the ones where someone has write-access.

there was A LOT of indexing going on, which always prevented uploads even hours and iirc, possibly even after a full day or more

yeah, this is kind of a bummer... but I haven't found a solid way to avoid it yet. The biggest practical issue is that it would mess with the deduplication if someone's allowed to upload a dupe which just hasn't been discovered locally yet, but it'd also require some proper cleanup in the indexing/database code before that would be safe to even consider! so maybe one day hehe

I appended the no index argument, like so --no-idx \..+

yup, there we go -- that's gonna skip any file that has a file extension :grin: this one should work better: --no-idx '/\.' to catch any files with a dot immediately following a directory separator (and note the single-apostrophes to prevent the shell from eating the backslash, just in case)
that's all I can think of though, your setup looks alright aside from that :+1: :+1:

PS - sorry for my long-windedness

not at all! thx for the effort, happy to help :>

okay, think I have an idea how to avoid the indexing time if you're alright with possibly accepting duplicate uploads while it's indexing...
if you change the -e2dsa -e2ts to just -e2d -e2t (to keep the database enabled but without filesystem scanning) and start it like normal, but then, while that is up and running, you start a second instance of copyparty with the exact same arguments plus these extra ones: -e2dsa -e2ts -p 0 --ign-ebind-all --exit=idx
the second copyparty doesn't accept any connections, and then runs through the indexing before it exits on its own once it's finished. After that you can change back the main copyparty to -e2dsa e2ts like before so it'll catch any new files added to the filesystem the next time you start it.
I think this should be safe, since the actual database backend (SQLite) is real solid stuff and should prevent any crosstalk between the copyparties. Let me know how it goes if you decide to give this a try! :pray:

Wow, that might well be a new record for timeliest dev reply, at least as far as personal experience goes. So thank you for that but more so for the comprehensive and thoughtful response therein!
I generally respond to text-based communication in a linear fashion, but I'm sorry... I can't help but still be laughing at myself for effectively instructing copyparty not to index anything with a file extension, so essentially everything and anything, yet still have the audacity to wonder why searches we're coming back empty-handed 😂 🤣 Though, it does make me wonder about the 5 or so files that regularly did turn up prior to executing the search but after clicking on the 🔎 ¯\_(ツ)_/¯
Either way, thank you! Despite being what I closed out the issue with this was the one that cropped up most often and was the most bothersome of the issues I mentioned by a substantial margin, including the quasi-issue itself.
Speaking of, you were exactly right regarding your suspicion about differing shebangs. I guess I've grown accustomed to looking at everything following but not the shebang itself. Sorry for the oversight. Also I'm not sure this is of any consequence but it's actually the other way around, the binary at ~/user_name_here/.local/bin/ is the working one and the one at /use/bin/ fails with the error I'd given. Further, because I use an Arch-based distro and therefore pacman, I get the externally managed environment error when trying to use pip without venv and to my knowledge, you can't use the --user -U portion of python3 -m pip install --user -U copyparty with venv. Or at least I wasn't able to and am not well versed enough w/ pip to know otherwise. That said, the command without that chunk did work to install or reinstall copyparty and I thought everything was going to be golden as far as the quasi issue was concerned, but unfortunately that particular error regarding the module being missing persists. So at least for the time being I'll have to make do with the ultimate installation which is not much of an inconvenience at all. I appreciate your input in an open to any further suggestions you might have in this regard but don't feel obligated by any means. Also thank you for the refresher course on Python packaging, I genuinely needed it.

And here's the kicker; this sounds a lot like the kind of thing that would happen if you install the package locally, and then later something like one of these things happens:

python is upgraded, from say 3.11 to 3.12 or such, so it goes looking in ~/.local/lib/python3.12/ instead of ~/.local/lib/python3.11/
or maybe the packages in ~/.local/lib/pythonX.Y/site-packages are deleted

Are you wondering aloud if one of these might be what's happened in my case or just commenting on the somewhat disjointed nature of python packaging? Or perhaps "door №.3?"

as for the questions,

is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior?

if they do appear as text in your browser, then my first hunch is that either copyparty is sending the wrong mimetype to the browser, or the browser doesn't understand it... But maybe I'm missing some details here;

is this something that happens if you click the filename to download it, or are you clicking the txt link next to the filename, or maybe something else?

So, if I click on the thumbnail in grid view, I get e.g., "this file is 355.4 MiB large -- really view as text?" However, if I click on the file name in list view then it tries to open the video straight away but the video fails to open and has the "broken media"  symbol within the player. Interestingly, when I click a TS file located among search results (because search is working, yay!!), depending on the browser it either prompts a download or automatically starts it.

Oh and, I'm guessing your copyparty is running on a Linux distro, but what browser/OS are you accessing it from?

You guessed correctly! it's an Arch-based distro called Garuda Linux. However, the browser(s) I've most often experienced this TS files issue with is on my Android tablet and phone on which it occurs in any browser, but I just checked on my Linux machine and the same behavior is seen in any browser there as well. Or at least the few that I tried, I assume it will persist across any others.

At any rate, we can at least verify the mimetype suspicion with curl: curl -vo /dev/null http://<IP_ADDRESS>:3923/some.ts should mention Content-Type: video/mp2t near the end, if that's working as intended.

Ding ding ding, you nailed it again! You're pretty good at this 😂
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Content-Length: 62
< Connection: Keep-Alive
< Date: Tue, 07 May 2024 10:03:03 GMT
< Vary: Origin, PW, Cookie
< Cache-Control: no-store, max-age=0
< X-Robots-Tag: noindex, nofollow
< ⭐Content-Type: text/html; charset=utf-8⭐
<
{ [62 bytes data]
100    62  100    62    0     0  58106      0 --:--:-- --:--:-- --:--:-- 62000
* Connection #0 to host <IP_ADDRESS> left intact

both -e2dsa and -e2ts (this isn't the question, but is that necessary or redundant?

almost, but not quite -- -e2ts does enable -e2ds, but the a in -e2dsa makes it scan through absolutely all volumes, and not just the ones where someone has write-access.

I'm thinking that perhaps this is a bit extreme for my needs but I'll continue to mull it over for a bit before I make any changes.

there was A LOT of indexing going on, which always prevented uploads even hours and iirc, possibly even after a full day or more

yeah, this is kind of a bummer... but I haven't found a solid way to avoid it yet. The biggest practical issue is that it would mess with the deduplication if someone's allowed to upload a dupe which just hasn't been discovered locally yet, but it'd also require some proper cleanup in the indexing/database code before that would be safe to even consider! so maybe one day hehe

well, it's still indexing at the moment but it also hasn't got a chance to index anything in weeks or months even so we'll see if/when that chills out and take it from there but fingers crossed for one day, if for no other reason then to make your life easier, or the lives of your users, which for a developer like yourself, I would imagine is effectively the same thing.

I appended the no index argument, like so --no-idx \..+

yup, there we go -- that's gonna skip any file that has a file extension 😁 this one should work better: --no-idx '/\.' to catch any files with a dot immediately following a directory separator (and note the single-apostrophes to prevent the shell from eating the backslash, just in case)

Still laughing at this 😆😅😂🤣🤡

that's all I can think of though, your setup looks alright aside from that 👍 👍

Now if I could just figure out that pesky TS file issue. That one really has me stumped especially now that I find it's the browsers which seem to be the issue despite them supporting the file format or do I have what curl returned backwards?
Regardless, words really can't express how grateful I am for both your feedback to my low priority issues and the promptness of it and generally for your dedication to the software upkeep/maintenance. I'm sure you get it fairly often because it's very abundantly clear but you really are an exemplary developer/programmer and the software shows it!
I'll give you an opportunity to respond but it's pretty safe to say that this could be closed out whenever you think appropriate or upon my next response whichever comes sooner. The TS file thing is fairly low priority.

okay, think I have an idea how to avoid having to wait for indexing, if you're alright with possibly accepting duplicate uploads while it's indexing...
if you change the -e2dsa -e2ts to just -e2d -e2t (to keep the database enabled but without filesystem scanning) and start it like normal, but then, while that is up and running, you start a second instance of copyparty with the exact same arguments plus these extra ones at the end: -e2dsa -e2ts -p 0 --ign-ebind-all --exit=idx
the second copyparty doesn't accept any connections, and then runs through the indexing before it exits on its own once it's finished. After that you can change back the main copyparty to -e2dsa -e2ts like before so it'll catch any new files added to the filesystem the next time you start it.
I think this should be safe, since the actual database backend (SQLite) is real solid stuff and should prevent any unintended competition between the copyparties.  I was sitting here rereading my response and editing as needed and looked up and you had responded again. Words fail me other than to say thank you. Although I have to ask, I'm sure you wouldn't suggest it if it didn't work but in the past when I've tried to run a second instance of copy party I always get:
Exception: port 3923 is busy on interface blah.di.blah.1
Or, by

the second copyparty doesn't accept any connections, and then runs through the indexing before it exits on its own once it's finished.

Are you essentially saying that it will do just that but not before performing whatever index it needs to perform according to the command plus arguments?
Still assuming you are correct which I think is a pretty safe assumption This is a perfectly workable solution as 99.9% of the time I'm going to be the only one accessing my files from the copyparty server and I should be able to manage not duplicating uploads, although I've screwed up simpler things lol
Seriously man, thank you!

Are you wondering aloud if one of these might be what's happened in my case

mostly this, but I assumed it was the /usr/bin launcher that was working and the ~/.local one that didn't, so ignore that part :p
though since you mentioned you're on an arch-based distro, are you also using the aur package to install copyparty?

to my knowledge, you can't use the --user -U portion of python3 -m pip install --user -U copyparty with venv.

ah yep that's right, while in a venv the correct thing would be just pip install -U copyparty , so not even the python3 -m

the video fails to open and has the "broken media" symbol within the player

just going by this, it sounds like your browser doesn't support playing .ts files - come to think of it, are there any browsers which do? I'm not seeing it listed on mdn, so my assumption would have been no; https://developer.mozilla.org/en-US/docs/Web/Media/Formats

if I click on the thumbnail in grid view, I get e.g., "this file is 355.4 MiB large -- really view as text?"

heh, yeah that's pretty pointless behavior -- assuming there's no browsers able to play .ts files, it should just start a download instead 👍

edit - is there a way to manually set MIME types for specific file extensions w/ copyparty?

not yet! that would be super useful, so I'll get that added

edit 2 - can you not move files while indexing is ongoing as well as upload?

nope ... :upside_down_face:

when I've tried to run a second instance of copyparty, I always get:"port busy"

yup! and usually that makes sense, because you can't have multiple things listening on the same tcp port, so you'd have to give it a different port with -p, but in this case we don't want the 2nd instance to be able to respond to any requests at all, just handle the indexing for us without getting too much in the way of the 1st instance... so we tell it to listen on port 0 (which makes no sense), and then to ignore the fact that it's not able to listen for connections with --ign-ebind-all

but.... here's the bad news -- it turns out the idea didn't really work :(
the 2nd instance grabs the database so hard that the regular copyparty isn't able to do much of anything at all. So unfortunately you'll still need to wait for indexing to finish before uploads/moves/renames becomes possible. Ahh...
at least it's only the initial indexing that takes forever, so once all the files are in the database it'll be MUCH faster the next time you restart it :>
and thanks for the nice words! best payment really 😁

Well it's a bummer that it doesn't work but at least that gets me off the hook for not finding the time to test it out yesterday 🥸

the video fails to open and has the "broken media" symbol within the player
just going by this, it sounds like your browser doesn't support playing .ts files - come to think of it, are there any browsers which do? I'm not seeing it listed on mdn, so my assumption would have been no; https://developer.mozilla.org/en-US/docs/Web/Media/Formats

If this is the case then I feel like a first-rate jerk for even bothering you with it and now that you mention it I don't know that I've ever expressly been able to play a TS file in browser via the browsers media player but I know that different players hosted on X, Y or Z site is capable of playing them. I only know this because almost every single TS file I have was ripped from one of these sites. Sadly it seems like it's a very popular format these days so while most of my video files are in MP4 format, probably upwards of a good ¼ of them now which were ripped from an online source as opposed to downloaded in a prepared download if that makes sense are in the TS format. I'm sure I can find a SFW example if it would be of any help but it sounds like I might be SOL anyway. It just makes little sense to me that the browser would be able to play it in one context but not in another but that shows you how little I know about the inner workings of web browsers and things of that nature. Anyway I'm going to close this issue out with this comment because you have gone above and beyond to help me with everything that seems helpable but if you would be so kind as to let me know if and when you are able to incorporate some sort of manual override for MIME type, I would be very interested to know about it!
Thanks again for everything, my friend! It's been a pleasure chatting with you and while I don't wish for any issues to crop up with my copyparty server, I can rest easy knowing that if they should it'll be a pleasurable experience dealing with them if it means having to consult with you on the matter. If there's ever anything I can do on my end in terms of testing out any new features or features that you're considering but not sure about implementing in the release or anything of that nature, please don't hesitate to ask! It's the very least I could do!

If there was a way to share these files as videos to VLC or MX Player or whatever video player one might have on the device that would probably be the easiest fix by far but I couldn't figure out how to accomplish this. I can share mp4 files to a local player but sharing a TS file doesn't seem to shake the text/html MIME type

I  can share mp4 files to a local player but sharing a TS file doesn't seem to shake the text/html MIME type

funny you should mention, I just noticed a small detail from earlier -- when we tried to check the mimetype copyparty was returning for ts files, the response was a HTTP 403 Forbidden, so it was getting the login page instead of the ts files... so the mimetype was correct, since the login page is html hehe
I'm guessing the ts files are in a volume that is password-protected, so VLC is also getting that login page instead of the file.
there's two ways to solve that; either you can add your password to the end of the url, like ?pw=hunter2 -- but that's a bit of a hassle, so instead you might want to enable filekeys, which makes the server generate URLs which can be shared to access a file without having to log in.

Something as simple as highlighting/selecting a particular file and having a share button might accomplish it

so I think filekeys comes pretty close to this :>
you can enable filekeys by changing your -v .::A,yourname into this: -v .::A,yourname:g:c,fk -- this gives unauthenticated users get-access, meaning they can download files if they know the correct URL but not see directory contents, and then it locks it down a bit by also enabling fk (filekeys) so just knowing the URL isn't enough, but the url must also contain the server-generated per-file "password" -- so if you copy a link from the web-UI it'll have the correct filekey added to it already.
that said, a completely different workaround is to convert ("remux") the ts files into mp4 or mkv files, and that way you also save some disk space (around 15%) -- ts is very wasteful, as it was designed for unreliable networks, so it keeps repeating redundant data (headers). I think mp4 supports all the important features that ts files might use, and I think more players support mp4 than mkv, so that's probably(?) the better choice.
if you want to convert your ts files to mp4 files without any loss in quality, then something like this should work (copypaste all of it into a terminal in the folder with ts files):
for infile in *.ts;
do outfile="${infile%.*}.mp4";
[ -e "$outfile" ] && echo "outfile exists, skipping $outfile" && continue ;
if ffmpeg -nostdin -i "$infile" -map 0:v -map 0:a -c copy -movflags +faststart "$outfile" ;
then echo "converted OK; deleting $infile ..."; rm -- "$infile";
else echo "ERROR: failed to convert $infile"; break;
fi; done

the -movflags +faststart is only necessary for mp4 files and can be removed if you change it to mkv instead; what it does is move the file header from the end of the file to the start instead, so it doesn't have to do a bunch of seeking on the hdd before it can start playing
NOTE: the command deletes the input files if FFmpeg thinks they converted successfully! So if you have enough disk space and want to actually make sure they did, then remove the rm -- "$infile"; part :>

wait... does your .ts files have the file extension in uppercase? because I just noticed some code that assumes them to be lowercase 😅

Thanks for looking into this and providing all this helpful info! To answer your last question, no - to my knowledge every instance of a .ts file is in lower case, I also overestimated the number of them contained within my library. It's probably closer to 10% than 25%. As such and since I came to the decision prior to reading your responses here, I elected to go the conversion route. Although, I'm never really sure I completely understand the differences between re-encooding and remuxing. Also, I sure hope the answer is no because the process is already underway (but it just started so...), do you see anything wrong with the following?
#!/bin/bash

# Iterate
for file in *.ts; do
    # Get the filename without extension
    filename=$(basename -- "$file")
    filename_no_ext="${filename%.*}"

    # convert
    ffmpeg -i "$file" -c copy "${filename_no_ext}.mp4"
done

Thanks for the other info, too. Despite electing to go the conversion route, it will be handy to know how to proceed  in a manner that doesn't require converting everything every time. Especially considering  this is going to take a lot longer than I bargained for. I was like, "Oh, it's only like  800 some files..." But forgot that many of them are in 4K format and a good chunk of them are feature-length movie. D'oh!

the differences between re-encooding and remuxing.

as long as you have the -c copy in there, you are doing remuxing and not reencoding, which is good! remuxing is fast (barely used any cpu, so is only limited by your HDD speed) and it retains the original quality exactly as it was -- the only difference is the container (the wrapping, so to say). Moving the video and audio streams from a box and onto a shelf or something like that hehe
but reencoding actually converts the video stream itself, which causes a quality loss and uses a LOT of cpu, so it also takes forever - but sometimes it is a necessary evil, if you need to play a file on a device that doesn't understand some of the codecs within. Usually the issue is the audio, which is much faster to reencode than video, but many automated tools reencode both, which is a huge waste...

I sure hope the answer is no because the process is already underway (but it just started so...), do you see anything wrong with the following?

nope, that looks good! I was being overly careful with my suggestion (-map 0:v -map 0:a?), which is more eager to intentionally crash if it hits a file that isn't entirely compatible with converting to mp4, but honestly I can't think of any realistic examples where that could happen. And copyparty is entirely fine with not having the -movflags +faststart in there, but some other web servers might not be! And finally, not having -nostdin is fine because you're doing iteration with a for-loop, instead of something recursive like find -iname '*.ts' | while read filename; do ffmpeg -i "$filename" ...

This is entirely unrelated to any of the issues we discussed, but I probably should have gone with your script after all. My computer has been acting up lately and my external hard drive has recently developed a habit of unmounting and remounting itself once every few hours or so for some reason and obviously when this happens, anything that's using files on it crashes or at the very least is interrupted. This happened while the process of converting .ts files was underway. The script I posted here would have been fine in this case, as it didn't delete anything but because it effectively equates to double the number of files until deleted, before I knew it I had like 30GB left on my 10TB HDD and it was crawling at a snail's pace. So I deleted the unnecessary files and adjusted the script so that it would delete the converted files but only in the case of a successful full conversion. Obviously it didn't catch that last part and when the hard drive unmounted itself I found that the file that it must have been in the process of converting was both incomplete and the original gone. So because now I didn't trust myself to make the necessary adjustments since I had considered this eventuality and thought I planned for it but failed, I consulted the nearest LLM chatbot. It provided a further modified script, only when the hard drive did its thing again, it turned out that this script also deleted not 1, not 2 but 3 files. It was partially my fault because I tried to quit the script midway (since I had to pipe no through it but then realized that previously started but not finished files would also be skipped entirely) but I thought that there would be no harm in doing so since it was in process and wouldn't delete the file it was processing until after it was done at the earliest, and in which case it would be fine because it was done. Silly me. When I hit Ctrl+C, it just quit converting file, not the entire script and moved on to the next and idiot that I am, I just hit Ctrl+C again (I didn't yet realize that a file had already been deleted and I just deleted another) and it did the same thing. I finally quit in a way that actually quit the parent program/terminal window but not before deleting one more file in the process. And unfortunately as luck would have it, these files didn't just have some significance but also are apparently impossible to replace. Long story short I've since moved back to the script I posted here which involves the most amount of manual intervention but is also safe in that it won't delete anything. And of course the LLM chatbot shows no remorse 😂😭


Remove generated portion of package.json

Some parts of the package.json are generated when installing via npm.  These include the readme as well as other data about the project.  We can safely remove these sections since they do not need to
be in the package.json file, and are only added there when the module is installed via npm.

@antgraf I wanted to make you aware of this change before I committed it to master.  I noticed that these sections where in the package.json and there was some effort in keeping the readme portions in sync.  I felt we could remove these to avoid that work, and because it will automatically be added when installed via npm.
If you're good with this, I can also release.  I was thinking 0.5.0.

@dylants yes, please do.


Package rh-dotnet30-dotnet-runtime-3.0 seems to be missing System.Drawing.Common.dll

Problem encountered on https://dotnet.microsoft.com/download/linux-package-manager/rhel/runtime-current
Operating System: Linux RHEL 7 - x64
When installing rh-dotnet30-dotnet-runtime-3.0 as suggested here:
https://dotnet.microsoft.com/download/linux-package-manager/rhel/runtime-current
The folder:  /opt/rh/rh-dotnet30/root/usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.0
is missing System.Drawing.Common.dll
Which is part of the AspNetCore binaries here:
https://dotnet.microsoft.com/download/dotnet-core/3.0

The folder: /opt/rh/rh-dotnet30/root/usr/lib64/dotnet/shared/Microsoft.AspNetCore.App/3.0.0
is missing System.Drawing.Common.dll

There is a System.Drawing.dll at shared/Microsoft.NETCore.App/3.0.0/System.Drawing.dll. This is part of the regular .NET Core runtime, not ASP.NET Core.
What are you trying that uses System.Drawing.Common.dll?

I am not concerned with System.Drawing.dll We are using this assembly (System.Drawing.Common.dll) to edit png images.
When I install rh-dotnet30 it also installs the Microsoft.AspNetCore.App.  Below is the list runtimes command I run after I installed.

It is part of ASP.NET Core Binaries x64 binaries in the image above located:
aspnetcore-runtime-3.0.0-linux-x64 \shared\Microsoft.AspNetCore.App\3.0.0\System.Drawing.Common.dll
However the rh-dotnet30 seems to only be partially installing the Microsoft.AspNetCore.App runtimes.
Thanks for the help.

It is part of ASP.NET Core Binaries x64 binaries in the image above located:
aspnetcore-runtime-3.0.0-linux-x64 \shared\Microsoft.AspNetCore.App\3.0.0\System.Drawing.Common.dll

@dagood @crummel this is a difference between source-build and Microsoft build. I'm also surprised ASP.NET Core would pack a System.Drawing.*.dll, and not get it from corefx shared framework.
@davidgemel what doesn't work for you because this dll is missing? Is it the editing of the pngs? Or does that work? I'd expect some System.Drawing things to not work on RHEL7 because there is no native libgdiplus library available.

I'm also surprised ASP.NET Core would pack a System.Drawing.*.dll, and not get it from corefx shared framework.

What's included in the NETCoreApp shared framework vs. shipped as platform extension packages is determined by CoreFX, I'm not sure where to find what the decisions are. Hearing that a .Common extra assembly is pulled into ASP.NET's sharedfx from a CoreFX platform extension package doesn't seem strange to me though. (The name "Common" sounds weird in this context, but I don't know the reasoning behind that name either.)
I don't think there's any good reason for source-build to be different, somehow the dependency may have broken?

We are getting error that the assembly is missing.  Right now I am just suggesting to to copy the runtime over as workaround.  If assembly is in the folder everything works for me.

Filed https://github.com/dotnet/source-build/issues/1343 to track investigating/fixing this issue in future releases.

We are getting error that the assembly is missing.

When do you get this error? Are you executing some step using the sdk? Using some API in your application?

Looks like we are referencing System.Drawing.FontStyle, and System.Drawing.Imaging.ImageFormat.ImageFormat in System.Drawing.Common.  When we try and use these it will fail.

@dagood @dseefeld can we close this issue as duplicate of the new issue? Is there more to do here?


Make it possible to set max log size

Maybe we should have --max-log-size
Maybe we should also have an option that lets one control what happens with logs that are  bigger than --max-log-size.
Such logs could:

just get dropped, or
get trimmed to --max-log-size

I assume we should log such situations, so one can see when they have this problem.

+1 for cli and config entry
Currently this is possible with the features from logsene-js (used in logagent-js) by setting environment variables:
LOGSENE_MAX_MESAGE_FIELD_SIZE = 245760 # in bytes

The above setting would trims "message" field to 240K. Actually 240K is the new default in logsene-js.
Do we need the option to drop the message? I think its much better to trim it to the right size according to the field size (schema/field limitations) used in backends.

OK.  No need to add drop until we see the real need.


More tokens

For more productive development and testing our ideas we need more GitHub Personal Access tokens.
You can help us by submitting a token with at least repo: public_repo permission.
Leave it here, contact us or submit a pull request!

You can ask for permissions of Github Account from within the bot like @GitHubBot.


Process logging storage keys

Currently the logs, errors, processed errors, and warnings are stored in archives under either an ObsID or an ObsID+instrument+sub exposure ID combo.
This is somewhat tantamount to what the docstrings in the Archive class say, as they state either an ObsID or an ObsID+Instrument key combo.
I should consider having lower level instrument and then sub-exposure dictionaries to store the results/logs in, rather than ObsID+instrument+exposure ID. I intend to implement some sort of lookup method that can grab all results for an ObsID, or a specific ObsID instrument combo, and that would probably be easier with more distinct layers of dictionaries.

Now that emanom uses 'valid keys' - i.e. the ObsID+instrument+subexposure ID keys that the success flags are stored under currently - to determine which data can be processed further, that will have to be taken into consideration if I do change the logging keys


Issue random twitching of legs with no known changes in environment

When the bot is sitting idle after some sort of servo movement, it will occasionally go into twitching spells, where arbitrary legs make small repeated moves with apparently random timing

Some possibilities to ponder:
-I2C bus issues
- pull up resistor issues
- voltage or current variations

some flakey component
some flakey wiring

Some troubleshooting techniques:

increase 5V cap and see if twitching goes away.
scope the I2C bus to see if there's activity when we're not driving servos, but they're twitching
similarly, scope the +5 and +3.3 volts lines in various places
scope various grounds signals, to see if they agree
might be interesting to scope +5 / Gnd on a servo we're not driving at all.
see if there's any twitching when only one PWM board is connected. If not, it might be a contention issue
implement a second I2C bus so activity on both PWM boards can overlap without I2C competition

Haven't seen this twitching (except on initial startup) since increasing the caps on the PWM boards. If this remains the case, I'll close this issue, blaming the caps.
suspect the startup case can be solved by making sure the software sends some command to each servo before the 5V power is enabled, so the servo position isn't undefined when +5V appears. This could be done by the startup software if we put in a relay that controls the 5V power as we considered, and use pullups to ensure it comes up with 5V disabled until a command has been sent to all servos.

I have also noted the issue is much better with new caps.

We put larger capacitors on the controller boards and this seems to have greatly improved the issue. We are now discussing possibly making the caps  even larger to deal with boot up issues but we can consider this issue closed.


chore: move hardhat-solidity-tests to the EDR repo

Good catch, thanks!
While writing the readme I also realized that it makes sense to have the example in this repo too, so no pnpm link is needed at all for running it.


Error while accessing the CustomLearningAdmin.aspx page

I get an error while accessing the CustomLearningAdmin.aspx page.

Hi,
Thanks for your feedback. Do you have the PnP Correlation ID of the provisioning action? Thanks.

I am afraid I don't. Is there a way I can find out from the current Site ?

Without that information, I cannot figure out what happened. I'm sorry. Eventually, you can try to apply the template one more time, onto a new site, and see if the issue is still there. In case of any further exception, please open another issue providing the new PnP Correlation ID. Thanks.

Thanks. I provisioned a new Site and It Works fine. May be something got corrupted.
Vamsi Kotta
Chief Technology Officer
Work:<PHONE_NUMBER>    Cell:<PHONE_NUMBER>
Email:        vkotta<EMAIL_ADDRESS>Website:    www.SMART-INFRASTRUCTURE.comhttp://www.smart-infrastructure.com/
Office:        4152 W Blueheron Blvd, Suite 1114
Riviera Beach, FL 33404
<EMAIL_ADDRESS>The contents of this e-mail message and any attachments are confidential and are intended solely for the addressee.
The information may also be legally privileged. This transmission is sent in trust, for the sole purpose of delivery to
the intended recipient. If you have received this transmission in error, any use, reproduction or dissemination of this
transmission is strictly prohibited. If you are not the intended recipient, please immediately notify the sender by
reply e-mail or phone and delete this message and its attachments, if any.
From: Paolo Pialorsi<EMAIL_ADDRESS>Sent: Friday, August 07, 2020 9:15 AM
To: SharePoint/sp-provisioning-service<EMAIL_ADDRESS>Cc: Vamsi Kotta<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [SharePoint/sp-provisioning-service] Error while accessing the CustomLearningAdmin.aspx page (#253)
Without that information, I cannot figure out what happened. I'm sorry. Eventually, you can try to apply the template one more time, onto a new site, and see if the issue is still there. In case of any further exception, please open another issue providing the new PnP Correlation ID. Thanks.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/SharePoint/sp-provisioning-service/issues/253#issuecomment-670510458, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQQL6NFXTCZOYSOMRLVN6GTR7P45NANCNFSM4PWVBIUA.