feat(bindable): support coercing value

This PR add support for coercing value, before calling setValue on property, to address the issue: #96
Usage is similar to @jdanyow 's proposal
class MyViewModel {
  @bindable({ coerce: 'number' }) numProp
  @bindable({ coerce: 'boolean' }) boolProp
  @bindable({ coerce: val => convertValue(val) }) customCoerceProp
}

To modify / extends the built-in coerce types:
import {coerces} from 'aurelia-templating';

coerces.string = function(a) {
  return a === null || a === undefined ? '' : a.toString();
}

To add more coerce types:
coerces.longDate = function(a) {
  return new Date(a).toJSON();
}

// usage:
@bindable({ name: 'date', coerce: 'longDate' })
class DateControl {

}

All basic implementations:

const numCons = Number;
const dateCons = Date;
const _isFinite = isFinite;
const _isNaN = isNaN;

export const coerces = {
  none(a) {
    return a;
  },
  number(a) {
    var val = numCons(a);
    return !_isNaN(val) && _isFinite(val) ? val : 0;
  },
  string(a) {
    return '' + a;
  },
  boolean(a) {
    return !!a;
  },
  date(a) {
    return new dateCons(a);
  }
};

To avoid breaking changes/ bug, any fail attempt to get coerce function will result into a identity function, which is coerces.none
cc @EisenbergEffect @jdanyow

@EisenbergEffect if this passes, it will be trivial to support your proposal:
class MyViewModel {
  @bindable.number numProp
}

Cool. The other thing we'd want to do is support TypeScript metadata, so if they do this...
number: boolean;

...we should pick up the metadata and map it to a coerces function as well.

I have added the support for Typescript.
Now it supports this:

import {mapCoerceForClass} from 'aurelia-templating';

class Point {}

mapCoerceForClass(Point, 'point', function(value) {
  return new Point(value.split(' ').map(parseFloat));
});

class LineViewModel {
  point1: Point
  point2: Point
}

Basically you can register how to coerce a class. JavaScript users can also leverage this by setting metadata on property:

import {mapCoerceForClass} from 'aurelia-templating';

class Point {}

mapCoerceForClass(Point, 'point', function(value) {
  return new Point(value.split(' ').map(parseFloat));
});

class LineViewModel {
  @Reflect.metadata('design:type')
  point1

    @Reflect.metadata('design:type')
  point2
}

@EisenbergEffect one thing I'm not sure & confident about is naming of mapCoerceForClass & classCoerceMap (this is a map holding Class definition and their string representation for look up in coerces export)

I just re-read the issue again and it seems you @EisenbergEffect  agreed with @niieani that it shouldn't use metadata to automatically coerce value. So I'm not actually sure about the changes related to the metadata.

Don't worry too much about the api yet. We can tweak that. If you'd like to keep working on this (awesome work btw!) then I'm wondering if you can investigate the binding library's observable decorator. I can imagine wanting to support this with both observable and bindable, so I'm wondering if we can put the code in the binding library somehow, or at least the core of it that is independent of the decorators.

@bigopon, nice work! I added couple of comments tough

@jdanyow Can you work with @bigopon to address any issues and add support for this for both observable and bindable? This is a long time requested feature and one I know people will want to use.

@jdanyow  @EisenbergEffect
I have:

fixed all naming issues
added a flag to turn off the type auto pickup by default. Also export a function to turn it on
added a function createTypedBindable to create bindable with fluent syntax @bindable.string.
created built in for string, number, date, boolean with createTypedBindable

Please let me know if I missed something.

@jdanyow
I have added typings for bindable, BindableProperty. bindable.string, ...etc.
I haven't added typings for coerceFunctions and mapCoerceFunction because it would be more desirable to have both @observable and @bindable to have same sort of functionality (per Rob). So I think we will need to move coerceFunctions and mapCoerceFunction there. There was a PR in binding for that but I messed up my history and it's gone. Can easily recreate though.
I don't know if this is the final desired behavior, as currently it still doesn't play nice together with two-way binding. Basically it can't be synced well with input, for current binding notification model. And it would take not trivial amount work to support it properly, consider syncing between a date bindable property to an input, or select. We can, however, encourage user to use from-view and typed bindable.
What do you think ?

@jdanyow @EisenbergEffect doc is ready too. oh docs ...

@bigopon Can you give me an update on where we are at on this. Seems like it stalled out. I'd love to have types bindables/observables though.

This looks really awesome, @bigopon - great work, and much appreciated!
I do have a couple of concerns though:

The use with TypeScript seem too complicated. If the property type is number, and the value being set is a string, it should just always coerce by default - having to explicitly turn that on for every single bindable property will be a huge pain, and it would make no sense, as you already specified that the desired type, and the compiler will complain if you try to treat it as anything else.
Not coercing by default will just cause bugs, exactly as it does today, where the compiler and developer thinks a property will be of the type specified, but then at runtime, it ends up being something completely different. In our code, this is probably our single biggest source of bugs.

As I understand it, boolean attributes such as disabled are not handled automatically, but requires a coerce function to be specified. I think handling boolean attributes according to the HTML spec should be the default behavior.
Previously, we also talked about having a reflect option on the bindable decorator, which would control whether the property value is reflected to the DOM attribute or not - I think that’s also still relevant.

The concept of coerce functions do not support two way binding, as far as I can tell, and I’m wondering whether we need this new concept at all? We already have value converters, which essentially do the same thing, and support two-way conversion - maybe we should just allow those to be used, instead of introducing a new concept?
The only issue with that, would be how to support boolean attributes, where they have to not just convert the value, but also modify the attribute on the element. However, since boolean attributes are really a very special case, I think it might be ok to just hardcore support for that...
So if that reflect option I mentioned is set to true, and toView returns true, then the attribute is automatically added with an empty string as value. If toView returns false, null or undefined, then the attribute is automatically removed. Maybe with an exception for attributes whose name starts with aria-, as those do not follow the standard behavior, but requires the values “yes” and “no”.
It’s a simple solution, but it gets the job done, and it would also solve previously reported issues with attributes, such as src and href, which should only be present if they actually have a value.
An alternative approach might be to, when the value converter is used for coercion and that reflect option is true, pass the element and attribute name to the toView and fromView functions of the converter. Then the converter could make the nessesary attribute changes - and again, if we choose this approach, then I think the boolean converter we provide out of the box should do this by default.

Now, we might need options to opt-out of some of this default behavior until app code is updated, but I think that will be the best long-term strategy, as opposed to littering the app code with options to opt-in on every bindable property.
Thoughts, @bigopon, @EisenbergEffect?

@thomas-darling

The use with TypeScript seem too complicated. If the property type is number ...

Are you referring to bindable.usePropertyType(true) ? If that is true, then it shouldn't be a concern as it's supposed to be called only once. It is there for compatibility reason.

As I understand it, boolean attributes such as disabled are not handled automatically ...

View model properties are not supposed to be used with view only, but also interacted within JavaScript / TypeScript. so by default I don't think we can treat @bindable.boolean similar to Html boolean attribute. Consider "", it's true for Html boolean attributes but false for Js/Ts boolean. I think best we can do is to have another default: @bindable.booleanAttr where it treat "", or "required", "readonly" as true

The concept of coerce functions do not support two way binding, as far as I can tell, and I’m wondering whether we need this new concept at all? ...

It could be nice if we can avoid it, but for me, value converter are somewhat view related logic, while typed bindable/ observable are supposed to work seamlessly in script side. Beside that, it would bring huge overhead to make current value converter work within Ts/Js I think. @EisenbergEffect & @jdanyow can give better feedback

Previously, we also talked about having a reflect option on the bindable decorator ...

It's is being discussed here #560 . Ideally we should have it reviewed after this PR is decided to be merged or not

The concept of coerce functions do not support two way binding ...

It depends on the target of the binding, if it's a custom element / custom attribute, then it works well and as expected, except some cases like gotchas in the observable PR. For Html built-ins, value are always coerced to string so it's pretty much nothing we can do.

@EisenbergEffect, @jdanyow - i've been really dreaming about this issue since the beginning. Contrary to what @thomas-darling said, this feature is useful and easy to use with TypeScript, as reflect metadata is available at runtime and the fact that we'd need to call bindable.usePropertyType(true) just once during Aurelia app configuration doesn't make absolutely any difference for me.
Fantastic work @bigopon, i hope it will be merged soon!

I have some refactor suggestions, getting inspiration from the validation plugin.
In validation we're using static methods and properties to add custom rules without having an instance. Maybe we should be doing the same here?
ValidationRules.customRule(...);
So here are my suggestions
// Add fluent binding syntax for coercion converter by default.
// Optionally state a property type. Useful for TS users or JS using reflection metadata.
Coercion.customConverter(key, converter, propertyType?);

// Interpret of property types should be set on a Coercion object and not on bindable in my opinion.
Coercion.usePropertyType(true);
// or
Coercion.interpretPropertyType(true);


[SPARK-18191][CORE][FOLLOWUP] Call setConf if OutputFormat is Configurable.

What changes were proposed in this pull request?
We should call setConf if OutputFormat is Configurable, this should be done before we create OutputCommitter and RecordWriter.
This is follow up of #15769, see discussion here
How was this patch tested?
Add test of this case in PairRDDFunctionsSuite.

cc @rxin @mridulm

Test build #68384 has started for PR 15823 at commit 51d1fbe.

LGTM otherwise.

Test build #68391 has started for PR 15823 at commit 4e79c37.

Test build #68384 has finished for PR 15823 at commit 51d1fbe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68384/
Test PASSed.

Merged build finished. Test PASSed.

LGTM. Not at my laptop, would be great if you can merge @rxin, thanks.

Test build #68391 has finished for PR 15823 at commit 4e79c37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68391/
Test PASSed.

Merged build finished. Test PASSed.

Merging in master.


address the RPC detector not found complaint fro m GeometryService

%MSG-w LoadNewGeometry:  BeginRun 16-Dec-2023 12:07:17 CST run: 2159
RPC detector not found in gdml.
check your spelling.

Ah. This is because we had 2 RPCs for 1c, and they're labelled RPC0 and RPC1.
Phase1b won't have this complaint.
Perhaps we can make an RPC_phys volume that contains RPC0 and RPC1?

Fixed in PR #149


Add tag for all SecRules with SecDefaultAction

I'm trying to add a tag in all SecRules. In @crs-setup.conf this is  already outlined in an example for SecDefaultAction:
SecDefaultAction "phase:1,log,auditlog,redirect:'http://%{request_headers.host}/',tag:'Host: %{request_headers.host}'"
SecDefaultAction "phase:2,log,auditlog,redirect:'http://%{request_headers.host}/',tag:'Host: %{request_headers.host}'"
But if I uncomment this lines an error message is returned when restarting Caddy, describing that "SecDefaultAction must not contain metadata actions".

Hi, this is tricky because of inconsistent documentation.
SecDefaultAction "phase:1,log,auditlog,redirect:'http://%{request_headers.host}/',tag:'Host: %{request_headers.host}'"
SecDefaultAction "phase:2,log,auditlog,redirect:'http://%{request_headers.host}/',tag:'Host: %{request_headers.host}'"

These lines are 8 years old from the crs-setup.conf.example, but accordingly to documentation, both Coraza and Modsecurity are actually not supporting it.
Mentioning both  ModSec v2 and v3 docs:

Every SecDefaultAction directive must specify a disruptive action and a processing phase and cannot contain metadata actions.

With metadata actions including tag:

metadata actions (id, rev, msg, tag, severity, logdata)

I'm sharing this conversation in Slack #coreruleset: https://owasp.slack.com/archives/CBKGH8A5P/p1729806584858919, where we can evolve it with Coraza, CRS and Modsec people.


JAVA-2818: Remove root path only after merging non-programmatic configs

This PR reverts most of the changes introduced by JAVA-2657 to the programmatic config loader builder.

LGTM
For context, the "root path" mechanism exists so that different sessions can be configured with one config file, see this.

Interesting, I never thought about that possibility. I think I would have gone with different config files in that case (e.g. application1.conf and application2.conf). Parameterizing the prefix path seems a bit more involved.
Reading the docs you linked, it seems that this example is outdated:
DriverConfigLoader session1ConfigLoader =
    new DefaultDriverConfigLoader(
        () -> loadConfig("session1"), DefaultDriverOption.values());

I will look into it tomorrow.


Fixes #8 - try to find the correct symlink where the pgp keys are

Use the actual Bazel project name.

ping @MarkusTeufelberger  PTAL !


MinGW support

One of our devs is using Windows / msys for his go development. gpm doesn't seem to work with the bash, since it doesn't support full process redirection.
(This is mainly a placeholder to remind me to work on this.)

+1


JQuery Not Working In Modal

Error : $(...).modal('show') is not a function is shown in the console.
I am implementing modal and on clicking a button I have made call to a function and and in that function I have called for $('#exampleModal').modal('show').
The Modal is not getting Triggered.

@xidedix Thank you for your response.


support for generators and async functions

Exposing a generator (function*) or an async function (async function f(){...*) using eel.expose does not seem to work. From my investigation, the way eel.js extract the name of the function is a bit brittle. Why not using a regex or simply f.name in eel.expose ?
For the case of generators, the values yielded can't be retrieved as their return value is a Generator object.
I have worked on these limitations on my fork. Shall we consider a PR ?
Edit : Generators from the python side are more or less handled in this branch https://github.com/brumar/Eel/blob/consenting_adults_fast_track (but it also contains experimental features related to another project so it's not PR ready.  Still you can check the changes in _process_message to get the idea. This is the same principle used in the js side. Values are accumulated in a temporary list and sent via websockets only when the iterator ends. In retrospect, waiting is not really the most desirable implementation, this would be way better to have the possibility to send values one by one to a js callback if there is one. Still that may be a starting point.

Why not using a regex or simply f.name in eel.expose ?

Fundamentally it is because we need the function objects to appear as methods on eel object in Python before the JS engine is running.
I 100% agree it is brittle, it is the worst part of the library, but I can't think of a better way to do it.

Sorry, I think I was unclear.
At this line https://github.com/ChrisKnott/Eel/blob/master/eel/eel.js#L9 I think you can get the name of the function just by f.name, I think it's more reliable than parsing f.tostring() or maybe is there some edgecase I did not think of?


Fix/Improve makefile file recipes

Remove redundancy and blight in file recipes, e.g. %.html: %.rst
Output proper links for pdf and txt output files.
For example, ExampleTopic/TopicX/mdTopicX.md has a link to ExampleTopic/TopicX/rstTopicX.rst
After mwk all there will be html, pdf and txt output files for both the above markup files.
All of those files link to the html output file, but it makes more sense for:
html -> html
pdf -> pdf
txt -> txt
In other words, if you're viewing an html output file in a browser, and there's a link to another file generated from a Miki markup file, then you would naturally want to go to the generated html file version of the destination link. This happens now.
But while viewing a pdf file, you would want to follow a link to another Miki file to its pdf version, not its html version. Right now pdf output files link to html output files. Same problem for text output files.

Fixed in cca2452f45dba7983f005d110a53c98fc7a1acaf
From the commit log:
Fix makefile link generation, refactor makefile.
makefile:
- Make links in generated pdf and txt files point to
  other generated pdf and txt files, not to generated html files.
- Pull common sed actions into commonly used make variables,
  e.g. MWK_TO_HTML_SED
- sed remove ascii null '\x00' from source stream before final output
  generation.
  - Use of ascii null was added to enable inhibiting $MWK and link
    expansion when an author needs that.

mdStarter.md, rstStarter.rst:
- Add details on writing links.

mdTopicX.md, rstTopicX.rst:
- Use \x00 to inhibit suffix expansion in the example where the output
  file points to its source.

Readme.rst:
- Remove most of link writing section.
  - Now refers to better coverage in mdStart.md, rstStarter.rst.


'data.maybe_num_nodes' should not be used to guess the number of nodes without a warning

🐛 Bug
To Reproduce
I have a dataset of Data objects that have y, edge_index, and edge_attr attributes but no x attribute (the x is generated on the fly by randomly removing some values in y). When I train a network using batches of data generated using DataLoader with batch_size > 1, I observe terrible performance. Reducing batch_size to 1 results in better performance.
Long story short, after two days of debugging I realize that torch_geometric is incorrectly guessing the number of nodes in my graphs (since they may not be fully connected), resulting in shifted adjacency matrices for all subsequent graphs in the batch.
Expected behavior
In the very least, a warning should be emitted when  we are "guessing" the number of nodes in a graph. Also, y should potentially also be used to infer the number of nodes in a graph?

Hi, I'm really sorry for your lost days regarding this issue. I'm not quite sure what causes the performance issue, since "guessing" refers to a simple edge_index.max() call (which should by no means result in significant performance decreases). We could use y to infer the number of nodes, but this could potentially result in unexpected behaviour since we declare no constraints on the shape of y.
I think you have two simple ways to fix your issue:

Inherit from Data and create your own data object where you redefine num_nodes.
Add a small placeholder tensor to data.x with correct shape, e.g., data.x = torch.zeros(num_nodes, dtype=torch.uint8)

However, can you elaborate a little more why PyG is wrongly guessing the number of nodes?

I'm closing this due to inactivity. Feel free to reopen.

Sorry for the delay in responding.
I have brewed on this issue for a bit, and I guess it is more of a feature request than a bug report. PyG provides an awesome API for working with graph data, and it has greatly simplified my workflow. However, I believe that in some cases it provides a bit too much "magic" rather than following the "explicit is better than implicit" mantra. An example of this, which in my case led to a hard-to-find bug, is return maybe_num_nodes(self.edge_index) inside the Data.num_nodes property. Guessing the number of nodes in a graph from the edge_index vector is not going to be correct in 100% of the cases (this is why you appropriately named the function as maybe_num_nodes). The amount of time this function saves in 99% of the cases is more than offset by the amount of time it takes to find the cause of obscure bugs in 1% of cases. In my case, the discrepancy between maybe_num_nodes and the actual number of nodes led to incorrect edge information for all but the first datapoint in Batch objects generated using the DataLoader.
My personal opinion is that it should be mandatory to overwrite the Data.num_nodes property for data points without an x attribute. Maybe in the documentation there could be an example of how to do this with return maybe_num_nodes(self.edge_index).
Anyway, this is just my 2c. Feel free to ignore, as I am sure that you have more important things to work on, and this is not a "bug" as long as someone is aware of the implementation details.

Mh ok, I see what you mean there. We could make num_nodes an actual property that users can override. I leave this issue open until I come up with an elegant way to fix this. Thank you so much for your feedback.

Hi, I added this functionality to current master. You can now set data.num_nodes = ... explicitly, and are given a warning if PyG defaults to "guessing" based on edge indices. I'm closing this for now. Feel free to reopen if you encounter any issues. Thanks again!

Awesome, thanks!


Add obfuscated password storage

There are times when it would be nice to have the vSphere password stored for scripted use, but leaving it in plaintext is a little more easily shoulder-surfable than I'd like. I've added an option to store the vSphere password base64-encoded, so it's less obvious on casual inspection that my password matches my luggage combination.
(not sure if this is too hacky, but since it's useful to me I thought I'd toss it out here)

Hi Troy,
This is a useful feature, thanks for the pull request. As a thought, instead of having two different ways to configure a password (vsphere_pass and vsphere_pass_obf), what about allowing something like `knife[:vsphere_pass] = "base64:cGFzd29yZA==" which would let us push the looking up of the config and potential decoding into a method called from line 99?
Other other thing -- could you update the README to reflect the new feature, please?

Good thoughts. I'll take a stab at it when I get a chance

Ok, I've made the change and added a brief node to the readme about it

Sure -- should be updated now

Thanks, Troy!


Improved App/Sync exceptions

Closes https://github.com/realm/realm-kotlin/issues/524
The purpose of this PR is to improve how errors are being reported for Sync/App Services.
High-level exception hierarchy is as follows (feel free to bikeshed naming):

AppException: RealmException

ServiceException: AppException

BadRequestException: ServiceException
ConnectionException: ServiceException
AuthException: ServiceException

UserNotFoundException : AuthException
UserAlreadyConfirmedException: AuthException
UserAlreadyExistsException: AuthException
InvalidCredentialsException: AuthException

SyncException: AppException

UnrecoverableSyncException: SyncException
WrongSyncTypeException: SyncException

Primary classes to review: AppExceptions and RealmSyncUtils

Exceptions have all been moved from internal cinterop to public library modules in  io.realm.exceptions and io.realm.mongodb.exceptions.

All Sync/App errors are not being propagated as data classes and then converted into exceptions on the library side (so we can use the public API exceptions). This is happening in RealmSyncUtils

Many enums have a static of method for fetching the Kotlin enum value from the cinterop value. This is intended to be used internally. Because we need to map these values on the library side, I have added a public fromInt() method to all enums that need it. In some cases, it mirrors the of method completely (on JVM)`, which is a bit silly. For now, that seemed like the best way to split concerns. But we should probably discuss it.

All exception message attempt to follow the same format and will now look like this [<category>][<errorCodeDescription>(<errorCode>)] <Message>. Server log: <server link>. Example [Service][UserAlreadyConfirmed(44)] This user was already confirmed. Server log: https://link.to.server/logs

TODO

[x] Settle on an approach
[x] Define high-level idea driving the API and the rough categories.
[x] Define the exact set of initial exceptions we want to expose.

This PR should be ready enough to start reviewing, while I cleanup the last few remaining things.


Simplify Zipkin exporter creation with Options

The current Zipkin exporter has a cognitive load of
having to import many dependencies to get a basic
exporter started. This change simplifies that with
a function NewExporterWithOptions which abstracts
away endpoint creation, reporter creation and
the user at bare minimum just has to pass in
two strings:

EndpointHostPort e.g. <IP_ADDRESS>:8080
ReporterURI e.g. http://localhost:9411/api/v2/spans

usually bare minimum are local servicename and reporting uri. If you have no service name, the zipkin dependency graph will be crap, as will the drop-down in the UI. The ip/port info for the local endpoint is less important (usually it is possible to find a link-local IP automatically)

Thank you @adriancole for the education :)

Can we progress on this PR or abandon it if it is not required?

I can continue if Emmanuelle is busy.
Den tor. 1. mar. 2018, 21:34 skrev JBD<EMAIL_ADDRESS>
Reopened #453
https://github.com/census-instrumentation/opencensus-go/pull/453.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/census-instrumentation/opencensus-go/pull/453#event-1499886342,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC7sAiNB4ix3Ra0dDYijLXFahXcKUKzMks5taFtDgaJpZM4SHaAc
.

--
José Carlos Chávez
Spain: (+34) 672921339
Perú: (+51) 708-5422 Anexo 918
https://github.com/jcchavezs

Can we progress on this PR or abandon it if it is not required?

Doesn't seem like the consensus is tending towards merging it, I can abandon it

I can continue if Emmanuelle is busy.

@jcchavezs, thank you very much, please take over it, feel free to modify the code, address the concerns and when ready, please open a PR and we'll review it.
For now I'll abandon this PR as the benefits are barely marginal and the cost is too high.


Refine the runtime memory test cases

The stress-ng should be run in background, otherwise, the demand memory is not right.

Test results:
[LISAv2 Test Results Summary]
Test Run On           : 09/26/2019 12:32:42
VHD Under Test        : D:\vhd\Ubuntu_18.04.1v.vhdx
Test Category         : Functional
Test Area             : RUNTIME_MEMORY
Initial Kernel Version: 4.15.0-38-generic
Final Kernel Version  : 4.19.67-e0ea5c2d9592
Total Test Cases      : 9 (9 Passed, 0 Failed, 0 Aborted, 0 Skipped)
Total Time (dd:hh:mm) : 0:1:2

   ID TestArea             TestCaseName                                                                TestResult TestDuration(in minutes)
-------------------------------------------------------------------------------------------------------------------------------------------
    1 RUNTIME_MEMORY       RUNTIME_MEM_HOTADD                                                                PASS                 5.04
    2 RUNTIME_MEMORY       RUNTIME_MEM_HOTREMOVE                                                             PASS                 5.67
    3 RUNTIME_MEMORY       RUNTIME_MEM_SMALLINCREASE128                                                      PASS                 6.75
    4 RUNTIME_MEMORY       RUNTIME_MEM_SMALLDECREASE128                                                      PASS                 6.69
    5 RUNTIME_MEMORY       RUNTIME_MEM_SMALLINCREASE100                                                      PASS                 6.71
    6 RUNTIME_MEMORY       RUNTIME_MEM_SMALLDECREASE100                                                      PASS                  6.9
    7 RUNTIME_MEMORY       RUNTIME_MEM_MULTIPLEADDREMOVE                                                     PASS                 7.32
    8 RUNTIME_MEMORY       RUNTIME_MEM_STRESSHOTREMOVE                                                       PASS                 6.79
    9 RUNTIME_MEMORY       RUNTIME_MEM_HOTADD_REBOOT                                                         PASS                 5.11

@xian123 Can you post the test result of your final change? If passed, I can merge it.

Test result:
[LISAv2 Test Results Summary]
Test Run On           : 09/30/2019 02:12:13
VHD Under Test        : D:\vhd\Ubuntu_18.04.1.vhdx
Test Category         : Functional
Test Area             : RUNTIME_MEMORY
Initial Kernel Version: 4.15.0-38-generic
Final Kernel Version  : 4.19.67-e0ea5c2d9592
Total Test Cases      : 9 (9 Passed, 0 Failed, 0 Aborted, 0 Skipped)
Total Time (dd:hh:mm) : 0:1:3

   ID TestArea             TestCaseName                                                                TestResult TestDuration(in minutes)
-------------------------------------------------------------------------------------------------------------------------------------------
    1 RUNTIME_MEMORY       RUNTIME_MEM_HOTADD                                                                PASS                 5.24
    2 RUNTIME_MEMORY       RUNTIME_MEM_HOTREMOVE                                                             PASS                 5.94
    3 RUNTIME_MEMORY       RUNTIME_MEM_SMALLINCREASE128                                                      PASS                  6.9
    4 RUNTIME_MEMORY       RUNTIME_MEM_SMALLDECREASE128                                                      PASS                 6.79
    5 RUNTIME_MEMORY       RUNTIME_MEM_SMALLINCREASE100                                                      PASS                 6.99
    6 RUNTIME_MEMORY       RUNTIME_MEM_SMALLDECREASE100                                                      PASS                  6.9
    7 RUNTIME_MEMORY       RUNTIME_MEM_MULTIPLEADDREMOVE                                                     PASS                 7.41
    8 RUNTIME_MEMORY       RUNTIME_MEM_STRESSHOTREMOVE                                                       PASS                 6.69
    9 RUNTIME_MEMORY       RUNTIME_MEM_HOTADD_REBOOT                                                         PASS                 5.23


Revise coherence calculation

Currently calculation of coherence implies that a corpus.txt with the full training corpus needs to be created. Note I did this even for the Spark LDA, even though this is not in principle the format in which the training corpus is needed by Spark (we use bow in parquet format). For very large training corpus this is very inefficient, since that corpus.txt file which is only needed for coherence calculation can take up several Gigabytes !!!
We need to:
1/ Revise TMmodel generation process, so that if coherence calculation fails it does not crash (same could be done with pyLDAvis, by the way)
2/ Revise what is actually needed for coherence calculation. ¿Is there an alternative to using the original corpus? ¿Could we have a very large corpus per dataset that is used for everything?  ¿Could we use a smaller corpus sampling the original corpus?
3/ Revise also if corpus.txt is not needed after model training (or just needed for coherence calculation), could we delete this file?

In addition to this, is it possible to calculate coherence without building the Gensim dictionary again ???  This is inefficient.
Let us wait and see how long does it take to preprocess Spark vs Dask.  Because in case we do not see a clear gain in using Spark for 1/ Preprocessing and 2/ topic modeling, we could opt altogether to remove the option of training using a Spark cluster and make the whole topic modeling service one which is 100% dependent on a high-performance server with a lot of memory and many processors.  This could simplify things considerably.   Let us think about it as we progress on some of the issues.


Create lakshay451.json

Looks good

@accodes21 as previously mentioned do comment a PR is approve without doing a proper check - in this case the JSON was failing.  This causes confusion to the author and maintainer.


extruded buildings

Not a lot, just a little

Got this mostly working by using static extrude values like 20 and 100 and 200, but some of the building faces are still extruding up to the stratosphere.
I suspect the building shader needs to be revised? @meetar for help!

/cc @sensescape

It'd also be nice to remove the fisheye effect (so all buildings had an lower-right shadow at the top left of the screen and the bottom right of the screen).

On the case 🔎

Turns out the camera settings weren't being applied, so the default "perspective" camera type was used, which led to the "fisheye" effect – this is fixed by https://github.com/tangrams/outdoor-style/pull/69.
The "supertall" buildings were a bug in Tangram where min_height properties weren't being handled properly, which was discussed in https://github.com/tangrams/tangram/issues/337 and fixed by https://github.com/tangrams/tangram/commit/43dcac737e71e1b637d783238681415a0356a4d9.

@meetar @bcamper is https://github.com/tangrams/tangram/commit/43dcac737e71e1b637d783238681415a0356a4d9 in a released version of Tangram yet? I'd like to release the Outdoor style this week, seems like a nice to have for that.

Yep! 0.8.1 was released 5 days ago: https://github.com/tangrams/tangram/releases/tag/v0.8.1
The current outdoor-style index.html specifies 0.8 which means it will pull in any subreleases, which means it should be showing the fix now.


Client does not appear to set Origin: header

Hi,
Is there a way I can set this via calling code?
And, more generally, any other headers? Or is this a wsproto thing?
I have a rather badly behaved server (it's a sky set top box) and it seems to be not responding unless an origin header is present...
thanks
Brad

Ok... Here is my situation:
WORKING (go-http-client):
---req---
GET /as/system/status HTTP/1.1
Host: skyq:9006
User-Agent: Go-http-client/1.1
Connection: Upgrade
Origin: http://skyq:9006/as/system/status
Sec-WebSocket-Key: 3pDOFIejJw9aFZ84xAus+A==
Sec-WebSocket-Version: 13
Upgrade: websocket

---res---
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: 0INWfq7txLmN/1LGLgpPUl+OfA8=

<json payload follows>

NOT WORKING (trio-websockets + wsproto):
---req---

GET /as/system/status? HTTP/1.1
host: skyq:9006
upgrade: WebSocket
connection: Upgrade
sec-websocket-key: xNtNd7KbRC0kZu14wayhRA==
sec-websocket-version: 13

---res---
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: xCfjTlQDTFLgO1IFiV8a8eK1Ar8=

<empty body :(>

Any ideas on workarounds?  This basically blocks me completely :(

Also note the trailing ? on the trio/wsproto URL --- I didn't add that -- is it coming from the lib somewhere?

If you want to stick with Trio and need an immediate fix, the best suggestion I have is to clone wsproto locally and hack your origin header into it. This is the location you'd need to edit: https://github.com/python-hyper/wsproto/blob/master/wsproto/connection.py#L113
If asyncio is an option, this is a high quality websocket library that I've used extensively: https://github.com/aaugustin/websockets

Also note the trailing ? on the trio/wsproto URL --- I didn't add that -- is it coming from the lib somewhere?

Yeah, that's a bug that's currently fixed on master. I can post a new release on PyPI if you need it.

I had this issue with aiohttp, but never with aaugustin's stuff under asyncio, which just worked, so I agree -- its high quality.   But I'm off asyncio now and onto trio so looking to stick with that now.
Re the ? -- if its not too much trouble to do a patch release on pyPI I'd appreciate it.
thanks for your help and responsiveness Mark!

BTW, I just looked at bit of wsproto code and the headers there are Capitalised, but in my dump they're not. Are you lowercasing them somewhere else? I know the RFC says case-insensitive, but my server might be pinickity about that too.

Version 0.3.0 just released on PyPI.

thanks again @mehaase   👍

You might want to try confirming that it's the origin header first...  And then play around with the headers you send until you figure out what the minimal change is that switches it from working to not working.

turns out my issue was with the case of headers @mehaase but the questionmark is now nicely removed thanks...

The h11 issue is here: https://github.com/python-hyper/h11/issues/31


Multiselect with initial values is not showing any text for the selected elements

I have a multiselect select field with two options, both of which are selected.  When I initialize select2 for an ajax data source, the selected values show up in the box, but they have no text.  Upon inspection in chrome, I can see that the both the text and value were picked up, but instead of the text being put inside the appropriate tags, it's set as the "title" attribute of the appropriate tag.  This is bizarre.  It simply does not work as described, and it's costing me a lot of time.

I figured out what the problem was.  By default, the options are read and turned into objects with an "id" and "text" property.  The custom format I'm receiving from the server uses a "username" property instead of ID.  I had to change my format function to return data.username || data.text, and that fixed it.  It feels like a design flaw, however, since I'm re-defining a behavior.   Instead of defining a function that maps a value from a datasource, why isn't there just a property to redefine the default name?  That way it's read from the server with that name, and the default option reader would assign the option text to that name as well, eliminating the need for the function.  DataTables does this for redefining their row id field.

I have the same issue, you save me @triynko :)

By default, the options are read and turned into objects with an "id" and "text" property. The custom format I'm receiving from the server uses a "username" property instead of ID.

https://select2.github.io/options.html#my-objects-dont-use-id-for-their-unique-identifiers-what-can-i
https://select2.github.io/options.html#my-objects-use-a-property-other-than-text-for-the-text-that-nee

why isn't there just a property to redefine the default name?

Select2 previously allowed this, through the use of an id function/option which did the mapping. In 4.0, a decision was made to standardize everything to use id and text instead of referring to custom mapping functions.

Thanks @triynko


regression: page-to-alto is missing

Recently, we lost ability for page alto conversion. For some reason, during make install, no page-to-alto gets installed. Investigating...

Thanks, I'll have a look.

Thanks, I'll have a look.

have a look at #154, then :)

Culprit is here

I tested f799fdda56adc43e2211017ec593f2ad72891bb0 (the one before the "culprit"), and it also did not install page-to-alto.

gcv2hocr is missing, too.

I checked all my local ocrd_all builds from 2022 and only found a single one from May 15 with bin/page-to-alto. @bertsky and @kba, are you sure that there is a regression?


ci: Enable macOS build

https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

Not free yet,
https://github.com/actions/runner-images/issues/8439#issuecomment-1743526558


tsdb: expose hook to customize block querier

In TSDB, during query time, blocks are only included or pruned based on time range and the logic is controlled in blockChunkQuerierForRange and db.Querier function.
We want to experiment with different strategies so that Querier can apply additional filtering logic in the hook based on BlockReader. See the unit test added TestBlockQuerierAndBlockChunkQuerier. The filtering can be based on metadata from the block.

Thank you @bboreham for the review. Is it ready to be merged or requires another approval? Thanks

Can we get this PR merged? Thanks

At #14354 I am working on more efficient querying where the head has both in-order and out-of-order chunks.
I could extend your hooks with a third function for that case, but it seems to be getting ugly.
Do you have any other ideas?

@bboreham TBH I am not too familiar with the change so I don't see an easy way to do it using the hook.
I looked at the current implementation and I think it is clean and easy to understand. It doesn't impact my use case as I want to customize querier for non-inmem blocks only.
If you feel it is easier by extending the hook then I am all good to extend it.

Thanks for response.  I'll probably leave it as-is right now, but it could surprise someone.
Maybe you could feed back in a couple of weeks how yoyr experiment is going, and we reconsider?


Documentation for many modules doesn't render

Documentation link
https://registry.terraform.io/providers/PaloAltoNetworks/panos/latest/docs/resources/panorama_zone
Describe the problem
When loading the page I don't see any documentation. This also applies to panos_zone which has some details but no Import Names.

Seeing similar issues on https://registry.terraform.io/providers/PaloAltoNetworks/panos/latest/docs/resources/panorama_ipsec_crypto_profile

This will be fixed in the next release.


chore: remove unnecessary config step for user

The fewer steps, the better for the user.

Awesome! Thanks @khuezy. Release in v0.1.0.

Thanks @fwang! Looking forward to the v1.0.0 release!


Enhancement: Connection parser – automatically parses connection string from info, can read passwords from either env or file

Enhancement: Connection Parser
Hey there!
I had some time to contribute an enhancement this week. It's related to issue #150, which I raised last week about reading passwords from secure files. I went a step further and implemented an extension of the config package that automatically parses connection strings for each database using its components - user, password, hostname, etc. While doing this, it gives the option to read passwords from either an environment variable or a file by prefixing the password with either "ENV:" or "FILE:". The connection data is stored in a struct called ConnConfig (see config/conn.go), which is included in each CrudStub under the field name "Connection".
Below is the implementation of ConnConfig:
// largely based on go-sql-driver's ConnConfig struct
// 	see: https://godoc.org/go-sql-driver/mysql#ConnConfig
// also based on lib/pq's connection string parameters
// 	see: https://godoc.org/github.com/lib/pq
type ConnConfig struct {
	User		string	`json:"user" yaml:"user"`
	// Auth provides source of password, either env or file
	Auth		string	`json:"auth" yaml:"auth"`
	DBName		string	`json:"dbName" yaml:"dbName"`
	// Protocol ex: "tcp"
	Protocol	string	`json:"protocol" yaml:"protocol"`
	Host		string	`json:"host" yaml:"host"`
	Port		string	`json:"port" yaml:"port"`
	SSLMode		sslMode	`json:"sslMode" yaml:"sslMode"`
	SSLCert		string	`json:"sslCert,omitempty" yaml:"sslCert,omitempty"`
	SSLKey		string	`json:"sslKey,omitempty" yaml:"sslKey,omitempty"`
	SSLRootCert	string	`json:"sslRootCert,omitempty" yaml:"sslRootCert,omitempty"`
	Params		map[string]string `json:"params,omitempty" yaml:"params,omitempty"`
}//-- end SQLConnConfig struct

Note: sslMode is based on lib/pq's sslMode param, which has four possible values: "require" , "disable", "verify-ca", and "verify-full". In databases with only true or false options for ssl, disable equals false and all other settings equal true.
The file config/parseconn.go contains functions to parse connection strings for all three of the supported databases from a ConnConfig struct.
The file config/generate.go has been modified to generate a ConnConfig struct from a survey and save it to the input struct during space-cloud init. This makes use of the method ConnConfig.FromSurvey(), which is located in the file config/conn.go.
To ensure backward compatibility, the original "Conn" field is still present in each CrudStub; if it is non-empty, space-cloud run defaults to using it instead of parsing a connection string.
Specifying how to get a password
The ConnConfig struct, as shown above, contains a field 'Auth", which is where a user can specify how to get a password. There are three possible options for this: a plain string, a file, or an environment variable. These are specified by the prefixes "STRING", "FILE", and "ENV". Below is an example of each:
connection:
...
    auth: STRING:my_password
    auth: FILE:/run/secrets/my_password.txt
    auth: ENV:MY_PASSWORD
...

Additional Changes:
The following changes were made to support this enhancement:

Each of the structs in config/config.go has been modified to make use of "omitempty" field tag where appropriate. This prevents the yaml parser from writing the fields to a file if they are empty.
The func writeConfig in config/generate.go has been rewritten to use a yaml encoder instead of an html template to write to the config file. Because of the previous change, the func does not write empty fields to the file, producing output identical to the output from the template.
The template in config/template.go has been rewritten as a plain yaml string instead of a template. The func writeConfig uses yaml.Decoder.Decode() to generate a default Project struct from this template before adding the user input and writing to a file.

Hey buddy. Thats a lot of work i see here! Thanks a ton. I'll review this tomorrow or day after. To start with, can you squash all commits into one?

Hey @ollykel. I'm sorry i haven't been able to go through your code yet. Will clear this in a couple of weeks.

It's ok, it's a big commit haha. I've been busy this week too, I'll finish squashing the commits today.
Should I reopen the pull request on v0.8.0, btw? I didn't see that last time.

@ollykel yes. Change the base to v0.8.0.

Its too big a request. We should break this into smaller chunks. Closing it for now


plumbing/format/idxfile: add new Index and MemoryIndex

Closes #893
This PR adds the Index and EntryIter interfaces to idxfile and renames Idxfile to MemoryIndex as described in the design document.
The new MemoryIndex now works like the JGit index reader, it loads everything as byte slices. The only change I did that diverges from the JGit implementation is adding a mapping between the fanout index and the slice index to drastically reduce the memory usage.
In JGit they use an array of 256 byte arrays for the name, offset and crc32 tables. This is such an overkill when you don't have a gigantic packfile that uses all slots of the array. By doing this, we only use as much memory as we need.
According to the benchmarks, the speed remains the same decoding the idx file, and the memory consumption is slightly increased. Access to the entries should be way faster now due to how the accesses are done.
Benchmarks have been added for all operations on idxfiles.
Warning
This will leave the repo in a broken state (which is why it's being PR'd to a feature branch), since previously there was a packfile.Index that was writable and now there is not. This is a work in progress that will probably need some of the work being done by @jfontan right now to be fully working.
Everything inside idxfile package is expected to be working correctly, though.
Caveats
As mentioned in #893, some way of retrieving hashes by offset may be needed for the packfile decoder. After some discussion with @jfontan we agreed that it's better to wait until all the pieces are ready to see how they play together before adding such a change. That is only needed when the source is not seekable, so another solution might be to just build a map from offset to hash for those cases.

If we are using the Decoder to fill a MemoryIndex struct, the Index interface is useless in my opinion.
We should use the Decoder inside the implementation, and depending on that implementation, use the decoder in different ways (loading everything at the beginning, loading each part on demand, and so on).
So, instead of doing that, we should refactor the decoder to be able to obtain the data, instead of sending a struct to fill it.
If that doesn't make sense, Index interface should be deleted and MemoryIndex should be Index. Doing that, the implementation of new Indexes will be way more difficult.
We can go even further; Instead of having a Decoder, we can have some decode helpers. Using the helpers (basically parsers) we can obtain the data on idx file in different ways, depending on our use case or index implementation.
WDYT?

Yeah, right now Index is kind of useless. What is the use case of having other index implementations? Because if we can't think of any case for implementing other indexes maybe there is no point in having that interface right now and we could just have the MemoryIndex be the index.
If there is such a case and we keep the Index interface I think, as you say, we should refactor the decoder so it can be used by any other index instead of passing the MemoryIndex struct. What I don't know is what shape the new Decoder or helpers should have, because the decoding right now is very coupled with how the MemoryIndex does things (fanout and fanout mapping).


Update domains.json

Added vote.fndz.io
Hi! What is your PR about?

[ ] Add or edit a skin
[X] Add a custom domain for your space
[ ] Add an alias to migrate your space

Closing this PR for now, feel free to open it again once you've done the changes.


Bad results when testing on real world images

Hi, thanks for sharing this work, looks great!
I was impressed by the initial sample test results until I started testing on real world images.
Please see the attached input + output, I've tested on many real world cases where the SR just fails and image quality of original image looks superior although it has 15% the size.
input - "input-sd-test.jpg" - 1200 × 819 resolution 0.2MB
Output1 - "Output_x2_k1_drunet_color.png" - 1200x818 resolution 1.6MB
Output2 - "Output_x2_k0_drunet_color.png" - 1200x818 resolution 1.5MB
Please advise.
input

Output1

Output2

Use https://github.com/cszn/DPIR/blob/master/main_dpir_sisr_real_applications.py for real applications.


Running into "Sign in to confirm you're not a bot" on deployed version

Describe the bug
The downloads worked properly for ~20-30, but after that I am getting hit with this captcha. This is deployed on AWS Lambda, which should (in theory) have a rotating IP address. Is there anything you recommend as the user agent/proxy setup to get past this issue?
Debug File
Environment

@distube/ytdl-core version: 4.14.3
Node.js version: 20
Operating system: AWS Lambda

Same ive deployed my backend node app to render and its showing it via the frontend

Estou com o mesmo problema

yep, the same, and this is the first time i see this one. Suppose someone at youtube taking a look into this repo regularly, to trump all the ways that ytdl tries to get bypass...

Alguém tem a case adaptada? Se tiverem por favor envie para ajudar

works locally, does hit this bot thing on Heroku deployment.

funciona localmente, atinge essa coisa de bot na implantação do Heroku.

Estou usando na aws e estou com esse erro

The guys at cobalt created an OAuth Token based solution around this. Doesn't seem like a good permanent solution but it might help in the short-term.
https://github.com/imputnet/cobalt/issues/551

is there any temporary solution to quickly resolve this issue?

At the moment I think the most promising is to use OAuth tokens, but I don't know how to implement that in ytdl-core.
(Text translated using Deepl translation.)

is there any temporary solution to quickly resolve this issue?

The problem was resolved in #70 by using a proxy implementation.

Duplicated of #21


pull request for e2e test at 11/21/2016 15:56:05

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Theme Master Update infinite loop

I've got into a Theme Master Update infinite loop that I can not get rid off.
But I noticed this message.
ThemeMaster: line 424: [: missing `]'
https://github.com/christianhaitian/arkos/assets/9194198/24cc367f-e69b-4caa-b8f1-42f001270510
Maybe you can fix this.

Post the issue on the thememaster github here: https://github.com/JohnIrvine1433/ThemeMaster/issues

Done. https://github.com/JohnIrvine1433/ThemeMaster/issues/24
Thanks

New version released with a fix. Thanks


General questions regarding lidar data

Hello, I have a couple of doubts regarding how I can train the model on my own data.
1 - Do I need to use specifically one of the Velodyne lidar sensors to gather data?
2 - Does it automatically produces a calibration.txt file of every frame when saved? Essentially what is calibration for?
3 - How can I label my point cloud data?
Would this software work?
https://scale.com/3d-sensor-fusion
It would be great if you could help me clarifying these questions.
Thank you.

@nicholasguimaraes
Hello,
Have you solved the third problem?  Did you use that software you mentioned?  Thanks!


Doc test tool failure: xt/aspell.t

Problem or new feature
While testing the current doc master branch on a Debian 11 host I get test failures from xt/aspell.t:
make xtest >& xtest.log

The full output of the log:
`if [ "" != "" ]; then prove -j  -e raku t xt; else prove -e raku t xt; fi
t/00-meta.t ................ ok
t/02-pod-valid.t ........... ok
t/03-tests-valid.t ......... ok
t/04-pod6.t ................ ok
t/05-tabs.t ................ ok
xt/01-raku-version.t ....... ok
A worker in a parallel iteration (hyper or race) initiated here:
  in block <unit> at xt/aspell.t line 68

Died at:
    A react block:
      in block  at xt/aspell.t line 102

    Died because of the exception:
        Malformed UTF-8 near bytes 74 e2 20

Failed to write 299 bytes to filehandle: Broken pipe
  in block <unit> at doc/Type/CurrentThreadScheduler.pod6 line 15

Failed to write 4541 bytes to filehandle: Broken pipe
  in block <unit> at doc/Type/Rational.pod6 line 172

Failed to write 6449 bytes to filehandle: Broken pipe
  in block <unit> at doc/Language/setbagmix.pod6 line 235

xt/aspell.t ................
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 846/846 subtests
xt/braces.t ................ ok
xt/camelia-invocations.t ... ok
xt/check-signatures.t ...... skipped: To run check-signatures, please specify the path to the Rakudo git checkout with the RAKUDO_SRC environment variable
xt/double-dots.t ........... ok
xt/duplicates.t ............ ok
xt/examples-compilation.t .. ok
xt/final-newline.t ......... ok
xt/headings.t .............. ok
xt/lib-test-files.t ........ ok
xt/links-not-links.t ....... ok
xt/perl-nbsp.t ............. ok
xt/return-type.t ........... ok
xt/routine-definitions.t ... ok
xt/search-categories.t ..... ok
xt/space-after-comma.t ..... ok
xt/trailing-whitespace.t ... ok
xt/type-graph.t ............ ok
xt/vim-mode.t .............. ok
xt/word-variants.t ......... ok
xt/words.t ................. ok

Test Summary Report
-------------------
xt/aspell.t              (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 846 tests but ran 0.
Files=27, Tests=14383, 772 wallclock secs ( 2.26 usr  0.15 sys + 1392.27 cusr 86.56 csys = 1481.24 CPU)
Result: FAIL
make: *** [Makefile:68: xtest] Error 1

Note Aspell is locally installed from source to its current version: 0.68.0.
Suggestions
This occurs on all my Debian hosts running versions 10 or 11 (Debian is the the only OS I use except Windows which does not have Raku installed). I plan to try the same test on an Ubuntu host.
UPDATE
I get the same results with Ubuntu 22.

For the version that is failing; please be specific about the OS version & the aspell version, and if what language packs you have installed for aspell

Failing on:

Debian 11, Aspell 0.68.0, "en"
Ubuntu 22, Aspell 0.68.0, "en"

Do you need to be running all of xtest? do you get the same issue running e.g. 'xt/aspell.t doc/Type/CurrentThreadScheduler.pod6'? do you need to be running against more than one file?

Failed to write 299 bytes to filehandle: Broken pipe sounds like the aspell process was killed and you can't write to its stdin.

please experiment with the env var RAKUDO_MAX_THREADS (might need at least 2)

Do you need to be running all of xtest?

I get the same problem running classtut.po6 only on the master branch.

do you get the same issue running e.g. 'xt/aspell.t

I'll check again

so you need to be running against more than one file?

If you run aspell.t multiple times does it fail on the same file each time?

what version of rakudo are you using?

2022.12

experiment with RAKUDO_MAX_THREADS

WILCO

also what is raku -e 'say $*SCHEDULER.max_threads'

I get the same: 64

I still haven't heard of anyone else successfully running on Linux. Most devs running doc tests seem to be Mac ppl.

I've repeatedly run the xt/aspell.t test on Linux (including just now, to make sure it still works).  I ran into some trouble initially, but added instructions in #3597 and haven't had any issues since then.

I don't have a Debian box with Rakudo installed at the moment, but I can run xt/aspell.t on the following two distros (note that I tested two Rakudo versions on Void):
Distro: GNU Guix 1.3
Rakudo: v2022.07
Aspell: International Ispell Version 3.1.20 (but really Aspell 0.60.8)
Dictionary: en 2020.12.07
Distro: Void Linux 5.19.17
Rakudo: v2022.07, v2022.12-1-gd52342eb0
Aspell:  International Ispell Version 3.1.20 (but really Aspell 0.60.8)
Dictionary: en 2020.12.07

Thanks, David. Are you running those as VMs on your own host? Or a docker somehow that I could replicate?
What I'm trying to do is use the identical setup as you (or anyone else who is having success). It's easiest for me to use a distro available on Digital Ocean, Linode, or Vultr. Generally that is Debian, Ubuntu, or Centos, or similar.

Thanks, David.

Glad I could help (but it's "Daniel", actually).

Are you running those as VMs on your own host?

That's on my own hosts (Guix on a desktop, void on a laptop).

It's easiest for me to use a distro available on Digital Ocean, Linode, or Vultr.

I have a debian server that I can test on later today or tomorrow – it just doesn't have raku/the docs site installed at the moment.

(but it's "Daniel")

Daniel, I know that, sorry, please chalk it up to a "senior moment," I just recently talked to a "David."
And thanks for your help. I need to find a good, repeatable solution for testing docs so  @[Coke] won't ban me! I'm looking for a Mac Mini but having a budget crunch a the moment.
I do have a new local host I was going to install Debian on. Are Guix or Void good choices for a long-time Debian user? I'll have to look at them. I have been considering using FreeBSD but that's a bit of a learning curve.

@tbrowder: I bought a M1 Mac Mini two years ago to do the rakudo.org precomp releases. I sold it recently, because there are now online services that offer access to M1 Mac machines. I have applied and been granted a free MacStadium account for my work on Rakudo. I believe you can to.

@patrickbkr, thanks, great idea!

I'm willing to provide access to a Debian host on one of the cloud services iif someone will use it to find out why I'm having problems with aspell.t on Debian. Just let me know here or on IRC #raku.

I just ran the xspell test on my debian buster server (on linode, incidentally) without any problems (after updating the dictionary as described in the aspell test file – that's the bit of setup instructions I mentioned upthread).
Here's the info:  (I know it's an old Rakudo version; let me know if you'd like me to update – it's just a slow server, so that'll take a bit).
Distro: Debian Buster
Rakudo: v2021.08
Aspell: International Ispell Version 3.1.20 (but really Aspell 0.60.8)
Dictionary: en 2020.12.07

Are Guix or Void good choices for a long-time Debian user?

Probably not.  Both are rolling release distros (and Guix in particular still has some growing pains to work through – though it's super promising in the long term).
As an aside, I'm a bit surprised to hear you say that you Raku developers seem more likely to use Macs.  I haven't done any formal survey or anything, but my sense has been that Linux is 60%+.  But maybe that's wildly wrong…

Daniel, I followed those instructions. I also tried to search for bad utf-8 chars in all the pertinent input files using "grep -axv '.*'" (from reddit or stack overflow) and thought I found some bad chars in some of the pws files, but that changed later. I'm kinda pulling my hair out here because I can't seem to eliminate any possibilities to zero in on the problem (remember the definition of insanity), especially when I seem to be the only person having the problem.

Hmm, I reran with an up-to-date version of Rakudo (v2022.12) and it still ran fine on the debian buster linode server, so I really have no idea what the issue could be.
I'll spin up a totally new linode server tomorrow and check with a clean install and see if I can reproduce the issue – it really seems like we should be able to get to the bottom of this.

Yup, that seems like a reasonable ask, will update tonight if no one else gets to it.

We used Lock in 3 places in the test suite, but only aspell has the need to write to the process that was launched, so it makes sense you'd only experience issues there.

I tried with the Lock::Async change and my runs still fail.
I see that the Makefile provides options for using a dockerfile. I tried make docker-xtest but no go. How can I get a dockerfile to try that?

I've asked @JJ to tell repo users how to use the Makefile Docker targets which may help isolate my problem during aspell testing.

I am unable to duplicate your error on the debian box you gave me access to. I followed the following directions:
sudo apt-get install build-essential git libssl-dev
git clone https://github.com/rakudo/rakudo.git
cd rakudo
perl Configure.pl --gen-moar --gen-nqp -- backends=moar
make
make install
cd
export PATH=$HOME/rakudo/install/bin:$HOME/rakudo/install/share/perl6/site/bin:$PATH
git clone https://github.com/ugexe/zef.git
cd zef
raku -I. bin/zef install .
cd
git clone https://github.com/raku/doc.git
cd doc
sudo apt-get install aspell
xt/aspell.t doc/Language/*tut*.pod6

(zef wasn't needed)
$ xt/aspell.t doc/Language/*tut*.pod6
1..6
@(#) International Ispell Version 3.1.20 (but really Aspell 0.60.8)
not ok 1 - doc/Language/classtut.pod6 (text) has 2 spelling errors
# Failed test 'doc/Language/classtut.pod6 (text) has 2 spelling errors'
# at xt/aspell.t line 132
#     Errors:
#     =======
#     runtime
#     runtime
ok 2 - doc/Language/classtut.pod6 (code) has 0 spelling errors
ok 3 - doc/Language/grammar_tutorial.pod6 (text) has 0 spelling errors
ok 4 - doc/Language/grammar_tutorial.pod6 (code) has 0 spelling errors
ok 5 - doc/Language/optut.pod6 (text) has 0 spelling errors
ok 6 - doc/Language/optut.pod6 (code) has 0 spelling errors
# You failed 1 test of 6

This is just a regular test failure due to the aspell library not being up to date (which it isn't)

@tbrowder reports that on his box he still gets the error; I no longer have rates on the box so I cannot see what his environment is. Mine is below (on a fresh login, doesn't include the PATH updates)
$ env
SHELL=/bin/bash
PWD=/home/coke
LOGNAME=coke
XDG_SESSION_TYPE=tty
MOTD_SHOWN=pam
HOME=/home/coke
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
LC_TERMINAL=iTerm2
SSH_CONNECTION=<IP_ADDRESS> 55286 <IP_ADDRESS> 22
XDG_SESSION_CLASS=user
TERM=xterm-256color
USER=coke
LC_TERMINAL_VERSION=3.4.18
SHLVL=1
XDG_SESSION_ID=115
XDG_RUNTIME_DIR=/run/user/1001
SSH_CLIENT=<IP_ADDRESS> 55286 22
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
SSH_TTY=/dev/pts/0
_=/usr/bin/env

got sudo back; seems to work fine if I run it as @tbrowder
$ sudo su - tbrowde
$ export PATH=$HOME/rakudo/install/bin:$PATH
$ echo $PATH
/home/tbrowde/rakudo/install/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
$ ../rakudo/install/bin/rakudo xt/aspell.t doc/Language/*tut*
1..6
@(#) International Ispell Version 3.1.20 (but really Aspell 0.60.8)
ok 1 - doc/Language/classtut.pod6 (code) has 0 spelling errors
not ok 2 - doc/Language/classtut.pod6 (text) has 2 spelling errors
# Failed test 'doc/Language/classtut.pod6 (text) has 2 spelling errors'
# at xt/aspell.t line 132
#     Errors:
#     =======
#     runtime
#     runtime
ok 3 - doc/Language/grammar_tutorial.pod6 (text) has 0 spelling errors
ok 4 - doc/Language/grammar_tutorial.pod6 (code) has 0 spelling errors
ok 5 - doc/Language/optut.pod6 (code) has 0 spelling errors
ok 6 - doc/Language/optut.pod6 (text) has 0 spelling errors
# You failed 1 test of 6

Anyone have any ideas about what elsoe @tbrowder is doing that we can compare?

@tbrowder - can you post the env you see when you login directly so there's no potential crosstalk with my env?

On Sun, Jan 1, 2023 at 16:31 Will Coleda @.***> wrote:

@tbrowder https://github.com/tbrowder - can you post the env you see
when you login directly so there's no potential crosstalk with my env?

Yes, I will do that today.
-Tom

@tbrowder  - are you still having this issue? I'm not sure what else I can do since I cannot duplicate it using your own setup.

Sorry I've been out of touch lately. I will try one more time on a new, local host.

Without further info from @tbrowder, closing this ticket - I had it working on his OS setup.
Regards.


Reference code for NLNN with multi-head attention?

The graph networks paper points out the ability to express transformer like archs within graph net framework. Is there some example code of making multi-head attention work within this framework or the graph_nets library? I'm getting up on the segment_fn semantics.

Hey,
A transformer is effectively a Graph Attention Network applied to a fully connected graph.
In order to use Jraph to accomplish such a model you would:

create a GraphsTuple with a directed edge between every two nodes, this would be 2 * n^2 edges. This is accomplished by using the sender and receiver attributes.
Use the GAT model from jraph:

import jraph
import haiku as hk

def attention_query_fn(feats: jnp.ndarray) -> jnp.ndarray:
  """Attention Query Function."""
  net = hk.Sequential(
      [hk.Linear(128), jax.nn.relu,
       hk.Linear(128)])
  return net(feats)

def attention_logit_fn(sender_features, receiver_features, edge_features):
  """Attention Logit Function."""
  del edge_features  # transformers do not use explicit edge features
  return jnp.dot(sender_features, receiver_features)

transformer_like = jraph.GAT(attention_query_fn, attention_logit_fn)

Multi-head attention is unlikely to be as performant using a dedicated transformer library since we use a lot of gathers and scatters, but if you have a graph you want to apply multi head attention to, then you can use this.

Hey,
A transformer is effectively a Graph Attention Network applied to a fully connected graph.
In order to use Jraph to accomplish such a model you would:

create a GraphsTuple with a directed edge between every two nodes, this would be 2 * n^2 edges. This is accomplished by using the sender and receiver attributes.
Use the GAT model from jraph:

import jraph
import haiku as hk

def attention_query_fn(feats: jnp.ndarray) -> jnp.ndarray:
  """Attention Query Function."""
  net = hk.Sequential(
      [hk.Linear(128), jax.nn.relu,
       hk.Linear(128)])
  return net(feats)

def attention_logit_fn(sender_features, receiver_features, edge_features):
  """Attention Logit Function."""
  del edge_features  # transformers do not use explicit edge features
  return jnp.dot(sender_features, receiver_features)

transformer_like = jraph.GAT(attention_query_fn, attention_logit_fn)

Multi-head attention is unlikely to be as performant using a dedicated transformer library since we use a lot of gathers and scatters, but if you have a graph you want to apply multi head attention to, then you can use this.

I figured out what was confusing me. The example above is treating sender/receiver_features as you would input node features in a transformer, as simply taking the dot product of (N, d), (N, d) gives you (N, N) matrix which represents the fully connected adjacency matrix in a transformer. But here the first dimensions of sender/receiver are already edges, so you actually just want to do something like (sender_features * receiver_features).sum(-1).

I figured out what was confusing me. The example above is treating sender/receiver_features as you would input node features in a transformer, as simply taking the dot product of (N, d), (N, d) gives you (N, N) matrix which represents the fully connected adjacency matrix in a transformer. But here the first dimensions of sender/receiver are already edges, so you actually just want to do something like (sender_features * receiver_features).sum(-1).

This makes working with multihead attn surprisingly easy (I'm not sure re: performance but I may work on a PR with docs/examples of how to do this and maybe compare w/ dgl)

This makes working with multihead attn surprisingly easy (I'm not sure re: performance but I may work on a PR with docs/examples of how to do this and maybe compare w/ dgl)

Thanks! I've updated the example.
You could also decoder the attention_logit_fn with @jax.vmap and then keep the np.dot product. The vmap simplifies some logic but can make things a little less transparent :)
I'm glad you think this makes it easier! We would be very happy if you'd like to contribute some docs/examples especially if you think this way of working is good !

Thanks! I've updated the example.
You could also decoder the attention_logit_fn with @jax.vmap and then keep the np.dot product. The vmap simplifies some logic but can make things a little less transparent :)
I'm glad you think this makes it easier! We would be very happy if you'd like to contribute some docs/examples especially if you think this way of working is good !

Hey, quick update--I really enjoy the API and have been able to make some interesting architectures. However:

Multi-head attention is unlikely to be as performant using a dedicated transformer library

Unfortunately I think you're right, but I find even simpler message passing to be slow. Even with sparse connections for molecular graphs, preliminary experiments building on your examples (both haiku and flax based) show jax/jraph being orders of magnitude slower than expected (compared to pytorch/dgl impl. of transformers, GNN variants). GPU util sits at 0% most of the time almost regardless of batch size. I'm assuming this means the bottleneck isn't in any matrix multiplications but almost totally in the graph datastructure wrangling. Am I missing something obvious?

Hey, quick update--I really enjoy the API and have been able to make some interesting architectures. However:

Multi-head attention is unlikely to be as performant using a dedicated transformer library

Unfortunately I think you're right, but I find even simpler message passing to be slow. Even with sparse connections for molecular graphs, preliminary experiments building on your examples (both haiku and flax based) show jax/jraph being orders of magnitude slower than expected (compared to pytorch/dgl impl. of transformers, GNN variants). GPU util sits at 0% most of the time almost regardless of batch size. I'm assuming this means the bottleneck isn't in any matrix multiplications but almost totally in the graph datastructure wrangling. Am I missing something obvious?

Hey,
That is a little concerning. Do you have a code gist or something I could look at?
I think it could be a few different things:

is the graph net accidentally not jitted?
Is there are lot of extra jitting going on because of a dynamic shape slipping in? This will cause recompilation on every call of the function.
Are the many jitted programs? (This would lead to going to and from the host a lot, so all the time would be spent in communication overhead)
Is Jax picking up your GPU? You can find this by checking jax.local_devices()

I would hope that is not in a fully jit compiled function, since that would imply sparse operations are slower in jax that pytorch.

Hey,
That is a little concerning. Do you have a code gist or something I could look at?
I think it could be a few different things:

is the graph net accidentally not jitted?
Is there are lot of extra jitting going on because of a dynamic shape slipping in? This will cause recompilation on every call of the function.
Are the many jitted programs? (This would lead to going to and from the host a lot, so all the time would be spent in communication overhead)
Is Jax picking up your GPU? You can find this by checking jax.local_devices()

I would hope that is not in a fully jit compiled function, since that would imply sparse operations are slower in jax that pytorch.

Did a little bit more logging, now I see that for your minmal examples, the bottleneck is the data_utils.DataReader which for me takes 4.5s to get a batch of 128 graphs. Forward/backward pass are fine (backward .2s, so if dataloading took 0s, it'd be similar to my torch impl.). For the stuff I've done, I will do some more profiling because I'm using a lot more operations that look like: dense1(nodes)[receivers] * dense2(nodes)[senders] + dense3(globals_)[global_to_edge_broadcast_idx] Maybe this kind of ad-hoc indexing is not able to be optimized well.

Did a little bit more logging, now I see that for your minmal examples, the bottleneck is the data_utils.DataReader which for me takes 4.5s to get a batch of 128 graphs. Forward/backward pass are fine (backward .2s, so if dataloading took 0s, it'd be similar to my torch impl.). For the stuff I've done, I will do some more profiling because I'm using a lot more operations that look like: dense1(nodes)[receivers] * dense2(nodes)[senders] + dense3(globals_)[global_to_edge_broadcast_idx] Maybe this kind of ad-hoc indexing is not able to be optimized well.

Hey,
I would recommend using either the tensorflow dataloader or the pytorch data loader, with a wrapper that converts DGL (or equivalent) to GraphsTuple.
I think, if you are using GPUS (not TPUs) that sort of indexing should be fine. My understanding is that bottlenecks with indexing are more due to hardware rather than XLA compilation (although there may be custom cuda kernels that can speed things up in special cases). Let me know if you find something to the opposite.

Hey,
I would recommend using either the tensorflow dataloader or the pytorch data loader, with a wrapper that converts DGL (or equivalent) to GraphsTuple.
I think, if you are using GPUS (not TPUs) that sort of indexing should be fine. My understanding is that bottlenecks with indexing are more due to hardware rather than XLA compilation (although there may be custom cuda kernels that can speed things up in special cases). Let me know if you find something to the opposite.

Hey,
I would recommend using either the tensorflow dataloader or the pytorch data loader, with a wrapper that converts DGL (or equivalent) to GraphsTuple.
I think, if you are using GPUS (not TPUs) that sort of indexing should be fine. My understanding is that bottlenecks with indexing are more due to hardware rather than XLA compilation (although there may be custom cuda kernels that can speed things up in special cases). Let me know if you find something to the opposite.

Thank you for this suggestion. Because also, similar to @sooheon, DataReader takes about 4 or more seconds, and I have been working on optimizing it throughout the week. Maybe I should move to the TensorFlow data loader or the PyTorch data loader as you suggested.


Enable Sonarcloud

Coverage remained the same at 96.518% when pulling f632a587689874aee202755752cae35bce12dccf on sonarcloud into 26cbaaa515105f399d9c2b6d2d405d1ab47341ac on master.

Coverage remained the same at 96.518% when pulling f632a587689874aee202755752cae35bce12dccf on sonarcloud into 26cbaaa515105f399d9c2b6d2d405d1ab47341ac on master.


Kafka_exporter

https://github.com/danielqsj/kafka_exporter/blob/refs%2Fheads%2Fmaster/charts%2Fkafka-exporter%2Ftemplates%2Fsecret.yaml

| server.tls.mutual-auth-enabled | false          | Enable TLS client mutual authentication                                                                                |
| server.tls.ca-file             |                | The certificate authority file for the web server                                                                                |
| server.tls.cert-file           |                | The certificate file for the web server                                                                                |

https://github.com/danielqsj/kafka_exporter/blob/refs%2Fheads%2Fmaster/charts%2Fkafka-exporter%2Ftemplates%2FNOTES.txt


[bootstrap] Emit debug info when compiling a debug stage1 swiftpm.

While debugging a crash in the stage 1 bootstrap swiftpm, I noticed that the stage 1 bootstrap is not compiled with debug info even when it is compiled in debug mode. This PR adds such support.

@aciidb0mb3r Just a small fix I found.

@swift-ci smoke test

Thanks!


Config commit message for file operations

Fix #278

Something wrong in my commits. Will create a new one later.


[VTOM-121] [GPT4V] Add Frame Indices, TImestamps, and Their Respective Dictionaries to GT

null
VTOM-121

Also add response markdown string


Update GitHub Template

Update GitHub Template
Related Issue
Closes #71
Description
Updated the following templates:

Bug Report
Feature Request
Pull Request

Discussed and approved in meetings. Templates are agreed upon and valid. Thanks for completing this!


fix position

I don't think meshes failing to load should be ignored. We should make sure a placeholder is loaded.

Is this one still valid?

was a trash pr


Add example scripts with ThreadPoolExecutor and URL patterns

Updates #355, closes #143

Coverage remained the same at 98.992% when pulling e477e0d21d2db4dacc562e9afbb33c51208ff5cf on JWCook:docs into a88d1ab3a47b31ab6101f99efa18a0aeda645b2d on reclosedev:master.


VideoChat2_it_hd_mistral(config=cfg.model) 'bert base uncased' Error

Hi I'm trying to follow this demo for testing, and encountered an issue related to Bert Tokenizer , below is the the error message, can you help me to fix it ? thanks !

Hi! Please set local_files_only=False if you do not download the weights.

Hi, we will close this issue.
Feel free to contact us if you have other questions.


mv do not cp/tar

Instead of unpacking the runner implementation on VM start, try unpacking during the installation and to simply move the unpacked installation to another directory before use. This would still ensure unique naming of the path, but could be lead to quicker initialisations of the runners.

When moving, the second runner will not be able to access the (moved) installation diretory. Thata means that we should be re-creating the microVM at each loop in runner.sh instead of creating them once and for all in orchestrator.sh.


Decode PubkeyScript causes DataIntegrityError

Hi @dr-orlovsky,
I tried to decode a PubkeyScript using lightning_deserialize from message shutdown channel by clightning, but it occurred an error:
Err(DataIntegrityError("parse failed: data not consumed entirely when explicitly deserializing"))

According to the BOLT-2 spec, we can receive one of the following scriptpubkey:

OP_0 20 20-bytes (version 0 pay to witness pubkey hash), OR
OP_0 32 32-bytes (version 0 pay to witness script hash), OR
if (and only if) option_shutdown_anysegwit is negotiated:
OP_1 through OP_16 inclusive, followed by a single push of 2 to 40 bytes (witness program versions 1 through 16)

The Script::lightning_deserialize cannot distinguish which type of script is being passed.
You can test this by adding the bellow test in bitcoin.rs:
#[cfg(test)]
mod test {
    use bitcoin::Script;
    use crate::LightningDecode;

    #[test]
    fn real_clightning_scriptpubkey() {
        // Real scriptpubkey sent by clightning
        let msg_recv = [
            0, 22, 0, 20, 42, 238, 172, 27, 222, 161, 61,
            181, 251, 208, 97, 79, 71, 255, 98, 8, 213, 205, 114, 94,
        ];
        let message = Script::lightning_deserialize(&msg_recv);
        println!("{:#?}", message);
        assert_eq!(true, message.is_ok());
    }
}

Expected Result:
Ok(PubkeyScript(Script(OP_DUP OP_HASH160 OP_PUSHBYTES_20 00142aeeac1bdea13db5fbd0614f47ff6208d5cd OP_EQUALVERIFY OP_CHECKSIG)))

I bypassed the issue by deserializing each type of script according to its size.
I hope this help.
Thanks

So we need an ability to analyze the length of the received message?

So the error was completely unrelated to what was described in this issue. The real case of the problem was different: https://github.com/LNP-WG/lightning_encoding/pull/4#issuecomment-1464874585

The open_channel message (bolt2) has a field called shutdown_scriptpubkey; this field is optional. The lightning decode returns DataIntegrityError when this field is empty.
I'm not sure how to solve this issue, but the TLV fields need a way to check if vector bytes are empty. If true, "skip" the parsing process.


JavaCC 7.0.12, Java 21, try-with-resources block causes error

I have the following try-with-resources block in the Java code section of my .jj file:
try (InputStream is = new FileInputStream(fileName)) {
}
JavaCC errors out when it sees the '(' in front of the try statement, and says it's expecting a '{'.
I'm using Java 21, so try-with-resources should be recognized.

The javacc parser embeds a java 1.7 syntax and a partial java 1.8 syntax, no more. One day we'll catch up the latest java versions...
Try to put your java code (e.g. your main(...) and other glue) outside the .jj file.

That's odd, because the JavaCC plugin version 7.0.12 for Eclipse 2024-03 doesn't have the same reaction to the try-with-resources block, but processes it just fine when I run JavaCC from within Eclipse.

Well, this is really odd. I believe the story is that the JavaCC Eclipse plugin maintainer (me, when I was not also the javacc maintainer) silently fixed a few things (for my personal use) in the (embeded) javacc jar he shipped with the plugin and kept it with the same name / version.


In table "Requisition Line" field "Type" should be Enum

Hello,
Our customer needs to extend the field "Type" in "Requisition Line", this is critical.
But this field still Option, could you please make transform field to Enum?

Version: BC17 Cu03

Please add [Enum Request] prefix to the title if you requesting for a new enum or a change to the existing enum.

This is already implemented in 18.x
    field(4; Type; Enum "Requisition Line Type")
    {
        Caption = 'Type';


Support HDMI CEC control

From @mithro on April 1, 2013 4:11
Raspberry pi has support, would be good if we could duplicate it.
Copied from original issue: timvideos/HDMI2USB-jahanzeb-firmware#20

This can be done now we have the Numato Opsis board which has the CEC pin connected.
Take a look at http://libcec.pulse-eight.com


LLVM 11

Work in progress: Port AFLGo to LLVM 11.
TODO:

[x] Update build script
[x] Make afl-llvm-pass.so.cc compile
[x] Update gen_distance_fast.py
[x] Update genDistance.sh
[x] Have a look at build process as clang fails in compiling afl-gcc which isn't used anyway (afaik). Building with make clean && AFL_NO_X86=1 make works
[x] Update README.md
[x] Maybe something else
[x] Testing

Just a quick update: I haven't forgotten this pull request, I'm just busy with other stuff.

Have a look at build process as clang fails in compiling afl-gcc which isn't used anyway (afaik). Building with make clean && AFL_NO_X86=1 make works.

Hmm. That's weird. afl-gcc doesn't use the LLVM pass. Does it work with vanilla AFLGo?

Yes, the same happens with https://github.com/mirrorer/afl, however it doesn't happen with https://github.com/google/AFL. I found that the problem was patched https://github.com/google/AFL/pull/33. The Makefile in llvm_mode has the appropriate patch, but the Makefile in the root dir of AFLGo didn't have it. So this issue is now also resolved. Now, I'll have only to recompile everything from scratch to see if it works.

I did some brief tests, and it seems to work as expected :)
Before merging, it might be a good idea to create a tag llvm-4.0.0 on the current master branch. That way it's easy to navigate to the older version that uses llvm 4.0.0.

I think this should address the raised points. I fixed the error handling in a separate pull request (#91). Thanks for your feedback, @mboehme!

Awesome! Thanks @Liblor. LGTM.


Airflow set up, part 3: Fixing issue with docker compose syntax

Closes https://github.com/opentargets/issues/issues/3143.
Third and final part of updates for https://github.com/opentargets/issues/issues/3143, split for readability and easier review.
For me, on Ubuntu, docker-compose (with hyphen) works, while docker compose (with space) does not:
docker: 'compose' is not a docker command.
See 'docker --help'

I installed it in a standard way via apt (sudo apt install docker.io).
It looks like this distinction has a long history: https://github.com/docker/compose/issues/8630; and certainly there are fixes and workarounds about this.
However, I was wondering: @d0choa @ireneisdoomed, does docker-compose (with hyphen) work for you on your machines?

If yes, we can leave this option as seemingly the most compatible.
If no, then I'll amend this PR and specify this discrepancy and potential workarounds.

I have no issue using docker-compose:
❯ ls /usr/local/bin/ | grep docker
com.docker.cli
docker
docker-compose
docker-compose-v1
docker-credential-desktop
docker-credential-ecr-login
docker-credential-gcloud
docker-credential-osxkeychain
docker-diagnose
docker-image-tool.sh
docker-index
kubectl.docker

Thank you very much for the pointers @d0choa! I investigated further, and apparently the issue was that my Docker installation on Ubuntu was done years ago, before it even had official Ubuntu packages. So all this time I was using an unofficial package docker.io, which, while it kept updating the actual Docker versions, didn't port the compose command to new syntax.
I've reverted syntax back to "docker compose" and added a note in case anyone encounters this in the future.

Codecov Report

Merging #223 (528136e) into tskir-3143-2-fix-for-uid (ebc383e) will not change coverage.
The diff coverage is n/a.

@@                    Coverage Diff                    @@
##           tskir-3143-2-fix-for-uid     #223   +/-   ##
=========================================================
  Coverage                     85.64%   85.64%
=========================================================
  Files                            81       81
  Lines                          1853     1853
=========================================================
  Hits                           1587     1587
  Misses                          266      266

We can live without this warning. We can probably assume someone is using a moderately modern version of Docker.

@d0choa Here's the thing though: I did have the latest version of Docker installed and still ran into this error. It's not about the version, it's about the packages which were used to install it.
Docker only started officially providing Linux packages two years ago. So everyone who installed it on Debian/Ubuntu more than two years ago using the docker.io package (which was the recommended way to install at the time), and then kept updating to the latest version as I did, will still run into this problem.
My opinion remains that having this warning, which might apply to a sizable portion of Debian/Ubuntu users, would be helpful. Especially seeing as we do keep a platform specific warning about macOS memory issues :upside_down_face:


Added more functionality to cache and create resized images

Create and cache large sized images
Added caching and creating resized images for facebook

👍


The training data are extracted every time being compiled

When compiling the unit tests the training data is extracted but this is not necessary and is a slow process that hinders development.

Thanks for creating this issue. I noticed that problem too, and intended to write a bug report on it. In Linux, the extraction process is so fast (about 500 - 800 ms) that it is barely noticeable. On Windows, however, for some reason it takes about 5 - 7 seconds to extract the data, and I can see how that would become annoying.
I tried to fix the CMake rule that extracts the archive, but I could not find a way to make it work correctly while using only one CMakeLists.txt file in the root directory. Do you know how?
In a small simple project like this, there's a certain elegance in having all the configuration defined in a single location instead of being distributed throughout the directory hierarchy. If we have to create a new CMakeLists.txt inside the images/digits/ directory, we can do that, but I would prefer a solution with a single configuration file if that is possible.

You could check in the CMakeLists.txt if a file of the digits.zip file exists in image/digits folder, if the file does not exist extract the digits.zip file and if exists continue the compiling process.

That was what I thought too, but I haven't found the right syntax yet. Currently, the rule is:
add_custom_command(
    OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/test-1.bmp
    COMMAND ${CMAKE_COMMAND} -E tar xzf ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/digits.zip
    DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/digits.zip
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/
    COMMENT "Preparing images/digits/..."
    VERBATIM
)

It should be like this:
if(NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/images/digits/test-1.bmp")
    add_custom_command(
        OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/test-1.bmp
        COMMAND ${CMAKE_COMMAND} -E tar xzf         ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/digits.zip
        DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/digits.zip
        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/
        COMMENT "Preparing images/digits/..."
        VERBATIM
    )
else()

The "if" statement seems to be evaluated only at the time cmake is run. After running cmake, if the image/digits/ directory is erased and then restored with "git checkout," the archive will not get extracted and "make test" will generate an error. Is there a way to write the rule so that "make test" will trigger the extraction whenever needed?

I found a more clean way in the documentation to solve this bug:
add_custom_command(
    TARGET
        neural2d
    PRE_BUILD
    COMMAND
        ${CMAKE_COMMAND} -E tar xzf ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/digits.zip
    DEPENDS
        ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/digits.zip
    WORKING_DIRECTORY
        ${CMAKE_CURRENT_SOURCE_DIR}/images/digits/
    COMMENT
        "Preparing images/digits/..."
    VERBATIM
)

I don't know if this works with lower versions of CMake.

That syntax is nice, but it seems to work the same as the previous one -- it will fail to re-extract the .zip archive if the *.bmp files have been deleted.


Add renovate bot integration for advanced dependency management

Like in this project, it would be great to have automatic pull request suggestions from the Renovate Bot. It shall scan for go.mod entries and find newer versions and then suggest according change sets.

Thanks @synergiator for the suggestion, I'll keep this ticket open for tracking so that we can work on it for next release.
Other issue I could see from this is, existing cloudstack terraform provider may not work if the provider SDK is generated with latest terraform versions as it has changed few implementations without backward compatibility. So cloudstack provider will need some rework inorder to generate SDK with latest terraform version.

We have added dependabot in this PR: https://github.com/apache/cloudstack-terraform-provider/pull/70. It's purpose is same as the renovate bot. Closing this issue.


Integration tests using spectron

Fixes #398

@nevalla @jnummelin PTAL

Just wondering if there's some specific reason not to enable integration tests on Windows pipeline? Can be done in separate PR, need to remember to add an issue on that.

Didn't have time to test this on Windows. I'll create a separate issue about it.


Xu - GLP-1 RAs[GLP1]

GLP-1 RAs
General Search Terms for GLP-1RAs:
Glucagon-like peptide-1 receptor agonist
GLP-1 receptor agonist
GLP-1RA
GLP-1R agonist
Glucagon-like peptide-1 analog
GLP-1 analog
Glucagon-like peptide-1 analogue
GLP-1 analogue
Glucagon-like peptide-1 mimetics
GLP-1 mimetics
Incretin mimetics
Incretin-based therapy
Anti-hyperglycemic agent
Gut hormone modulator
Drug-Specific Search Terms:
(FDA-approved)
Exenatide
Liraglutide
Dulaglutide
Semaglutide
Lixisenatide
Albiglutide
Tirzepatide
(Not yet FDA-approved, investigational drug)
Efpeglenatide
Retatrutide

(Glucagon-like peptide-1 receptor agonist) OR (GLP-1 receptor agonist) OR (GLP-1RA) OR (GLP-1R agonist) OR (Glucagon-like peptide-1 analog) OR (GLP-1 analog) OR (Glucagon-like peptide-1 analogue) OR (GLP-1 analogue) OR (Glucagon-like peptide-1 mimetics) OR (GLP-1 mimetics) OR (Incretin mimetics) OR (Incretin-based therapy) OR (Anti-hyperglycemic agent) OR (Gut hormone modulator) OR (Exenatide) OR (Liraglutide) OR (Dulaglutide) OR (Semaglutide) OR (Lixisenatide) OR (Albiglutide) OR (Tirzepatide) OR (Efpeglenatide) OR (Retatrutide)