Encrypt Session Sticky cookie content

Do you want to request a feature or report a bug?
request a feature

What did you do?
I enabled session sticky under docker.

What did you expect to see?
The Traefik session sticky cookie content to be encrypted text.
What did you see instead?
The cookie currently shows the internal uri with internal container IP in clear text, which is a security risk if deployed in internet.
Output of traefik version: (What version of Traefik are you using?)

1.4.0

What is your environment & configuration (arguments, toml, provider, platform, ...)?
# (paste your configuration here)

If applicable, please paste the log output in debug mode (--debug switch)
(paste your output here)

I vote for this feature too. I like traefic very much but without this it is'nt possible to use it. HAProxy do it by no ecryption but by configurable cookie value for the given server, it will be enough.

Totally agree that  encryption is not a must. A unique lookup key will suffice and be efficient than handling encryption and decryption

Yes please, a unique key will do, but it would be nice to be able to configure URI/ID (for debugging)

Feature is also needed by company security policy. Is there any work planned by containous wonderful team on this subject ? I would gladly have worked on it, but my go-fu is far from enough to be of any help.

I would like this as well. Having the internal URI in a cookie sent to a client does not look good.  Setting a cookie value per server like haproxy is the best and simplest option IMHO

I'd love to not expose internal addresses to the public. Any plan on fixing this?

I hate to bump things like this, but we need this too because of a penetration test finding we are having.

Yes, please.

Are there any plans to implement this ?
@emilevauge @ldez has this been triaged yet ?
We use traefik 1.6.6 and we have vulnerability and security findings that need the cookie value encrypted / masked and not publish the actual IP address. We use the consul catalog provider.
cc @samart

Waiting for the update

Are there any updates on this ?

Cool, thanks for responding. Is there a plan on implementation?
On Tue, 4 Jun, 2019, 8:03 PM Damien Duportal<EMAIL_ADDRESS>wrote:

Hi @sriyer https://github.com/sriyer, thanks for your interest in the
project.
The issue is already triaged (check the labels) as explained in
https://docs.traefik.io/v2.0/contributing/maintainers/#labels .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containous/traefik/issues/2299?email_source=notifications&email_token=AB3FQ772WVGNU554ZPR52U3PYZ4MDA5CNFSM4EAHGLPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW4YNYI#issuecomment-498697953,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB3FQ72TRWJOQGVHVASFZ4TPYZ4MDANCNFSM4EAHGLPA
.

@dduportal looking further at the labels as you suggest, i see the priority set as P2 meaning in the future, if i'm not wrong. Would you be able to help get an answer to what the plan is on implementing this  and how soon? happy to help if need be.

There will probably need to be an added field under cookies with a shared secret or key. This value would need to be the same on all Traefik instances that need to decrypt the cookie. To reduce the risk of an operator choosing a weak key the format of the key should be enforced and docs should show how to generate an appropriate value (as an example: openssl ecparam -genkey -name prime256v1 | base64 -w 0)

Hello @sriyer, as Traefik v2.0 is currently being worked on (you can see the kanban board here: https://github.com/containous/traefik/projects/3 ), there are matters in the foundation of Traefik  being worked on that would block this issue: any attempt at planning and implementing this Sticky Session would change after 2.0. => Hence the priority/P2 which means will be worked after P1 :)
You should subscribe to this issue: as soon as someone will assign themselves here, it means discussion can start (or when the issue would appear in the v2 board).
Rest assured this will be done, but it's a matter of finishing requirements.
Thanks for your involvement!

@dduportal I'd like to take a shot at this with a simple implementation (which I think someone else pointed out here, as well): Symmetric encryption of the cookie's value (backend URL) through a user provided key (e.g.: AES).
However, it appears that all of the sticky session functionality is handled by Oxy. We could add a configuration key to Traefik for a path to the key which will then have to be passed to Oxy. If no key is provided or the key file is missing (e.g.: misconfiguration) we fall back to the plain text cookie value with a warning in the logs.
Question is that is there a ticket in V2 that could overlap or conflict with this? Is anyone else is working on this?

Hi @gheibia,
The project board does not have (yet) a card for tracking this. However the discussion will soon start around the load-balancing implementation in Traefik v2.x. Such discussion could lead to breaking changes in the core, that would make any contribution on this topic wasted. Hence the priority (again) because no PR would be accepted (not for 1.7.x has no feature request should be done on this versions, only bugfixes, and not for 2.0 because of the priority).
I hope you understand this is only a transitioning phase, and that your contribution might well be welcomed after this "LB" discussion would have taken place.
Thanks for your patience!

Thanks for the explanation. I wasn't aware that 1.7.x no longer accepts feature requests.

Cookie value per server in the load-balance group (rather than encrypted) would be really useful/important .

Hi @dduportal,
Thanks for your explanations above - I'm looking for an update on your comments from 11 Jun 2019 or directions as to the state of the discussion regarding load balancing which was blocking this issues.
Any update would be awesome. Thank you

Hi @areThereAnyUserNamesLeft , as I am not related to the project and not a maintainer, I cannot tell you more. It looks like that the aformentionned issue bout Load Balancing had been worked on and delivered in Traefik v2.x (x >= 1).
Ping @SantoDE ?

Sorry and thank you, your answer from last year seemed so involved I
thought you were a maintainer.
On Mon, 6 Apr 2020, 16:55 Damien Duportal<EMAIL_ADDRESS>wrote:

Hi @areThereAnyUserNamesLeft https://github.com/areThereAnyUserNamesLeft
, as I am not related to the project and not a maintainer, I cannot tell
you more. It looks like that the aformentionned issue bout Load Balancing
had been worked on and delivered in Traefik v2.x (x >= 1).
Ping @SantoDE https://github.com/SantoDE ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containous/traefik/issues/2299#issuecomment-609880175,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFA4WTPNRFZXJ7PUTPVB4C3RLH3OPANCNFSM4EAHGLPA
.

@areThereAnyUserNamesLeft you are right: I used to be but I changed jobs and backed off :)

Hello @SantoDE - Is there any update on this? I couldn't find it on the kanban board or in the releases.

Hello @areThereAnyUserNamesLeft,
as mentioned on the Issue, it's a P2 so "will be done". Just out of now, I can't give you a full ETA. Sorry. When I have one, we'll share it here in the issue.

@emilevauge could you look at the PR I opened to fix that ? vulcand/oxy#203

Hi @sylr,
In your PR on oxy, your changes are breaking due to the alteration of the cookie value.
Please add an option in oxy to enable/disable the hashing.
Thank you.

Hi @sylr,
In your PR on oxy, your changes are breaking due to the changes made on the cookie value.
Could you add an option in oxy to enable/disable the hashing?
I would also suggest that adding a consistent hashing load balancing algorithm (https://github.com/containous/traefik
/issues/1139) could be a great workaround. If anyone is interested to work on this, we could definitely help :)
Thanks!

I improved the PR to honour old unhashed cookies.
However I fail to understand how consistent hashing and stickiness are related.

Thanks for the modifications on your oxy PR.
We will try to take a look asap.

May I ask about any update & ETA when it can be available? One of our security audit also raised this concern about revealing private IPs.

May I ask about any update & ETA when it can be available? One of our security audit also raised this concern about revealing private IPs.

Is this issue still planned for 2.5? Are there any news or known workarounds? Thx in advance.

Hey again. Is ther any chance that we can get this fixed. Can I try to work on this? (I'd doit if somebody supports me.)

@szwoelf The Oxy library has an open PR about hashing the IP address (which is likely a sufficient resolution for most people). You can follow that PR for updates. I don't know the release timeline on 2.5 but I would assume this is not going to make it.

Thx for the reply. Then let's observe it there.

Hi, anyone knows if the issue is actually fixed or not?

Hi, anyone knows if the issue is actually fixed or not?

It's fixed in the sense that it will be part of the next major Traefik release. Support for this has been merged into the master branch.

Except that is the next minor release, not major :-)

@SantoDE but when?

As traefik has been released - but I cannot find the documentation - what (apart from upgrading to traefik 2.4.9) do I have to configure (in kubernetes) to force the sticky cookie to be hashed / encrypted. Thx in advance. I looked at the code of @tomMoulard ... but I didn't find options. Thx so much in advance.

minor not patch :wink: (major.minor.path)

Ooops.


How to execute packageExe normally on 1.2.0 stable

'packageExe'…

> Task :wrapper

BUILD SUCCESSFUL in 213ms
1 actionable task: 1 executed
> Task :checkRuntime UP-TO-DATE
> Task :compileKotlin UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :createRuntimeImage UP-TO-DATE
> Task :jar UP-TO-DATE
> Task :inspectClassesForKotlinIC UP-TO-DATE
> Task :prepareAppResources NO-SOURCE
> Task :unpackDefaultComposeDesktopJvmApplicationResources UP-TO-DATE
> Task :downloadWix SKIPPED
> Task :unzipWix UP-TO-DATE
> Task :packageExe FAILED
8 actionable tasks: 1 executed, 7 up-to-date

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':packageExe' (type 'AbstractJPackageTask').
  - In plugin 'org.jetbrains.compose.ComposeCompilerKotlinSupportPlugin' type 'org.jetbrains.compose.desktop.application.tasks.AbstractJPackageTask' property 'runtimeImage' specifies directory '...\build\compose\tmp\main\runtime' which doesn't exist.

    Reason: An input file was expected to be present but it doesn't exist.

    Possible solutions:
      1. Make sure the directory exists before the task is called.
      2. Make sure that the task which produces the directory is declared as an input.

    Please refer to https://docs.gradle.org/7.4.2/userguide/validation_problems.html#input_file_does_not_exist for more details about this problem.

How can I execute packageExe normally on 1.2.0 stable ?

Solved, when the path exists in Chinese, the generated compose directory will not be in the file directory of the current project due to garbled characters

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.


Replaced Raphael with Snap.svg to allow for allow for extra features …

…like styling
Removed the dependency on the Daniel font. The font can be set using style sheets. However
this introduced a requirement to delay rendering until the font is loaded completely so that
text size can be calculated accurately.

Very nice! I quickly scanned the change, but going to wait until the weekend to fully review/pull.
I fully want to do this migration, but some quick comments/concerns (off the top of my head)

I would like to make the loading of the hand drawn font seamless. So no work for the user other than having the font in the right path.
Is it possible to put your snap.svg work into a separate theme, so Rapheal and snap can coexist? I don't know what my migration plan is, but is it possible to offer both for a while?
I will do a major version bump after I pull this change.

No hurry, I don't have any time to do anything the next week

I understand, and agree. I could not find a quick solution a short amount of time, but it should be possible.
It is possible, but I am not sure if it is worth the hassle. Having to maintain Raphael & Snap.SVG in one codebase can be a challenge. Putting 1.x on a branch will keep code more clean and allow more innovation.
I think a is a good solution. Maintain the Raphael version for a while and focus on new features for the SVG version. I think we can make upgrading to the new version easy, but I don't know how this will affect old browser support. Probably >=IE9 will be possible.

Nice effort @StefRave!

I've pulled your code into a new branch, and started hacking on it. There are some inconsitances with the old vs new rendering. I'm also refactoring this so there is a RaphaelTheme and a SnapTheme. It was quite easy to keep both, without duplicating code. This is mainly useful so I can easily compare old to new (to check for regressions). I'll update you shortly.

I've pulled this into a new SVG branch: https://github.com/bramp/js-sequence-diagrams/tree/svg and started to heavily refactor to allow both Raphael and Snapsvg at the same (if you so wish). I had to cleanup some of this new snapsvg code to match the existing API correctly (for example, snapsvg assumed the container was a SVG, where previous Raphael accepted div/td/etc). Still a work in progress, but getting there. I will close this PR, and track the work in #127


SS-710: Adding a CLOSED functionality for closed courtrooms in the Duty Roster screen

Looked good to me.


Settings directory is created twice if two settings files are used

When the settings directory doesn't exist and two settings files are used, the module tries to create the settings directory twice. The second time fails, and thus the second settings file can't be created.
The error message [Error] ERROR Error: Uncaught (in promise): path: /home/user/.config/com.program.name/ (non recursive): File exists (os error 17) is shown when starting the program with the directory ~/.config.program.name/ not existing. If instead the directory exists (even if it is empty), both settings files are successfully created.
This bug lead to the application crashing because one of the two settings files is unavailable at the first run after deleting the settings directory.

Could you elaborate what you mean by two settings files?

Sure. I have two instances of  SettingsManager in different modules with different filenames (let's say settings-1.json and settings-2.json) being used in different modules of my (angular) app. They both get initialized at the same time. When the application is started for the first time (i.e. the settings folder doesn't exist yet) only one of the SettingsManagers is initialized successfullly and the other is null (and the error above is shown).
From how I understood the code of this module, there seems to be a race condition in this try-catch block: ensure-settings-file.ts:29. Both instances of SettingsManager check if the folder exists, and then both try to create it. But only one can succeed of course.
I'm going to try and fix this, but I haven't been able to get Typescript/Angular to accept my local copy of the module.

From how I understood the code of this module, there seems to be a race condition in this try-catch block: ensure-settings-file.ts:29. Both instances of SettingsManager check if the folder exists, and then both try to create it. But only one can succeed of course.

Hmm, will have to investigate. As a temporary fix, perhaps you can initialize the second SettingsManager once the first one has been initialized using await or .then().

That's a good idea, thanks! I didn't expect you to get active so quickly, appreciate that! :)

Closing this.


investigate reingest process

we are not doing. reingest architecture is complex and we don't need such an involved record


Create MessageQueueInterface and Mock

This change creates an Interface for the MessageQueue in order to create a MessageQueueMock to allow test code to be written more easily.
The templates involved might look strange, but it uses the Curiously Recurring Template Pattern (CRTP) to allow a base class (MessageQueueInterface) to define a function that calls a template function on a derived class (MessageQueuev201::MessageType for instance).  Due to the template functions related to push() not being able to be defined as pure virtual, the CRTP was the way to create an interface and mock while still allowing the code to function as is.
The contains_transaction_messages() and contains_stop_transaction_message() now move their type-specific code into template specialization code in their respective versioned message_queue.cpp files.
Checklist before requesting a review

[x] I have performed a self-review of my code
[ ] I have made corresponding changes to the documentation
[ ] If OCPP 2.0.1: I have updated the OCPP 2.0.1 status document
[x] I read the contribution documentation and made sure that my changes meet its requirements

Overall this looks good, but my main question is how do we use this to, say, check that we send a specific response in one of our message handlers?
If I have a test that does this:
    auto periods = create_charging_schedule_periods({0, 1, 2});

    auto profile = create_charging_profile(
        DEFAULT_PROFILE_ID, ChargingProfilePurposeEnum::TxProfile,
        create_charge_schedule(ChargingRateUnitEnum::A, periods, ocpp::DateTime("2024-01-17T17:00:00")), DEFAULT_TX_ID);

    SetChargingProfileRequest req;
    req.evseId = DEFAULT_EVSE_ID;
    req.chargingProfile = profile;

    auto set_charging_profile_req =
        request_to_enhanced_message<SetChargingProfileRequest, MessageType::SetChargingProfile>(req);

    SetChargingProfileResponse response;
    response.status = ChargingProfileStatusEnum::Accepted;

    ON_CALL(*smart_charging_handler,
            validate_and_add_profile(profile, DEFAULT_EVSE_ID, DEFAULT_REQUEST_TO_ADD_PROFILE_SOURCE))
        .WillByDefault(testing::Return(response));
    EXPECT_CALL(*message_queue, push(testing::_, testing::_));

    charge_point->handle_message(set_charging_profile_req);

Shouldn't I expect to see push() get called somewhere if I'm sending something on the queue? Because currently I get this result:
/workspaces/libocpp/tests/lib/ocpp/v201/test_charge_point.cpp:668: Failure
Actual function call count doesn't match EXPECT_CALL(*message_queue, push(testing::_, testing::_))...
         Expected: to be called once
           Actual: never called - unsatisfied and active


chore: import repository https://github.com/borderx/jx4.git

this commit will trigger a pipeline to generate the CI/CD configuration which will create a second commit on this Pull Request before it auto merges

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign
You can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Question regarding Scaling Laws on PILE

Thank you so much for releasing codes for this inspiring work!
I found the scaling laws experiments on The PILE particularly attractive, showing the superiority of Mamba over many modern models.
(1) I am wondering if the perplexity numbers in Figure 4 are evaluated on the validation set of PILE, or the training subset of that run?
(2) It would also be tremendously helpful if the specific perplexity numbers in Figure 4 for all models could be released so that the community could better study and extend Mamba and its experiments.
Thank you very much for your time and help! @tridao

The perplexity numbers are on the validation set.
I can try to collate these for the next version of the paper.

A small follow-up question for scaling laws config, regarding "linear learning rate warmup with cosine decay":

(1) cosine decay to lr=0 in the end?
(2) how many steps for warmup?
(3) is weight decay and gradient clipping applied to all kinds of parameters?
(4) do word embedding and classification head share parameters (like in nano-gpt)?
Thank you very much for your time and help!

@LeoXinhaoLee This might help: https://github.com/state-spaces/mamba/issues/184


Heading instead of issue

There's a heading in here that should by a github issue: "Need a new screenshot for new Admin Center enablement." Just so it won't be forgotten!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f737e278-7c89-e048-b60b-287615a829fb
Version Independent ID: 1df35258-7360-74c4-680a-e154c54291fe
Content: Microsoft Teams Guest Access checklist
Content Source: Teams/guest-access-checklist.md
Service: msteams
GitHub Login: @romanma
Microsoft Alias: romanma

@biilmanndk The article has been updated and the screenshot is now current. I am closing this issue at this time. If this issue is not resolved please re-open this issue.


Upgrade log4j from 1.x to 2.x

Signed-off-by: Yuqing Wei<EMAIL_ADDRESS>Description

Resolves #1030
How was this PR tested?

local spark is using log4j 2.17.2
Does this PR introduce any user-facing changes?

[ ] No. You can skip the rest of this section.
[ ] Yes. Make sure to clarify your proposed changes.

Do you know why we suddenly started getting this error, does old version of log4j does not work with some of the other new dependencies?

Do you know why we suddenly started getting this error, does old version of log4j does not work with some of the other new dependencies?

Log4j 2 introduces some breaking API changes, I've updated all places that use it.


Enable werror when safe

These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).
SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

I think it might be better for developers if this is set to ON by default and disabled for specific platforms.  That will make it less likely for developers to check in code (albeit on a branch) that is doomed to failure in CI.

I think it might be better for developers if this is set to ON by default and disabled for specific platforms. That will make it less likely for developers to check in code (albeit on a branch) that is doomed to failure in CI. But I can go either way. What do you think?

In my experience -Werror by default is always painful for who just does git clone and wants to try it.
It increases friction because we don't know all arch / compiler version / OS version / libc version combinations, so in those unknown environments people will just get an error and they will have to look for how to disable that option.
We should rather suggest to enable it before contributing, but still only the CI can check changes against a number of different environments, so we need to deal with this kind of issues when submitting a PR.

Excellent point. Maybe we should just add a note to the "Contributing" section of the docs that suggests developers use this flag when locally testing builds before they submit a pull request.


Procedure to initialize a constant to maximum precision

I guess this may be more of a conceptual question than one related to your implementation. I'm hoping an answer might be useful not only to myself, but to others looking to use the library -- perhaps this discussion could be added to the documentation.
Suppose I would like to load a constant as a CompensatedNumber<double> with as much precision as possible. This constant can be computed using MPFR, Wolfram Alpha, PARI/GP, Sage or anything else capable of arbitrary precision floats. For the sake of example, let's use π, and I'll show some code in PARI/GP with the default 38-digit precision, which should be plenty.
First, I compute a double-precision approximation to π, which calls for normalizing the mantissa to the interval [1,2) (requiring a division by 2 in this case) and subtracting the implicit integer part of 1:
? Pi_d = 2.0 * (1 + round((Pi/2 - 1)*2^52)/2^52)
%1 = 3.1415926535897931159979634685441851616

Next I compute the error:
? e = Pi - Pi_d
%2 = 1.2246467991473531772260471018642244504 E-16

Given these results, is this the correct way to initialize a CompensatedNumber<double> to achieve the best possible approximation to π within the available precision?
CompensatedNumber<double> Pi(3.141592653589793, 1.2246467991473532e-16);

You have the key ideas. I would do it as follows (note that this could easily be automated):
PI = // you get this from MPFR or something else
pi_double = static_cast<double>(PI); // or any good double approximation for pi
pi_double_error = static_cast<double>(PI - pi_double); // our approximation error, encoded in double as precisely as we can

pi_compensated = CompensatedNumber<double>(pi_double, pi_double_error); // number as represented by the pair

Do not hesitate to submit a PR to improve the readme if you believe this information could be more widely useful.


Scale a single item in height

Hey, I would like to scale a single item in height,
and as such the whole row that is beneath it. (not the rows beside).
Is there an easy way to implement this in Wookmark ?
If not how would one approach developing this function ?

Hi,
with rows you mean columns?
After changing one item in height just trigger the update on wookmark.


[2217] Module functionality redesign

Fixes #2217

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: vladsch:x: LeonidVeriga

This look awesome! I have a quick question
If I understand it right, this makes it possible to generate/output the javascript using modules (commonjs, umd... etc.) but will it make it possible to require("other-modules")? Because as of now, when writing a definition library we assume the javascript is already bundled and does not need to be require-ed anymore.

Hi @Zaid-Ajaj
By default, Bridge will generate module loading via require automatically if a module is used inside your project (you have to attach javascript loader library to yor html page only). So, any code which uses code from module will wrapped by require method (as callback)
If you want to disable automatic loading then just set it in bridge.json, like
loader: {
   manualLoading: true
}

In this case, Bridge will not generate require instruction.


Provide type and subtype from IGSDB in the product data network

@simon-wacker For each data set, IGSDB has the key type with the possible values glazing orshading. It has also the key subtype with the possible values monolithic, coating, coated glass,  film, applied film, interlayer and laminate. For at least one of our associated partners, it would be very helpful to have these values available in the GraphQL-API of the IGSDB. When you succeed with the thickness, can you easily add the two keys?
@danielmcquillen @RDmitchell Would it be ok for you if we do this?

More information about the subtypes is available at https://igsdb-v2.herokuapp.com/docs/help/whats-changed/type-subtype . Thank you @danielmcquillen !

@christoph-maurer When you run the query
query AllData {
  allOpticalData {
    edges {
      node {
        type
        subtype
      }
    }
  }
}

against the IGSDB, for example after signing in on https://igsdb-v2.herokuapp.com/graphql/, you receive type and subtype of all data. Is that just what you were asking for?

Yes, this is what we currently need. Thank you @simon-wacker !


Replace fancyregex with regex

Turns out that the plain regex crate can handle all regexes of uap-core. From what I could see, fancyregex tries to forward as much as possible to a plain regex::Regex in such cases, it's possible to get rid of the dependency. As part of the refactor, I was able to simplify a few double-references.
There is one difference: The regex crate rejects meaningless escape sequences. There are some instances in uap-core both in the referenced version 0.10 as well as in the latest 0.15. I've added a clean_escapes method that removes such escapes if they occur.
To clean invalid escapes I chose a regex, since we already have the dependency in. There are more elaborate manual ways to write this with little performance gains, but I opted for less code.

I will have to play with that in a follow-up, but this move would allow us to use RegexSet, see the example here.

I also noticed that we can use RegexBuilder methods in device::Matcher::try_from. Let me know if you'd like me to refactor that method, too.

Appreciate this, thank you! And thanks for looking into RegexSet as an alternative.


Errors while running test_views

The last time I tried to run our test suite, I got a bunch of errors.
Here's the traceback for two of the failing tests; the other failures are similar to one or the other of these:
======================================================================
ERROR: test_provide_suggested_fulltext (main_app.tests.test_views.SourceEditChantsViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 1780, in test_provide_suggested_fulltext
    response = self.client.get(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 135, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 204, in get
    return super().get(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 144, in get
    return self.render_to_response(self.get_context_data())
  File "/code/django/cantusdb_project/main_app/views/chant.py", line 1665, in get_context_data
    request = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

======================================================================
ERROR: test_update_chant (main_app.tests.test_views.SourceEditChantsViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-cid/446604 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 1754, in test_update_chant
    response = self.client.get(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 135, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 204, in get
    return super().get(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 144, in get
    return self.render_to_response(self.get_context_data())
  File "/code/django/cantusdb_project/main_app/views/chant.py", line 1665, in get_context_data
    request = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in send
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in <listcomp>
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-cid/446604 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

One thing that confuses me is that django is not identifying the locations in test_views.py where errors are occurring.

A few things I have tried while investigating this:

switching from the develop branch to the staging branch and running the test suite there
connecting to the staging server and trying to run tests there.

Locally, I get the same number of errors while on the staging branch as on the develop branch. This confuses me, since all tests were passing on that branch a couple of weeks when we last merged develop into staging for #549.
And when I try running the suite locally vs. on the staging server, I get different numbers of errors. I don't understand why this should be the case.

Testing locally:

on the develop branch: 8 errors
on the staging branch: 8 errors

sshing into http://<IP_ADDRESS>:

on the develop branch: 111 errors
on the staging branch: all tests pass

Investigation is ongoing! I'll post more details here as they emerge. If anyone has any ideas of what might be going on, or of things worth trying, don't hesitate to let me know!

Some of the errors above suggest that something may have changed on cantusindex.org, causing some of the tests to break.
When I try to visit https://cantusindex.org/json-cid/446604 in my browser (suggested by the last line of the traceback posted above), I get a page that loads properly, consisting of an empty list. curling the url seems to work too, at least most of the time:
~ curl https://cantusindex.org/json-cid/446604
curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer
~ curl https://cantusindex.org/json-cid/446604
[]%
~ curl https://cantusindex.org/json-cid/446604
[]%

occasionally, we get similar error messages, but for /json-con/ urls as well:
======================================================================
ERROR: test_url_and_templates (main_app.tests.test_views.ChantDetailViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-con/014866 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 530, in test_url_and_templates
    response = self.client.get(reverse("chant-detail", args=[chant.id]))
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/detail.py", line 109, in get
    context = self.get_context_data(object=self.object)
  File "/code/django/cantusdb_project/main_app/views/chant.py", line 112, in get_context_data
    response = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in send
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in <listcomp>
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-con/014866 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

Applying a label of "urgent", because we should really get this fixed before this year's training week, for the benefit of anyone just getting started working on CantusDB.

Detective work: locally, test_functions (contains only 3 tests) runs fine.

The plot thickens! I'm going through the test suite, commenting out tests. I run the test suite - 1 error. Comment out the function causing the error, and run the test suite again - 1 error, but on a different test! Run the test suite again, without making further changes - 2 errors this time!
One of the tracebacks mentioned the python threading library, which we only use (afaik) when fetching lists of concordances from Cantus Index. My best guess at the moment is that calls we're making to CI's/others' APIs are leading to responses that are wonky in one way or another, in different ways from one request to the next, leading to this unpredictability.

Okay, something is wrong with Cantus Index:
$ curl cantusindex.org/json-con/703437
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<title>301 Moved Permanently</title>

Moved Permanently

The document has moved here.

further,
$ curl https://cantusindex.org/json-con/703437
[]%
$ curl http://cantusindex.org/json-con/703437
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<title>301 Moved Permanently</title>

Moved Permanently

The document has moved here.

Which suggests that it's a difference between http and https.

@dchiller and I spent some time troubleshooting this today. We're still stumped, but we discovered a couple of things that are potentially helpful:

we are, in fact, getting line numbers within test_views.py - see, for example, the following, hidden in the middle of one of the tracebacks:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 1780, in test_provide_suggested_fulltext
    response = self.client.get(

the above suggests that something is going wrong in Django's handling of requests/responses
While some of the affected views depend on external resources (e.g. Cantus Index's APIs), we found at least one view where errors are occurring where there are no requests made to external resources. We ran just this test 5 times, and it generated an error 4 times
I ran the full test suite a half dozen times, and kept track of which tests failed each time. None of the tests led to an error every time - each one worked at least once. Every time I ran the test suite, a new test failed that hadn't previously.

Further thoughts: so yesterday, while @dchiller and I were pairing on this, we switched to the staging branch (for which all the tests were certainly working the last time we merged develop into staging), and ran the test suite several times - we observed the same sort of errors outlined in this issue's first few comments. Just now, I ran the test suite two times on the staging server (i.e. on the staging branch), and two times on the production server (on the production branch). All four times, all tests passed this time. This suggests that I might have messed up my local environment somehow. I might try uninstalling and reinstalling the project to see if that helps.

I don't know how this got closed last week.

I don't know how/why this was closed last week.

You automatically linked #629 to this issue. When you include "X + Issue number" in you PR text, the issue is linked if "X" is one of a few reserved words (one of which is the culprit here: "fix")-- see here for the full list: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue


ArchLinux compilation error on SDL2_ttf

During ArchLinux compilation meet with error message:
# pkg-config --cflags  -- SDL2_ttf
Package SDL2_ttf was not found in the pkg-config search path.
Perhaps you should add the directory containing `SDL2_ttf.pc'
to the PKG_CONFIG_PATH environment variable
Package 'SDL2_ttf', required by 'virtual:world', not found
pkg-config: exit status 1

Could you please suggest how to fix it?

Hi @vit1251, it seems like the sdl2_ttf package hasn't been installed. Does it work if you install the package?

Yep. My fault. Sorry,


In case my project is hosted on both maven central and bintray, only the latest version from maven central is hosted

I use bintray to deploy early releases, and maven central repository to host stable ones.
Here's a snapshot from the latest releases

javadoc.io will only detect the ones on maven

Expected: it should detect both releases and show the latest one, or give an option to sync from bintray or sync from maven central

It doesn't seem this lib is released to bintray? https://jcenter.bintray.com/io/github/mohabmohie/SHAFT_ENGINE/maven-metadata.xml
Closing this for now. Feel free to reopen if you still have the issue after bintray is enabled.

I'm think you're referring to this file, correct?
https://dl.bintray.com/mohabmohie/SHAFT/io/github/mohabmohie/SHAFT_ENGINE/:maven-metadata.xml


RqHref

Let's replace RqURI and RqQuery with RqHref

@rultor release, tag is 0.8.2

@rultor release, tag is 0.8.2

@yegor256 OK, I will release it now. Please check the progress here

@rultor release, tag is 0.8.2

@yegor256 Oops, I failed. You can see the full log here (spent 11min)

@rultor release, tag is 0.8.2

@rultor release, tag is 0.8.2

@yegor256 OK, I will release it now. Please check the progress here

@rultor release, tag is 0.8.2

@yegor256 Done! FYI, the full log is here (took me 18min)


Scripts are not evalutated in header and footer HTML templates for PDFs.

Steps to reproduce

Puppeteer version: 1.1.1
Platform / OS version: Windows 10, version 1709, build 16299.248
Node.js version: v7.9.0

What steps will reproduce the problem?
First, I setup a basic test page like this:
const browser = await puppeteer.launch({ignoreHTTPSErrors: true})
const page = await browser.newPage()
await page.setContent(`

    <title>Test Page</title>

    I'm an HTML page.

`)
const pdf = await page.pdf({
  format: 'Letter',
  printBackground: true,
  headerTemplate: headerTemplate,
  displayHeaderFooter: true,
  margin: {
    top: '1in',
    right: '0in',
    bottom: '1in',
    left: '0in'
  }
})
await browser.close()

The headerTemplate looks like this:
const headerTemplate = `

Page number

`

What is the expected result?
The docs don't mention whether or not script tags are supported in the header and footer templates. I would expect then that the code above would render a header with a red background and remove the page number node. (The code above was just for testing purposes).
What happens instead?
The script appears not to evaluate. Instead the resulting PDF has no background, and a page number div.
If evaluating scripts in these templates is not supported, might I suggest updating the PDF section of the docs to state as much. Thanks for all the hard work the Puppeteer team has put into this project!

I also need this feature, because I do not see a normal way to make different headers for different pages. For example on first page (intro page) is one header and on the internal pages different.
If not possible run scripts in header/footer templates I have another proposal that can be solved via CSS:

Intro header

Internal header

#page1 {
  display: none;
}
#page1:target {
  display: block;
}
#page1:target + .header {
  display: none;
}

Maybe it helps, or is an drive for your own thoughts: https://github.com/GoogleChrome/puppeteer/issues/1853#issuecomment-377509877

I also met this problem,but I dont know how to slove this problem.my code doesn't seem to be a bit of a problem; await page.pdf({         path: ${path}/${title}.pdf`,
scale:0.7,
margin: {top: 75, bottom: 75},
format: 'letter',
headerTemplate: '',
// footerTemplate: '',
footerTemplate:'',
displayHeaderFooter: true,
});

I also met this problem,but I dont know how to slove this problem.my code doesn't seem to be a bit of a problem;
await page.pdf({         path:${path}/${title}.pdf,         scale:0.7,         margin: {top: 75, bottom: 75},         format: 'letter',         headerTemplate: '

',         // footerTemplate: '

',         footerTemplate:'

',         displayHeaderFooter: true,       });

I am stuck on this too.. I am trying to replace wkhtmltopdf, which allows scripts in the header/footer.  They also have [section] and [subsection] which render as the contents of the most recent  and , respectively.    Currently, chrome has no equivalent functionality.
If, when the header/footer was rendered, there was some information available via javascript that allowed locating elements relative to the boundaries of the page, to serve this same purpose, that would allow the same type of behavior, eg printing the name of a section that started on a previous page in the header or footer of the current page.
Thanks to all for your great work.

I would also appreciate some development concerning these issues. Javascripts in headers and footers or a header footer function was absolutely useful. Also have external css loaded in headers work would be a great reduction of pain ;-)

I know this question,I found a way to solve the problem.The reason is that I have been packaged by others.

From: raffaelmeier<EMAIL_ADDRESS>Sent: Tuesday, September 25, 2018 7:38:14 AM
To: GoogleChrome/puppeteer
Cc: zhoupeng; Comment
Subject: Re: [GoogleChrome/puppeteer] Scripts are not evalutated in header and footer HTML templates for PDFs. (#2167)
I would also appreciate some development concerning these issues. Javascripts in headers and footers or a header footer function was absolutely useful. Also have external css loaded in headers work would be a great reduction of pain ;-)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/GoogleChrome/puppeteer/issues/2167#issuecomment-424368905, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Af9UjMZcd1YK512RK3yEOssT07ngCr8sks5uej_WgaJpZM4SjMI_.

Hi skymon4. please descibe your solution. What do you mean by "The reason is that I have been packaged by others." - i do not understand this sentence.

@raffaelmeier I also execute JavaScript in the header and footer. Maybe my code will help you.

https://github.com/GoogleChrome/puppeteer/issues/2167#issuecomment-377510808
https://github.com/cakebake/markdown-themeable-pdf/blob/rewrite/lib/api/convert/template.js#L37

this is chromium source code print_header_footer_template_page.html
https://cs.chromium.org/chromium/src/components/printing/resources/print_header_footer_template_page.html
function setup(options) {
  const body = document.querySelector('body');
  const header = document.querySelector('#header');
  const footer = document.querySelector('#footer');

  body.style.width = `${options.width}px`;
  body.style.height = `${options.height}px`;
  header.style.height = `${options.topMargin}px`;
  footer.style.height = `${options.bottomMargin}px`;

  header.innerHTML = options['headerTemplate'] || `

`;
  footer.innerHTML = options['footerTemplate'] || `

      /

`;

  options.date = new Date(options.date).toLocaleDateString();
  for (const cssClass of ['date', 'title', 'url', 'pageNumber', 'totalPages']) {
    for (const element of document.querySelectorAll(`.${cssClass}`))
      element.textContent = options[cssClass];
  }
}

headerTemplate and footerTemplate content was used innerHTML method to render , so it can't execute JavaScript, but we can use other method to execute javascript, we can use under code:

header test

the result is js modify test color to green.

@yale8848 I've run your snipplet, but I've got a black 'header test' text and I cannot reproduce your result. I've tested on a Chrome 69.0 in headless mode, in Win10 environment, with Node.js v8.11.3.
Could you please detail your setup? OS? Chrome version? Etc?
I've also tested without the onload attribute, and the image has not loaded. If I set a base64 data img, then the image is displayed, but with the onload attribute, the JS does not run in this case neither. I'm not able to run JS at all in the templates.
My goal is to modify the counter in the pageNumber span, to count from a given number, instead of one.
Any ideas are welcome. Thank you for your answer.

@user9179380  I test in url-to-pdf-api project docker container,   https://github.com/yale8848/url-to-pdf-api-docker
os： Debian GNU/Linux 8 (jessie)
node：8.12.0
chrome：571375
I try to modify 'header test' color  in js code is ok , but try to get 'pageNumber' value in js code , it get empty.
I will try to find other method in next time.

@yale8848 Thank you for you answer. I will check the linked docker container. Do you use any specific Chrome flag? The list of flags are available here:
https://peter.sh/experiments/chromium-command-line-switches/
How did you try to get access to pageNumber? By using innerHTML? Please try to run the following snipplet for me, maybe you can modify the pageNumber this way:

If it doen't work, please also try an external image source. Theoretically the code modify each pageNumber by 10. Does it work for you?
I've posted a question about the issue on stackexchange:
https://stackoverflow.com/q/52729615
Thank you for your contribution.

I need the headerrTemplate/ footerTemplate can support  function setting.

yeah，as some reason,this way is some diffcult.but if we do use truth way, we can solve this question

Something that might be quick to deliver is to allow this resource file to be replaced at run time:
https://cs.chromium.org/chromium/src/components/printing/resources/print_header_footer_template_page.html
Then I would be able to write my own custom setupHeaderFooterTemplate function. I could then do a bit more than replace the pageNumber text, such as adding a conditional class based on the page number.

I have similar needs but simpler. I want to have a "Back to top" link in footer which can navigate to page 1 when clicked. Tried Back to top or Back to top in footerTemplate option and didn't work.
PS: i place a  after the starting .

@luckydrq did you find how to manage that ?
I have the exact same needs and even if links for table of contents works well, as soon as the link is in the footer, it's ignored.

@luckydrq did you find how to manage that ?
I have the exact same needs and even if links for table of contents works well, as soon as the link is in the footer, it's ignored.

Not yet, seems that links not supported in footer.

I have been trying to do the same (run code in header/footer) but haven't been successful. Ultimately I would like to have number page 1 from the second page. And remove headers from 1st page.

@yale8848
I've been playing with running an onload script and thought it might work as your example was producing numbered pages. However it seems this is achieved via the class name rather than the script execution.

Simply using the class="pageNumber" prints the page numbers for me.

@buzzie-bee
Yes, you are right, class="pageNumber" can show page number OK.  I just try to find other more flexible way to modify header and footer, so i find onload script. It was just a trick  :)

@user9179380 I test in url-to-pdf-api project docker container, https://github.com/yale8848/url-to-pdf-api-docker
os： Debian GNU/Linux 8 (jessie) node：8.12.0 chrome：571375
I try to modify 'header test' color in js code is ok , but try to get 'pageNumber' value in js code , it get empty. I will try to find other method in next time.

@yale8848
Hi, I run your docker container, but it doesn't work. Can you give me some advice or help?

code
`

header test

`
2. result
![20220331114410](https://user-images.githubusercontent.com/23287946/160972253-da133154-00d9-43f1-9580-4f1ebeafd3eb.jpg)
Waiting for your reply, thank you.

@user9179380 I test in url-to-pdf-api project docker container, https://github.com/yale8848/url-to-pdf-api-docker
os： Debian GNU/Linux 8 (jessie) node：8.12.0 chrome：571375
I try to modify 'header test' color in js code is ok , but try to get 'pageNumber' value in js code , it get empty. I will try to find other method in next time.

@yale8848 Hi, I run your docker container, but it doesn't work. Can you give me some advice or help, please?

code

header test

result

Waiting for your reply, thank you.

try to change onerror=  to  onload= .

@yale8848  I tested onerror/onload, but it still doesn't work properly.

🤷

I have similar needs but simpler. I want to have a "Back to top" link in footer which can navigate to page 1 when clicked. Tried Back to top or Back to top in footerTemplate option and didn't work.
PS: i place a  after the starting .

Hello @luckydrq, @lesfauches were you able to achieve "Back to top" at all using puppeteer ? 4 yrs later I want to achieve the same  thing but I am stuck. Any insight, help or workaround would be appreciated.


Characters appearing out of the window + other

What is your question?
How do I make a watermark at the bottom left of the screen which says your computer username, a character appearing out of the window, and a notification appearing from the app like in Exploitation from Dave and Bambi 3.0? Oh, and that one feature where in that song, the pause menu has a stroke.

https://github.com/MoldyGH/VsDave

https://github.com/MoldyGH/VsDave

indeed
if only people would do research these days

It’s kade engine

It’s kade engine

still, it should work with psych engine


Update for Akka 2.4.0

Currently akka-kryo-serialization works with Akka 2.3.x, however it doesn't work for Akka 2.4.0

Two tests were failing:
[info] KryoSerializer
[info] - should be selected for lists
[info] - should be selected for ActorRef *** FAILED ***
[info]   false was not true (AkkaKryoSerializationTests.scala:163)
[info] - should serialize and deserialize ActorRef successfully *** FAILED ***
[info]   false was not true (AkkaKryoSerializationTests.scala:173)
[info] - should produce smaller serialized List representation when compression is enabled
[info] - should produce smaller serialized huge List representation when compression is enabled
[info] - should produce smaller serialized huge Map representation when compression is enabled
[info] - should produce smaller serialized Seq representation when compression is enabled
[info] - should produce smaller serialized huge Seq representation when compression is enabled
[info] - should produce smaller serialized huge Set representation when compression is enabled

These have been fixed in the pull request. Please review.

Thanks a lot for your contribution!
A new release with your changes and @luben's improvements is rolled out and available on Maven Central.  Closing.


Implements an option to prevent the automatic encoding

I'm implementing an Atom feed generator and I'd like to be able to disable the automatic string encoding.

This change looks good :+1:
Would you mind dropping a bit more of an explanation, including references, for the use-case here? This will help with documentation and other people trying to do the same thing.

Sure!
This PR adds a global escape option. When set to true, special characters in strings passed to xml won't be escaped to HTML-entities. This could be used for many things. One example is atom-feed entries. They can have a type="html" attribute. In most use cases you would get the HTML as a string and render the feed using node-xml. That's when you'd want to disable escaping. Making it a global option is simply the easiest way to implement that.
Here's a usage example I used in the tests:
xml(
  [ { x: 'test' } ],
  { escape: false }
)

This would return
<x>test</x>

We'd probably also want a local way to do this. Similar to how _cdata works:
[ { x: { _raw: 'test' }  } ]

I'm interested to understand fully the reason for doing this in atom - you say "[feed entries] can have a type="html" attribute.", I'm trying to understand the difference between "can" and "should", is there a reference in the atom docs or to something about best practice that you can link here?
There is a very related issue on the node-rss repo (which directly depends on node-xml): https://github.com/dylang/node-rss/issues/27 - node-rss is supposed to output atom-compatible feeds, and the discussion there suggests that escaping should happen only if there are chars which need escaping, E.g. an 'auto' mode rather than on/off.
Just trying to get an idea of what the best path forward is for both node-xml and node-rss in respect to this issue.

Sorry, I may have been a bit wage in my explanation. Text elements can have an attribute type="text|html|xhtml". So the feed reader would interpret the content as HTML.

If type="html", then this element contains entity escaped html.

Here's the relevant part on their Website
The example on that page isn't very clear. But as I interpret it, you could do
<content type="html">
  Yolö
</content>

That Yolö would in most cases come from some CMS or something. So you could simply use it as is inside the <content> of an article inside the Atom-Feed.
As far as I understand the problem in dylang/node-rss#27 is only remotely related. The auto detection he's talking about isn't something that I would include in node-xml itself (except maybe as an option), since it would give the programmer less control.

I wanted to check up on the state of this issue, since the Atom feed on my project is invalid ATM.
Will this get merged?

I also have a project that would benefit from this additional functionality.

I'm sorry, for some reason all the notifications from this repo started going into my spam folder so I had no idea there'd been any followup!
This PR currently has a merge conflict. If we could get that fixed up, then I'll look at pushing this out.
This PR is based from a master branch, making it slightly tricker than normal to fixup, but these steps should work assuming your repo is called origin and this one upstream in your remotes config:

git fetch upstream (fetch from this repo)
git pull -r upstream master (rebase in latest changes from this repo)
resolve any conflicts (I think it's just a newline at the end of xml.js)
git push origin master -f (push the update to this PR)

Hopefully this gets us moving again!

Great! Thanks a lot. :)
I'll probably get to it today.

Thanks a lot @ErisDS for the instructions :)

Is there still something that has to be done, in order for this PR to be merged?

We need this too!

Bump! Looking forward to this

Will this PR ever be merged? @halbgut  @dylang ..We need this!


Ckan provider (WIP "kinda")

This is the provider plugin for Ckan. This plugin is working correctly, but there is no Ckan live site to test the upload correctly. Until we find a Ckan site that allows us to add resources or someone gets back to me on how to upload files correctly, then I think we should wait on this merge.

So I tired working on this plugin again, but there seems to be a lot of issues still going on with the Demo Site that they have here: https://demo.ckan.org/. If anyone knows of a CKAN site that you are able to grab data and push data too let me know. I haven't been able to find one that works correctly, or one that has the rest api turned on. Maybe for now we just focus on getting data from CKAN rather than trying to store data.

Maybe we just need to stand up our own instance of CKAN.


[Podcast Addict Remote] Make sure widgets are hidden when fast loading...

... when using Fastload Utils.

I'm really not sure about this - under what circumstances does this not work? What app are you loading from, and does it have widgets shown? Does it happen just for these apps, all apps, or any apps that use widget_utils.hide()?
I feel like we need to fix this somewhere else rather than adding it to any affected app. For instance fastload could override Bangle.loadWidgets and just do it there...


ReadItLater: Save Clipboard should save binary files not create notes with binary files

⚠ Please check that this feature request hasn't been suggested before.

[X] I searched previous Ideas in Discussions didn't find any similar feature requests.
[X] I searched previous Issues didn't find any similar feature requests.

🔖 Feature description
If a pdf/png URL is open / selected with the ReadItLater: Save Clipboard, it should save that file and not create a new note and paste the binary content into the note.
✔ Solution
I'd like to be able to save pdfs directly into obsidian without having to download a link, save the link to my download folder, then drag/drop it into the vault
❓ Alternatives
No response
📝 Additional Context
No response

I just wanted to say i think this would be a great addition! I ran into this while trying to update my random product PDF links into downloaded articles with ReadItLater.
One solution is to simply create a stub page with an attachment to the file.
# some-article.pdf

![[assets/some-article.pdf]]

But that's just a glorified binary like the videos are, exploring this a bit further...
I thought it'd be cool if one could include the TOC from the PDF to make it more discoverable in the Vault search. But some PDFs don't have one.
For instance a Kingston PDF manual I was testing with was created with Adobe InDesign 17.3 and the TOC isn't text, it's just links to pages. Which isn't very useful. I confirmed it with a few browsers just to be sure.
However, I was surprised Edge's built-in Copilot generated a quite reasonable summary of the doc with the prompt "Generate a brief summary of this pdf". So I ended up throwing that in my my doc under a Summary. Fun stuff!


Fix empy

FiSigned-off-by: Pablo Garrido<EMAIL_ADDRESS>

Hi @pablogs9, is it intentional that empy is written in the dependency list in the 3rd place without a specific version, and then at the end with the 3.3.4 version? Shouldn't it only be defined with the 3.3.4 version and remove the other one?

Nope, just an error. I`ll open a fix

Done: https://github.com/micro-ROS/micro_ros_platformio/pull/157


Active tab not showing in tabs

When there is too many tabs in atom. Current active tab is not highlighted . In sublime there is one icon to scroll in tabs. But atom does not have icon or not automatically highlighting the active tab.

Thanks for reaching out!
We require the issue template to be filled out on all new issues and pull requests. We do this so that we can be certain we have all the information we need to address your submission efficiently. This allows the maintainers to spend more time fixing bugs, implementing enhancements, and reviewing and merging pull requests.
Thanks for understanding and meeting us half way :grinning:


Failed to initialize ORM engine: dial tcp <IP_ADDRESS>:5432: connect: connection refused

Gitea version (or commit ref): 1.13.3
Git version: 1.13.3
Operating system: Docker running on Ubuntu Focal
Database (use [x]):

[X] PostgreSQL
[ ] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[x] Not Relevant

Log gist:
Shown in screenshot

Description
I'm running the docker-compose.yml exactly as shown in documentation: https://docs.gitea.io/en-us/install-with-docker/#postgresql-database
Pretty much the same error as this: https://github.com/go-gitea/gitea/issues/3877
Screenshots

The docker-compose.yml is exactly as written in documentation:
version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:1.13.3
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    ports:
      - "5433:5432"
    volumes:
      - ./postgres:/var/lib/postgresql/data

Ok, I found the issue. This should probably be added to the documentation.
If you incorrectly set the DB_HOST the first time, because it creates a local volume under ./gitea:/data it will store those settings in the file ./gitea/gitea/conf/app.ini and even if you update them on subsequent tries, it does not update those values which have already been stored. The solution is to erase the volume, update the values, and start again.

Feel free to send a PR to update the documentation.

This is resolved in 1.14-dev with the introduction of env-to-ini functionality which is use is documented currently.

@techknowlogick where is it documented? I am updating my docker to 1.14.1 and not sure how to get this to run.


Fix cppo -version and improve make release

This PR fixes issue #92 and proposes an improved command make release VERSION=X.Y.Z which creates and publishes a new release.

Looks good to me! (Feel free to merge when you feel comfortable with it.)

Thanks, I have merged.


typing mismatch,  js-markerwithlabel is using @types/google.maps instead of @types/googlemaps.

Getting build error when trying to use js-markerwithlabel package.
In order to use map typing I am using @types/googlemaps. I wanted to use js-markerwithlabel, but getting build errors.

Platform: Angular
It appears to issue with types mismatch @types/google[dot]maps vs @types/googlemaps

You can use https://www.typescriptlang.org/tsconfig#skipLibCheck.

Thanks @jpoehnelt, that fixed it.


ResultSet#close cannot be called multiple times

Instead, an exception will be thrown if called when it's already closed.
Therefore, the following code used to work with the legacy JDBC driver, won't work now:
try (Statement statement = connection.createStatement();
    ResultSet resultSet = statement.executeQuery(query)) {
    // do something with resultset
}

Statement closes ResultSet first and ResultSet explicit close will fail.

@fbiville I assume that you are referring at the HTTP module, can you confirm?
In that case I agree in what you said, as stated in the specifications for that method "Calling the method close on a ResultSet object that is already closed is a no-op."
I didn't see any problem with the BOLT module though.

Sorry for the late reply. Yes, it occurred when I used the HTTP module.


make gsettings work as a wayfire backend plugin

Some basic changes to make the plugin work as a config backend plugin for Wayfire (https://github.com/WayfireWM/wayfire/pull/933)
There are some ideas which I don't really know how you want to solve (and don't feel like implementing):

Input and output device configuration - Wayfire is transitioning to using <object>s for them, and config backend plugins get called when such sections are required, so wf-gsettings can create them on the fly or stuff like that. I assume @damianatorrpm will be interested in this since he's currently patching output-layout to get that functionality.

Loading the options immediately from gsettings in the init method, so that Wayfire has the correct options from the start.

Maybe override default config values with some system .ini file like the default backend does?

Input and output device configuration

This already works (if object patches are applied).
Basically the dynamic sections should be mentioned in org.wayfire.gsettings, and the path that would be checked for each such section is "/org/wayfire/section/" + obj_type_name + "/" + section_name + "/" where e.g. for core.output:DP-1 the obj_type_name is core/output (dots get replaces by slashes) and section_name is DP-1.

Maybe override default config values with some system .ini file like the default backend does?

That's already happening inside gsettings. There are both gsettings vendor overrides (here ctrl+f vendor), intended for use by distributions, and dconf overrides, intended for use by sysadmins.

Loading the options immediately from gsettings in the init method, so that Wayfire has the correct options from the start

Yeah, the init method would have to use the pipe to wait for the initial read, I'll do this.


Revert "[front] Reuse sparkle tree in datasources selection"

Remaining issue with table in tree
Reverts dust-tt/dust#5632

:+1:


Wrapper scripts only work if called from the current directory

For example call /opt/deepce/docker-wraper.sh and the script will fail as it will look in current dir for the script instead of the correct path.

Thanks for this, have just fixed it!


Shard-level buffer

These changes improve how the buffer uses memory.
Fix

[x] Add a unit test to verify that the fix works.
[x] Update the CHANGES log.

Codecov Report

Merging #339 (4b9ad81) into main (2cf42aa) will decrease coverage by 0.53%.
Report is 1 commits behind head on main.
The diff coverage is 58.18%.

@@            Coverage Diff             @@
##             main     #339      +/-   ##
==========================================
- Coverage   39.87%   39.34%   -0.53%
==========================================
  Files         111      112       +1
  Lines       11938    12298     +360
==========================================
+ Hits         4760     4839      +79
- Misses       6709     6983     +274
- Partials      469      476       +7

Files
Coverage Δ

banyand/tsdb/shard.go
57.07% <100.00%> (+0.66%)
:arrow_up:

banyand/tsdb/tsdb.go
53.93% <100.00%> (+0.56%)
:arrow_up:

banyand/tsdb/block.go
50.91% <50.00%> (-0.21%)
:arrow_down:

banyand/kv/kv.go
0.00% <0.00%> (ø)

banyand/kv/badger.go
8.90% <65.38%> (ø)

banyand/stream/tstable.go
0.00% <0.00%> (ø)

banyand/measure/tstable.go
0.00% <0.00%> (ø)

banyand/tsdb/buffer.go
68.99% <67.77%> (+0.42%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Gnu make replacement

I am shopping around for a Haskell replacement for Gnu make / npm script. How would I accomplish this with turtle? Also, how would it compare to shake?

So, first off, I would highly recommend trying out nix or shake first before other alternatives.
However, I do have a "low-tech" build system that only requires about ~100 lines of Haskell code to implement.  Here's the implementation:
{-# LANGUAGE ExistentialQuantification #-}
{-# LANGUAGE OverloadedStrings         #-}
{-# LANGUAGE GADTs                     #-}

import Control.Applicative
import Control.Concurrent.STM (STM)
import Control.Monad.IO.Class (liftIO)
import Control.Monad.Managed (Managed)
import Data.Functor ((<$))
import Data.Monoid ((<>))
import Data.String (IsString(..))
import System.FSNotify (Event(..))
import System.FilePath ((</>))

import qualified Control.Concurrent.Async
import qualified Control.Concurrent.STM        as STM
import qualified Control.Concurrent.STM.TQueue as TQueue
import qualified Control.Monad.Managed
import qualified System.Directory
import qualified System.FilePath
import qualified System.FSNotify
import qualified System.Process

data Pair a b = Pair !a !b

-- | Variation on `Control.Foldl.FoldM` with a pure `done` function
data FoldM m a b = forall x . FoldM (x -> a -> m x) (m x) (x -> b)

instance Monad m => Functor (FoldM m a) where
    fmap f (FoldM step start done) = FoldM step start done'
      where
        done' x = f (done x)

instance Monad m => Applicative (FoldM m a) where
    pure b = FoldM (\() _ -> return ()) (return ()) (\() -> b)

    FoldM stepL beginL doneL <*> FoldM stepR beginR doneR =
        let step (Pair xL xR) a = do
                xL' <- stepL xL a
                xR' <- stepR xR a
                return $! Pair xL' xR'
            begin = do
                xL <- beginL
                xR <- beginR
                return $! Pair xL xR
            done (Pair xL xR) = doneL xL (doneR xR)
        in  FoldM step begin done

instance (Monad m, Monoid b) => Monoid (FoldM m a b) where
    mempty = pure mempty

    mappend = liftA2 mappend

data Buildable a = forall e . Buildable (Managed (STM e, FoldM IO e a))

instance Functor Buildable where
    fmap k (Buildable m) = Buildable (fmap (fmap (fmap k)) m)

instance Applicative Buildable where
    pure x = Buildable (pure (empty, pure x))

    Buildable mL <*> Buildable mR = Buildable (liftA2 merge mL mR)
      where
        merge (stmL, foldL) (stmR, foldR) = (stm, fold)
          where
            stm = fmap Left stmL <|> fmap Right stmR

            fold = onLeft foldL <*> onRight foldR

instance Monoid a => Monoid (Buildable a) where
    mempty = pure mempty

    mappend = liftA2 mappend

instance (() ~ a) => IsString (Buildable a) where
    fromString = watch

onLeft :: Monad m => FoldM m i o -> FoldM m (Either i x) o
onLeft (FoldM step begin done) = FoldM step' begin done
  where
    step' x (Left  i) = step x i
    step' x (Right _) = return x

onRight :: Monad m => FoldM m i o -> FoldM m (Either x i) o
onRight (FoldM step begin done) = FoldM step' begin done
  where
    step' x (Left  _) = return x
    step' x (Right i) = step x i

run :: Buildable a -> IO ()
run (Buildable m) = Control.Monad.Managed.runManaged (do
    (stm, FoldM step begin _) <- m

    let go x = do
            e <- STM.atomically stm
            x' <- step x e
            go x'

    x0 <- liftIO begin
    liftIO (go x0) )

listen :: (a -> IO r) -> Buildable a -> Buildable a
listen handler (Buildable m) = Buildable (fmap adapt m)
  where
    adapt (stm, FoldM step begin done) = (stm, FoldM step' begin' done)
      where
        begin' = do
            x <- begin
            handler (done x)
            return x
        step' x e = do
            x' <- step x e
            handler (done x')
            return x'

watch :: FilePath -> Buildable ()
watch file = Buildable m
  where
    directory = System.FilePath.takeDirectory file

    m = do
        manager <- Control.Monad.Managed.managed System.FSNotify.withManager

        pwd <- liftIO System.Directory.getCurrentDirectory
        let predicate event = System.FSNotify.eventPath event == pwd </> file

        tQueue  <- liftIO (STM.atomically TQueue.newTQueue)
        let action (Added    _ _) = STM.atomically (TQueue.writeTQueue tQueue ())
            action (Modified _ _) = STM.atomically (TQueue.writeTQueue tQueue ())
            action (Removed  _ _) = return ()

        let io = System.FSNotify.watchDir manager directory predicate action
        Control.Monad.Managed.managed (Control.Concurrent.Async.withAsync io)

        return (TQueue.readTQueue tQueue, mempty)

... and here's an example of how you would use it:
baz :: Buildable ()
baz = listen handler ("foo.txt" <> "bar.txt")
  where
    handler _ = System.Process.system "cat foo.txt bar.txt > baz.txt"

test :: Buildable ()
test = listen handler "test.c"
  where
    handler _ = System.Process.system ("gcc -c test.c -o test.o")

main :: IO ()
main = run (baz <> test)

... which basically implements a continuous make system that watches your filesystem and rebuilds things automatically, and the above code is somewhat analogous to this make file:
baz.txt: foo.txt bar.txt
    cat foo.txt bar.txt > baz.txt

test.o: test.c
    gcc -c test.c -o test.o

Try to run the above code.  If you create a foo.txt and bar.txt while it is running then it will auto-generate baz.txt and if you create a test.c file it will automatically compile it to test.o object code.

Wow, Buildable is a great type!

But spawning a separate directory-watching manager thread for every single distinct path in your makefile seems... ill-advised =)

Yeah, this is just a proof of concept.  I'd typically recommend Nix as a build system these days

Unless, of course, you're on Windows. :/ Nix sorta works on Windows, but it's not usable enough to be recommended.


Minimizing the window results in incorrect size

After minimizing the window for a period of time, it may not be possible to restore the window, and restarting the software may result in a very small software size
thanks

This will probably help you
https://github.com/hecomi/uWindowCapture/issues/34


weird cancellation behavior

Describe the bug
I've a worker service on my .NET application which runs as long as application is running. From templates, I got to the following simplified code:
namespace Micro.KeyStore.Api.Workers
{
    public class CleanupKeysWorker : BackgroundService
    {
        private readonly ILogger<CleanupKeysWorker> _logger;
        private readonly IKeyRepository _keyRepository;
        private readonly IDriver<Key> _archiver;
        private readonly ArchiveKeysConfig _config;

        public CleanupKeysWorker(ILogger<CleanupKeysWorker> logger, IServiceScopeFactory serviceScopeFactory, IDriver<Key> archiver, IOptions<ArchiveKeysConfig> config)
        {
            _logger = logger;
            _archiver = archiver;
            _config = config.Value;
            _keyRepository = serviceScopeFactory.CreateScope().ServiceProvider.GetRequiredService<IKeyRepository>();
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                while (!stoppingToken.IsCancellationRequested)
                {
                    var keys = await _keyRepository.FindCreatedBefore(
                        DateTime.Now.Subtract(TimeSpan.FromMinutes(_config.TimeToLiveInMinutes)), 100); // 100 is batch size, so it'd return 100 keys
                    foreach (var key in keys)
                    {
                        // make a api request, for simulation, let's do a Thread.Sleep without cancellation token
                       Thread.Sleep(500);
                    }

                    _logger.LogInformation($"Archived {keys.Count()} keys at {DateTime.Now}");
                    // I also tried doing: if (stoppingToken.IsCancellationRequested) return;
                    await Task.Delay(TimeSpan.FromSeconds(_config.BatchIntervalInSeconds), stoppingToken);
                }
            }
            catch (OperationCanceledException e)
            {
                // do nothing
            }
        }
    }
}

To Reproduce
Steps to reproduce the behavior:
Run the above code, if we press Ctrl + C as soon as it logs Archived 100 keys at {Time}, everything works fine, but if we press Ctrl + C before that happens (while it's still inside the loop), it throws the following exception once it completes the current loop:
Unhandled exception. System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.CancellationToken.ThrowIfCancellationRequested()
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Micro.KeyStore.Api.Program.Main(String[] args) in /home/ngautam/git/personal/microservices/Micro.KeyStore/Micro.KeyStore.Api/Program.cs:line 24

Expected behavior
Since I have cancellation token, it should not throw exception, rather it should stop with 0 status code (also notice the commend where I simply did a return if cancellation was requested)
Screenshots

Additional context
I'm using .NET core 3

I don't think this is extensions specific, it's just Tasks and Cancellation and async.  @stephentoub should we move to dotnet/corfefx?

I don't think this is extensions specific, it's just Tasks and Cancellation and async

It sounds like the OP is requesting that Microsoft.Extensions.Hosting.Internal.Host.StopAsync shouldn't throw a cancellation exception when cancellation occurs, and instead do something else?  The exception is coming because (assuming I'm looking at the right code) because the extensions code is explicitly calling ThrowIfCancellationRequested:
https://github.com/aspnet/Extensions/blob/ecc4d9fe9381ac65848968b672eb6098ec2a2cae/src/Hosting/Hosting/src/Internal/Host.cs#L75
so if that's a change in behavior that's desired, it would need to be in this extensions code to not make the above call.

Thanks!  Adding @anurse to triage.

It's odd we're doing that. @Tratcher do you recall why that code got added?

It's trying to short circuit calling stop on all the services in case they don't honor cancelled tokens. The services will still get disposed. Note there's another call to ThrowIfCancellationRequested after the loop as well.
100 keys * 0.5s is 50 seconds, it's going to trip the default 5s shutdown timeout. Note the shutdown timeout token being tripped here is not the same one that's passed into ExecuteAsync. The ExecuteAsync token tripps when it's time to start shutting down. The sample above should honor this token to break it's foreach key loop.

do you mean to say on each loop I should check if cancellation was requested and quit accordingly? Because even when I check on for loop and return if it was requested, it still throws exception,

You spend 50s in this loop, you need to cancel that early.
                    foreach (var key in keys)
                    {
                        // make a api request, for simulation, let's do a Thread.Sleep without cancellation token
                       Thread.Sleep(500);
                    }

even when I do that, it still throws exception, before Thread.Sleep() I also tried checking if (stoppingToken.IsCancellationRequested) return;, but behaves the exact same way,

I'll try and verify again one more time maybe today evening or this weekend and update nevertheless,

This is expected behavior when your IHostedService runs past the shutdown deadline. We throw in Host.StopAsync because the timeout elapsed while the service was shutting down.
What's the desired behavior here?

well, even when I have ```
foreach (var key in keys)
{
// make a api request, for simulation, let's do a Thread.Sleep without cancellation token
Thread.Sleep(500);
if (stoppingToken.IsCancellationRequested) return;
}

it still throws error for some reason, I'll try to get a minimal reproduction example by this weekend, I forgot last weekend,

desired behavior would be to stop it without throwing an exception. even though it's less than 50 seconds,

We know why it's throwing, it's the line in Host.StopAsync here:
https://github.com/aspnet/Extensions/blob/ecc4d9fe9381ac65848968b672eb6098ec2a2cae/src/Hosting/Hosting/src/Internal/Host.cs#L75
That code is iterating through each service, calling their StopAsync and then moving on to the next one. Even when you return without throwing, that code will throw when the code moves on to the next service.
My feeling is that it's correct to throw here. In general, any method which takes an CancellationToken should be expected to throw OperationCancelledException (most, if not all, other places in .NET do so, in my experience). In this case, the service has taken too long to shutdown, which should be an exceptional situation that would be signaled to the user.
There's also the breaking change angle. Removing an exception may break users who were using this exception for further monitoring purposes (to notify them when a hosted service runs over it's shutdown deadline).

Closing as we're throwing by design here. Even if you terminate your hosted service's stop method without throwing, the Host's loop still throws when cancellation occurs in multiple places.


Automated load of talib and tulind if available locally

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Feature

What is the current behavior? (You can also link to an open issue here)

Users were required to modify configuration files to enable or disable TALIB or TULIP INDICATORS as well as using NPM to manage their installation.

What is the new behavior (if this is a feature change)?

Removed the need to modify config files to indicate the presence of TALIB and TULIND, instead their presence is detected automatically and they are loaded when available. This bring module management down to NPM installation.

Other information:

Was a bit of a thorn in switching between my active branches, and branches used for development, to ensure that all configuration files for test processes were using the correct settings.
Since there isn't any real cost to loading the indicator modules when unneeded, or a likely source of harm, and users can easily control their presence through NPM, this change seems like it will facilitate configuration for all users.
The core/talib.js and core/tulind.js scripts now detect when the libraries are missing by catching the exception thrown by require and export a null module, this is then used to validate if the indicators are available. If they are not an warning is printed to indicate that the indicators will not be available and execution resumes normally.

I've tested various combinations of having talib and tulind loaded, and trying to run indicators from those modules.

Loving the idea, though I think stuff like this will break / not work correctly anymore:
https://github.com/askmike/gekko/blob/c86ce48d539a5f45cab2f2b14b59743ccccb8719/plugins/tradingAdvisor/baseTradingMethod.js#L10-L14

If you also think we need to catch these situations before applying to PR let me know and I can do it as well :)

Apologies, you are very right! Thanks a ton :)


Update AutoComplete example to use WriteDisposition.APPEND

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @bjchambers
CC: @dhalperi

Jenkins is green; @bjchambers PTAL

BigQueryIO changes split off to: https://github.com/apache/incubator-beam/pull/451

LGTM

LGTM


Editor stops responding when I use Find

Steps to reproduce:

I open a certificate file (.p12) in Webstorm using the "Open as Hex" option under the "File" menu
The hex file opens properly
I use the Ctrl-F shortcut to open the Find box and begin typing text.
The editor stops responding and I have to shut it down using End Task.

I don't have Webstorm, but I tested, that there is the same issue in the most recent version of idea. I'll take look into it.

Thanks!