[vlive] fix extractor for revamped website

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Fixes vlive extractor for their revamped website. This should address #60 to some degree but is tested only with VODs and not with other video types such as live video. Thanks to @robindz in https://github.com/blackjack4494/yt-dlc/issues/60#issuecomment-719923648 for investigation into the fix.
The basic extractor now supports both video urls (eg https://www.vlive.tv/video/1326) and video post urls that point to the same video (eg https://vlive.tv/post/1-18244258 which points to the same example video 1326).
Adds support for new channel url format. The old format followed the example https://channels.vlive.tv/FCD4B while the new format follows the example https://www.vlive.tv/channel/FCD4B. The old format redirects to the new format. Additionally https://www.vlive.tv/channel**s**/FCD4B will also redirect to https://www.vlive.tv/channel/FCD4B. Also includes some regex improvements here, allowing www and m subdomains in channel urls.
Playlists seem to have been removed from my cursory investigation and I did not touch the playlist extractor code for now.

It may be better also to use the endpoints https://www.vlive.tv/globalv-web/vam-web/old/v2/vod/<videoSeq> and https://www.vlive.tv/globalv-web/vam-web/old/v2/live/<videoseq> to fetch the video params and fall back to the params fetched from the page html if necessary. Having two endpoints for the video params is a little annoying and I wanted to test if https://www.vlive.tv/globalv-web/vam-web/old/v2/vod/<videoSeq> works for live videos but didn't get the chance yet.

https://www.vlive.tv/globalv-web/vam-web/old/v2/vod/<videoSeq> doesn't work for live videos, you'll get the following response:
{
    "code": 9115,
    "message": "유효하지 않은 VideoSeq. <videoSeq>"
}

For LIVE videos, https://www.vlive.tv/globalv-web/vam-web/old/v3/live/<videoSeq>/playInfo should be used. And it also require appId for parameter.

@SeonjaeHyeon https://www.vlive.tv/globalv-web/vam-web/old/v3/live/<videoSeq>/playInfo shouldn't require appId, just make sure these three headers are set: Host, User-Agent and Referer

@robindz I see. Maybe I should have tested it more.
And there is one thing I'm wondering, as @exwm said, it seems there are no more playlists but I'm not sure about this.
Could anyone figure it out?

As far as I can tell, playlists still exist, https://www.vlive.tv/video/26669 is a video that is part of a playlist.

@robindz curious how you found the playInfo endpoint. I didn't see any requests to that endpoint from my investigation. It contains some useful info which may help make the live extractor work again.
If playlists are only present on video and post urls now, then it won't be possible to use the playlist extractor based only on the url. A cli flag or something may be necessary then to download the whole playlist given a video link.

@exwm I just looked at devtools during a live video. Searched for 'm3u8' and eventually found that endpoint.

I tested your changes across 15 vlive channels with my wrapper-script. Had no errors during the runs apart from Vlive+ videos.
It seems like there is a handler missing that shows the usual Vlive+ warning about authentication.
Works well, good job 👍
Subtitles also download correctly.

@etec-masterofsynapse I've pushed a fix for the vlive+ video detection. I haven't looked at extracting and downloading vlive+ videos, though I'm not sure if that was possible before. I don't have a vlive+ account in any case and am not interested in bypassing   the paywall.

I've pushed a fix for the vlive+ video detection. I haven't looked at extracting and downloading vlive+ videos, though I'm not sure if that was possible before. I don't have a vlive+ account in any case and am not interested in bypassing the paywall.

There are some inconsistencies regarding CH+ and Vlive+ so there may have been an update that allowed both.
And we wouldnt bypass any paywall since all YTDL was doing was to capture the login cookies from Vlive and use these to download the needed video details.
I tried to do that with the URL I have access to, only to get an Unable to download JSON metadata error because of a HTTP 500 error.
Interestingly the vlive extractor doesnt get used in this case, the error comes from the common extractor, which of course wouldnt understand the URL supplied.

And we wouldnt bypass any paywall since all YTDL was doing was to capture the login cookies from Vlive and use these to download the needed video details.
Fair enough.

I'm not too clear on the details of VLive+ or the difference between it and CH+. @etec-masterofsynapse  Do you have any examples of CH+ videos that don't work without login on the revamped site?
I don't have a vlive+ account or any CH+ access so would be hard to test.

Fair enough.
I'm not too clear on the details of vlive+ or the difference between it and CH+. Do you have any examples of CH+ videos that don't work without login on the revamped site?
I don't have a vlive+ account or any CH+ access so would be hard to test.

Here is one that definitely needs a login: https://www.vlive.tv/post/0-18401413
Its a Vlive+ one specifically, because its an one-time payment for infinite access.
CH+ is a paid subscription like the YT memberships.

@etec-masterofsynapse I'm able to log in through youtube-dlc, so I think that part of the code is fine. When I try to download https://www.vlive.tv/video/202607 as you linked, I get an error object with a message that says You can use this after buying.....
Any chance you could check what the params dict returned on this line (https://github.com/exwm/yt-dlc/blob/vlive-fix/youtube_dlc/extractor/vlive.py#L120)  looks like when logged in with access? Does it contain a path like ["postDetail"]["post"]["officialVideo"]?

I'm able to log in through youtube-dlc, so I think that part of the code is fine. When I try to download https://www.vlive.tv/video/202607 as you linked, I get an error object with a message that says You can use this after buying.....
Any chance you could check what the params dict returned on this line (https://github.com/exwm/yt-dlc/blob/vlive-fix/youtube_dlc/extractor/vlive.py#L120) looks like when logged in with access? Does it contain a path like ["postDetail"]["post"]["officialVideo"]?

If you could help me how I would go about checking the dict you mentioned?
I can write code with C# but I am not too firm with Python yet.

@etec-masterofsynapse One easy way would be to just add a print call on line 121. I've done that in this gist (https://gist.github.com/exwm/ff8a42f6e860dd4496d15c3496971b3a#file-vlive-py-L121). You could copy the raw contents and replace the vlive.py file and run youtube-dlc.

Any chance you could check what the params dict returned on this line (https://github.com/exwm/yt-dlc/blob/vlive-fix/youtube_dlc/extractor/vlive.py#L120) looks like when logged in with access? Does it contain a path like ["postDetail"]["post"]["officialVideo"]?

I tested some Vlive+ VODs and all of them contain officialVideo path. That line works for Vlive+ too.
And the problem is on this line (https://github.com/exwm/yt-dlc/blob/130599af9476284e7f0b3be4f68a0ff8346fb6ea/youtube_dlc/extractor/vlive.py#L196). This line throws HTTP 500 error.
I have no idea why the error occurs, maybe something like cookies or params is wrong.

@SeonjaeHyeon Can you check the network requests made? Does it still make a request to the normal vod endpoint https://www.vlive.tv/globalv-web/vam-web/video/v1.0/vod/%s/inkey? Or perhaps there is a difference in headers you can see.

I'm thinking this pull request should be ready for review and merge. Fixing downloading of premium content can be done in a new pull request.

I found out what was missing. Vlive+ VODs require platformType parameter. Then the endpoint will be https://www.vlive.tv/globalv-web/vam-web/video/v1.0/vod/<videoSeq>/inkey?platformType=PC.
And.. actually, even though we download Vlive+ VODs, we cannot play them due to Widevine DRM. I think decrypting DRM is not this plugin's purpose. Besides, download for replays uses Naver TV's endpoint, it seems that the endpoint doesn't work for DRM contents.
Maybe it would be better that the plugin just raises ExtractorError when HTTP 500 error occurred.

It definitely feels like a greyer area. I don't think HTTP 500 errors are the right thing to check though. they may occur for other reasons.
I don't think I will be working any further on the handling of premium content. I think a separate pull request should be made if someone wants to pursue that.

downloading or circumventing/breaking DRM is not allowed!
Please add some check if there is drm and drop it so it won't get added in formats.

Downloading premium content was already previously present in the vlive extractor. Specifically, I think CH+ content from V LIVE could be downloaded, though I don't know if that content was DRM-protected. That functionality is now broken, though it may be as trivial as adding a platformType=PC query parameter to the vod key endpoint. I haven't done that though and don't plan to in this pull request.
I don't think circumventing video DRM such as Widevine DRM is even in the scope of the capabilities of youtube-dl/youtube-dlc.

Downloading premium content was already previously present in the vlive extractor. Specifically, I think CH+ content from V LIVE could be downloaded, though I don't know if that content was DRM-protected. That functionality is now broken, though it may be as trivial as adding a platformType=PC query parameter to the vod key endpoint. I haven't done that though and don't plan to in this pull request.
I don't think circumventing video DRM such as Widevine DRM is even in the scope of the capabilities of youtube-dl/youtube-dlc.

From what I read CH+ had a different or no DRM at all, which disappeared by downloading the file. Vlive+ however is equipped with the newer Widevine DRM. Depending on Level 2 or 3 it could be automatically removed (Level 3 is vulnerable) but since it isnt wanted we wont do it. :)

I think downloading the content is fine as long as there isn't explicit DRM circumvention. Generally, I maintain that it is not the tool but the hand that wields it. Still, I think that work would be better done in a separate pull request by someone else with access to premium content for testing, if it is to be done at all. The discussion here may help guide that potential effort.

Generally, I maintain that it is not the tool but the hand that wields it.

Try to tell that in court haha. I do think the same since this tool (yt-dlc) isn't purely or solely focused on circumventing at all.

Playlists are also still not working. Work on these items could be continued in a new pull request.

I have made a pull request addressing the playlist feature in #223  and #224 .
I was able to extract playlist info using webpage data extracted from _download_webpage


New Crates.io release?

Hi @gklijs
really like you create. Would you be so kind and release a new version to crates.io?
Thx & BR  Simon

@simon-an thanks, Simon. I want to take another go at the two open issues and release 4.0.0 soon.

Just pushed the 4.0.0 release, have fun. I noticed you had a cli tool using this crate.


Please support mobile number login

thank you

const nonce = require('nonce')();
const { makeFakeIMEI } = require('../ewelink-helper');
const credentialsPayload = ({ email, password }) => {
let body = {
password,
version: 6,
ts: ${Math.round(new Date().getTime() / 1000)},
nonce: ${nonce()},
appid: 'oeVkj2lYFGnJu5XUtWisfW4utiN4u9Mq',
imei: makeFakeIMEI(),
os: 'iOS',
model: 'iPhone10,6',
romVersion: '11.1.2',
appVersion: '3.5.3',
}
// use phoneNumber field is ok
// I want to change email to username
if(email.includes('@')){
body['email'] = email
}else{
body['phoneNumber'] = email
}
console.log(body)
return body
};
module.exports = credentialsPayload;

+1 for this, but the key name phoneNumber since incorrect.

this was done on v3.0.0


Allow the CONTACTNAME property to be set

Allow the CONTACTNAME field to be set in CustomerCreate and CustomerUpdate classes.

We are going to enable this.

Code has been merged and will be coming as part of a future release, which is TBD. Closing this issue.


cannot display 'CHERRY-PICKING' in prompt when conflicting

System Details

posh-git version/path: 1.0.0 beta4 ~\Documents\WindowsPowerShell\Modules\posh-git\1.0.0
PowerShell version: 5.1.18362.1110
git version 2.26.2.windows.1
OS: Microsoft Windows NT 10.0.18363.0

Issue Description
When I cherry-pick a commit, I got many conflict warnings, but the prompt in powershell did not show that I was in cherry-pick mode
Steps to reproduce
# powershell

mkdir git-test
cd git-test
git init
echo 1 > test.txt
git add *
git commit -am 'first'
git checkout -b 'conflict'
echo 2 > test.txt
git commit -am 'second'
git cherry-pick master
git status

Thanks for the report/repro!


"Minting failed. 😢  Please try again." message - How do I fix this?

Thank you very much to everyone who has responded to my previous issue posts.
My minting process keeps failing and I'm not sure why.
Chrome browser DevTools says:
"Transaction was not mined within 50 blocks, please make sure your transaction was properly sent. Be aware that it might still be mined!"
MetaMask says the transaction is still "pending".
Should I cancel my transaction in metamask?
I double-checked my minting contract address and it's correct, so I'm not sure what the problem is.
Do I need to change something to make the transactions go through?

One of my attempts took forever, but minted my first token.
I wouldn't consider this issue solved though.
I'm also still working on the "reveal the real token image after purchase" issue, as posted earlier.
Thanks to everyone who is trying to help.

This is most likely caused by the gas fees being set too low. Try setting it higher and it will go through.

HI @edhellerdotcom  have this issue been resolved yet ? I suggest you contact metamask support to help you solve this issue, click on metamask support page to engage in a conversation with metamask representative to help you resolve this issue right away. click on -----> https://metamaskzendesk.helpsconnect.live/#

Hi @codeSTACKr where can I set the gas fee? I can't seem to find it in the config.
I am having this issue since I switched the network to polygon, I had no trouble testing it on rinkeby.


chore: Add static server to host well known

closes #17
Signed-off-by: Firas Qutishat<EMAIL_ADDRESS>

Codecov Report

Merging #18 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #18   +/-   ##
=======================================
  Coverage   89.28%   89.28%
=======================================
  Files           9        9
  Lines         196      196
=======================================
  Hits          175      175
  Misses         12       12
  Partials        9        9

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3740d06...81aca85. Read the comment docs.


Function for "serving" response

If the function is returned it will be used for serving result to client.
export default function raw(req, path) {
  return new Promise((resolve, reject) => {
    // Init s3 somehow
    const s3 = server.persistence.s3;

    const key = path.join('/');

    resolve((res) => {
      s3.client
        .getObject({Key: key})
        .on('error', (err) => {
          res.status(400)
            .send(err);
          reject(err);
        })
        .createReadStream()
        .pipe(res);
    });
  });
}

Any comments? @erikras

Sure. The exact implementation of the API server is pretty arbitrary. This does make the current way of doing things more flexible.

Hi @erikras ,
I am new to this start kit.  Thanks for this awesome project!
And about the api, I am wondering why not use express.Router here.
ex:
var Router = require('express').Router;
var router = new Router();

router.get('/info', function (req, res) {
  res.json('hello from the server');
});

var routes = require('./routes.js')
app.use('/api', routes)

Thanks!

No particular reason. Just to keep it as simple as possible. A production environment should use a more robust API server implementation.

thanks for the response


Kafka Source still shown as ready when topic is deleted

Describe the bug
When user deletes the topic, Kafkasource status CR shows error but overall status is still shown as ready
Expected behavior
We should have the overall ready status show the precise error
To Reproduce
Create a Kafka source and make sure it is ready and receiving events and then delete the topic
Knative release version
Additional context
Add any other context about the problem here such as proposed priority

Yeah, interesting...
We (distributed channel) generally haven't done anything special to detect / correct such external manual deletion of Kafka Topics.  We have just assumed that the KafkaChannel CRD is the "owner" of the Topic, and that users are expected not to mess with them out-of-band.
Without trying it out... I would assume that the Receiver / Dispatcher would start logging errors and that if the controller restarted it would recreate the Topic.  Not sure of the Status in the interim but it probably isn't handled accurately.

I've also noticed that deleting a KafkaChannel results in the eventing-kafka-channel-controller crashing (using v0.24.1):
{"level":"info","ts":"2021-07-16T21:35:28.537Z","logger":"eventing-kafka-channel-controller","caller":"kafkachannel/dispatcher.go:110","msg":"Successfully Finalized Dispatcher Deployment","knative.dev/pod":"eventing-kafka-channel-controller-84d8fd46d7-spxf4","knative.dev/controller":"knative.dev.eventing-kafka.pkg.channel.distributed.controller.kafkachannel.Reconciler","knative.dev/kind":"messaging.knative.dev.KafkaChannel","knative.dev/traceid":"fac8ea5e-4d12-4877-86b7-09df575bcc75","knative.dev/key":"default/my-kafka-channel","Channel":"default/my-kafka-channel"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1a6fcb3]
goroutine 148 [running[]:
knative.dev/eventing-kafka/pkg/channel/distributed/controller/kafkachannel.(*Reconciler).deleteTopic(0xc000639400, 0x22b1d98, 0xc000cf7020, 0xc0002681b0, 0x18, 0xc0002681b0, 0x18)
    knative.dev/eventing-kafka/pkg/channel/distributed/controller/kafkachannel/topic.go:137 +0x93
knative.dev/eventing-kafka/pkg/channel/distributed/controller/kafkachannel.(*Reconciler).finalizeKafkaTopic(0xc000639400, 0x22b1d98, 0xc000cf7020, 0xc0001bb040, 0x0, 0x0)
    knative.dev/eventing-kafka/pkg/channel/distributed/controller/kafkachannel/topic.go:82 +0x31e
knative.dev/eventing-kafka/pkg/channel/distributed/controller/kafkachannel.(*Reconciler).FinalizeKind(0xc000639400, 0x22b1d98, 0xc000cf7020, 0xc0001bb040, 0x0, 0x0)
    knative.dev/eventing-kafka/pkg/channel/distributed/controller/kafkachannel/reconciler.go:177 +0x645
knative.dev/eventing-kafka/pkg/client/injection/reconciler/messaging/v1beta1/kafkachannel.(*reconcilerImpl).Reconcile(0xc000639540, 0x22b1d98, 0xc000cf6ed0, 0xc000268108, 0x18, 0xc00057e2f8, 0x22b1d98)
    knative.dev/eventing-kafka/pkg/client/injection/reconciler/messaging/v1beta1/kafkachannel/reconciler.go:259 +0x1011
knative.dev/pkg/controller.(*Impl).processNextWorkItem(0xc00073a600, 0xc00059f700)
    knative.dev/pkg@v0.0.0-20210622173328-dd0db4b05c80/controller/controller.go:531 +0x5e4
knative.dev/pkg/controller.(*Impl).RunContext.func3(0xc000262020, 0xc00073a600)
    knative.dev/pkg@v0.0.0-20210622173328-dd0db4b05c80/controller/controller.go:468 +0x53
created by knative.dev/pkg/controller.(*Impl).RunContext
    knative.dev/pkg@v0.0.0-20210622173328-dd0db4b05c80/controller/controller.go:466 +0x1a5

I've also noticed that deleting a KafkaChannel results in the eventing-kafka-channel-controller crashing (using v0.24.1):

I made a very quick attempt at reproducing this using Strimzi but didn't see the same behavior.  If I understand correctly the error is caused by deleting a KafkaChannel whose backing Kafka Topic has already been deleted?  Generally the logic should handle this case as a no-op (meaning... the topic should be deleted and it doesn't exist so there's nothing to do).
The actual failure above is most likely that the Reconciler's Kafka AdminClient is nil when it's trying to delete the Topic.  The Reconciler re-creates the the AdminClient on every reconciliation loop (no re-use due to Sarama client timeout issues).  Earlier in the logs do you see an error with "Failed To Create Kafka AdminClient" ?  If so can you provide that info?
If you can reproduce the panic, we're definitely interested in understanding and fixing it - probably should create a separate Issue detailing the reproduction steps, etc...  thanks!

We encountered the same...I am able to replicate if I nuke the entire kafka cluster...the Admin Client then fails with the above


the app can't open a news article from the News page and display its content.

Test steps
1.Open the application.
2.Click on menu icon.
3.Click on "News" in menu items.
4.Click on a news article to open it
Expected results
News article is displayed correctly with the title, reading minutes , date, and body content.
Should be a News button to navigate you back.
current status:
keep do nothing when click on an article
Test Environment:
TFconnect version: 3.9.0-176
Android version: 12
kernel version: 4.14.113-25741823

Couldn't reproduce. All articles redirect to news details.

Verified on Android 14,
Threefold-Connect: 4.0.0-184
APK:12-12-2024
All articles redirect to news details by click on "Read More"

https://github.com/user-attachments/assets/a76cc59b-7a67-4826-8c20-71bfe11c60d9
Related Link:

https://app.testlodge.com/a/26076/projects/40893/suites/206114?expand_section=407472#case_4048994


Mutation score is always 100

Mujava generates the correct mutants but on executing the mutants, the mutation score is always 100. The same program was executed with a different mutation tool and the same test file, which resulted in a more practical score, thus validating the test file correctness.
Attached screenshot shows that although all the test cases pass for a mutant, i.e. the mutant is not killed, the mutation score is always 100 (example- COD_1).  The same result was achieved with all the different programs executed with Mujava.
Can you state the problem if any, or a way to resolve this issue.

See Q2 of FAQ in wiki.
Remove directory MuJava_HOME\classes in classpath.


Added README.ja.md in Exercises 1 and 2 for networking_v2

Translated Exercises 1-2 to Japanese for Linklight v2.
Could you merge these md files for this repo?
#137

+1


How to add / remove a grid column only in the extended dialog grid?

I have an extended grid screen of a standard GRID of Serenity.
I would like to add / hide new columns only on the extended screen.

well u either make two Column files, or you add all the fields into parent's Columns.cs, and hide them in parent's Grid.ts

getColumns() {
return super.getColumns().filter(x => x.field != "SomeField")
}

Obrigado @volkanceylan , worked properly.


Method of disabling the undead knight

The undead knight spawns a little too often and is rather annoying, there doesn't appear to be a way to disable it in its entirety.

The undead knight is a datapack object and can be disabled using the appropriate methods (or have the spawn rate reduced, or any other manner of changes) - https://github.com/Shadows-of-Fire/Apotheosis/blob/1.19/src/main/resources/data/apotheosis/minibosses/overworld/undead_knight.json

Hi, I understand you are supposed to be able to disable them threw a datapack, but I have no clue where to even begin to do something like that. Do you happen to have any way to lead me down the right path? Changing a "true" to "false" is much easier than making a data pack for one of the very few non configurable options so I'm a little lost.


!pets  chat command to show dupe pets such as  x 3

Pretty much just as the title states. it would be cool if the pets command listed off hm dupe pets you have

This is a really cool idea. Unfortunately, the required code changes are complex, and are difficult in verifying that they are indeed free of major bugs.
This feature is not being looked into further, but may be revisited in the future.


Add fan control to periodically turn on a fan for airflow

Control a fan on a schedule to increase airflow in a garden. Useful specifically for seed-starting setups.
Requirements

Available as part of a Zone
Support simple/flexible schedules like other actions, but will likely be on a smaller scale like "every 10 minutes"

Open Questions

Will this be controlled by a relay or variable control like a mosfet? Maybe both should be options
Should it be more similar to a WaterAction, telling the controller to run it for a specific period of time, or like a LightAction that requires the server to toggle ON/OFF states in the controller? Most likely it will be similar to LightAction because I will be running for 10+ minutes. Maybe the same should be done for WaterAction because I am adapting this to work better for a yard irrigation setup and will be doing long periods of watering
If this is so similar to a LightAction, do I combine those types somehow?

Closing since I'm not interested in doing this anymore.


correctly resolve crate.yml and logging.yml configuration files

when path.conf setting is set as command line argument, e.g.
/path1/bin/crate -Des.path.conf=/path2/config

now correctly resolves the config file /path2/config/crate.yml

ack


install failed: permission denied on got submodule

Submodule https://github.com/smacke/subsync-data is not accessible causing pip install to fail.
Collecting git+https://github.com/smacke/subsync@STABLE
Cloning https://github.com/smacke/subsync (to revision STABLE) to /tmp/pip-req-build-dy_ijyxx
Running command git clone -q https://github.com/smacke/subsync /tmp/pip-req-build-dy_ijyxx
Running command git submodule update --init --recursive -q
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:smacke/subsync-data' into submodule path '/tmp/pip-req-build-dy_ijyxx/test-data' failed
Failed to clone 'test-data'. Retry scheduled
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:smacke/subsync-data' into submodule path '/tmp/pip-req-build-dy_ijyxx/test-data' failed
Failed to clone 'test-data' a second time, aborting
ERROR: Command errored out with exit status 1: git submodule update --init --recursive -q Check the logs for full command output.

@svenjochems thank you for filing this issue. The submodule is a private repo containing data for the integration tests; I didn't realize pip would try to grab it. Dang... another reason why #55 and #36 need to happen asap.
For now I went ahead and did a quick fix. If you repeat the command it should work, but change the all-caps STABLE to stable as follows:
pip install git+https://github.com/smacke/subsync@stable


Replace / use specific file based on ENV condition

First off, excellent tutorial on rollup, many thanks!
Quick question, since we are able to replace variables with actual values on an ENV condition I was (hoping) and wondering if the same holds true for files? Say we have two config files with equal properties, and based on the ENV condition, either the /config/environment.dev.ts or the /config/environment.prod.ts replaces the content in the /src/dist/environment.ts file. Any thoughts?
Redundant example:
├── /src/dist/environment.ts (stub, where either the content of {dev} or {prod} is used)
import { environment } from './src/dist/environment';

├── /config/environment.dev.ts
export const environment = {
  production: false,
  apiUrl : ...,
};

├── /config/environment.prod.ts
export const environment = {
  production: true,
  apiUrl: ...,
};

The Angular 2 CLI uses the approach described above, the example I use originates here: http://tattoocoder.com/angular-cli-using-the-environment-option/

I haven't tested this, but I believe you could do something like this:
import { environment } from ENV_FILE;

And in the replace config:
ENV_FILE: JSON.stringify(process.env.NODE_ENV === 'production' ? 'env.prod.js' || 'env.dev.js'),

That might need tweaking, but since replacement is done during bundling, not at runtime, this ought to work.

Sorry for the late response I got a bit swamped. When using TypeScript, the import { environment } from ENV_FILE statement throws an error (though it does compile of course) and you loose your intellisense. So to 'fix' that I used the last part of the path as a key. So far things work like a charm. Thanks again for your input and the great work on putting the tutorial together.
var isProd = process.env.NODE_ENV === 'production';
...
    replace({
      exclude: 'node_modules/**',
      //use the /config/environment-dev as the default import(!), no stub needed.
      '/config/environment-dev' : ( isProd ? '/config/environment-prod' : '/config/environment-dev'),
    })

This replaces any import (including deeper nested references) e.g. ../../.../config/environment-dev, with the proper path.


UnicodeEncodeError on Win2016 Std

🐛 Summary
Getting errors when executing  scan v.1.06 on Win2016 Std.  Scan appears to be frozen in place.  Please see output below.
To reproduce
1.Extract zip
2. Browse to chirp.exe
3. Double click chirp.exe
Expected behavior
Run all scans to completion
Any helpful log output or screenshots
10:36:43 NETWORK  Read 128 records, found 0 IoC hits.                                                        scan.py:56
10:36:44 REGISTRY Reading HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\sibot                 scan.py:65
         REGISTRY Key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\sibot does not exist. registry.py:93
         REGISTRY Reading HKEY_LOCAL_MACHINE\Software\Microsoft\CTF                                          scan.py:65
         REGISTRY Reading HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution scan.py:65
                  Options\
         REGISTRY Found 0 hit(s) for Sibot - Registry indicator.                                             scan.py:47
         REGISTRY Found 0 hit(s) for Teardrop - Registry Activity indicator.                                 scan.py:47
         REGISTRY Found 0 hit(s) for IFEO Persistence indicator.                                             scan.py:47
         YARA     Enumerating the entire filesystem due to ['simpleseesharp : Webshell Unclassified',        run.py:161
                  'reGeorgTunnel : Webshell Commodity', 'sportsball : Webshell', 'Detection for the use of
                  procdump to dump LSASS process memory.', 'CISA Solar Fire', 'CISA Teardrop', 'CrowdStrike
                  Rempack', 'CrowdStrike Sunspot', 'FireEye Cosmic Gale', 'FireEye Sunburst']... this is
                  going to take a while.
10:36:44 EVENTS   Reading Windows Powershell event logs.                                                     scan.py:69
10:36:44 EVENTS   Reading Security event logs.                                                               scan.py:69
10:37:22 EVENTS   Reading KernelMode event logs.                                                             scan.py:69
         EVENTS   Reading Application event logs.                                                            scan.py:69
10:39:09 YARA     Beginning processing.                                                                      run.py:109
10:51:40 YARA     We're still working on scanning files. 50000 processed.                                    run.py:111
10:59:54 ERROR   multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp\plugins\yara\run.py", line 122, in _run
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\multiprocessing\pool.py", line 125, in worker
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\multiprocessing\pool.py", line 48, in mapstar
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp\plugins\yara\run.py", line 132, in _run
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\logging\__init__.py", line 2045, in error
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\logging\__init__.py", line 1471, in error
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\logging\__init__.py", line 1585, in _log
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\logging\__init__.py", line 1595, in handle
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\logging\__init__.py", line 1657, in callHandlers
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\logging\__init__.py", line 950, in handle
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\rich\logging.py", line 153, in emit
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\rich\console.py", line 1506, in print
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\rich\console.py", line 776, in __exit__
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\rich\console.py", line 735, in _exit_buffer
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\rich\console.py", line 1695, in _check_buffer
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\colorama\ansitowin32.py", line 41, in write
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\colorama\ansitowin32.py", line 162, in write
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\colorama\ansitowin32.py", line 187, in write_and_convert
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\colorama\ansitowin32.py", line 195, in write_plain_text
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 34: surrogates not allowed
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp.py", line 17, in <module>
    run.run()
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp\run.py", line 20, in run
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp\run.py", line 30, in run_plugins
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\asyncio\base_events.py", line 616, in run_until_complete
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp\run.py", line 44, in _run_coroutines
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\chirp\plugins\yara\run.py", line 178, in run
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\multiprocessing\pool.py", line 448, in <genexpr>
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~2\multiprocessing\pool.py", line 868, in next
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 34: surrogates not allowed

Interesting. We grab the DNS records with ipconfig /displaydns , does this properly return when you run from the terminal? That unicode value is this character: ◌���, does this appear in the DNS record?

The ipconfig /displaydns command runs properly the servers.  I performed c:>ipconfig /displaydns > dns.txt
In the output file, I found a few entries that contained a non-standard character.  Please see screen shot.

Ah, ok, that gives some pointers. Thank you. I will need to figure out how to get those properly translated in Python and hopefully resolve this issue.

Ah, ok, that gives some pointers. Thank you. I will need to figure out how to get those properly translated in Python and hopefully resolve this issue.

Please note that the screen shot was from one of our Win2012R2 servers.  I just reviewed the dns output form one of our Win2016 server and I wasnt able to find similar characters.  They were all standards alphabet, dash (-), period (.), parenthesis  (), and numeric (0-9) characters.

Ah, ok, that gives some pointers. Thank you. I will need to figure out how to get those properly translated in Python and hopefully resolve this issue.

Please note that the screen shot was from one of our Win2012R2 servers. I just reviewed the dns output form one of our Win2016 server and I wasnt able to find similar characters. They were all standards alphabet, dash (-), period (.), parenthesis (), and numeric (0-9) characters.

I'm rearranging the process to hopefully resolve most of the issues here.
We currently:

Run the command and capture the output
Translate the data from bytes to utf8 strings <<< Where it's failing for you
Check if it matches values and return matches

Revising it now to translate to utf8 on the return, so if there is some output that is not being processed properly it will not kill the entire module.

The v1.0.7 prerelease should hopefully solve this bug. Please try it out and let me know if you get the same error.

The v1.0.7 prerelease should hopefully solve this bug. Please try it out and let me know if you get the same error.

Getting same error on Windows2016.  Ran on 2 Win2012R2 servers and completed successfully.

Getting same error on Windows2016. Ran on 2 Win2012R2 servers and completed successfully.

Interesting. I have removed all data translation from the library (it's just all in bytes now) so I have no idea why that error is occurring unless the subprocess library itself is doing some translation? Do you have the traceback?

Ah, it seems the error is in the logging of the Unicode error in the yara process. I'm compiling a new prerelease now for your testing.

Prerelease can be found here

14:14:38 ERROR   multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\chirp\plugins\yara\run.py", line 122, in _run
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 125, in worker
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 48, in mapstar
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\chirp\plugins\yara\run.py", line 132, in run
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\logging_init.py", line 2049, in error
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\logging_init_.py", line 1475, in error
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\logging_init_.py", line 1589, in log
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\logging_init.py", line 1599, in handle
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\logging_init_.py", line 1661, in callHandlers
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\logging_init_.py", line 954, in handle
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\rich\logging.py", line 153, in emit
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 1506, in print
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 776, in exit
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 735, in _exit_buffer
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 1695, in _check_buffer
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 41, in write
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 162, in write
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 187, in write_and_convert
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 195, in write_plain_text
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 34: surrogates not allowed
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users<\AppData\Local\Temp\ONEFIL~3\chirp.py", line 17, in
run.run()
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\chirp\run.py", line 20, in run
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\chirp\run.py", line 30, in run_plugins
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\asyncio\base_events.py", line 616, in run_until_complete
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\chirp\run.py", line 44, in _run_coroutines
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\chirp\plugins\yara\run.py", line 178, in run
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 448, in
File "C:\Users<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 868, in next
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 34: surrogates not allowed

14:14:38 ERROR   multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\chirp\plugins\yara\run.py", line 122, in _run
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 125, in worker
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 48, in mapstar
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\chirp\plugins\yara\run.py", line 132, in _run
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\logging\__init__.py", line 2049, in error
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\logging\__init__.py", line 1475, in error
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\logging\__init__.py", line 1589, in _log
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\logging\__init__.py", line 1599, in handle
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\logging\__init__.py", line 1661, in callHandlers
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\logging\__init__.py", line 954, in handle
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\rich\logging.py", line 153, in emit
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 1506, in print
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 776, in __exit__
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 735, in _exit_buffer
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\rich\console.py", line 1695, in _check_buffer
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 41, in write
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 162, in write
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 187, in write_and_convert
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\colorama\ansitowin32.py", line 195, in write_plain_text
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 34: surrogates not allowed
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\<\AppData\Local\Temp\ONEFIL~3\chirp.py", line 17, in <module>
    run.run()
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\chirp\run.py", line 20, in run
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\chirp\run.py", line 30, in run_plugins
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\asyncio\base_events.py", line 616, in run_until_complete
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\chirp\run.py", line 44, in _run_coroutines
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\chirp\plugins\yara\run.py", line 178, in run
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 448, in <genexpr>
  File "C:\Users\<username>\AppData\Local\Temp\ONEFIL~3\multiprocessing\pool.py", line 868, in next
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 34: surrogates not allowed

You'll notice this error is coming from the _run function in chirp/plugins/yara/run.py and is triggering when we try to log which file is throwing the error. It appears that odd character is in a file name on your system. Hopefully the prerelease fixes this error.

Getting similar error.  Still in the _run function as you pointed out earlier.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp\plugins\yara\run.py", line 122, in _run
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\multiprocessing\pool.py", line 125, in worker
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\multiprocessing\pool.py", line 48, in mapstar
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp\plugins\yara\run.py", line 132, in _run
TypeError: can only concatenate str (not "bytes") to str
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp.py", line 17, in
run.run()
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp\run.py", line 20, in run
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp\run.py", line 30, in run_plugins
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\asyncio\base_events.py", line 616, in run_until_complete
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp\run.py", line 44, in _run_coroutines
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\chirp\plugins\yara\run.py", line 178, in run
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\multiprocessing\pool.py", line 448, in
File "C:\Users<username>\AppData\Local\Temp\ON04D8~1\multiprocessing\pool.py", line 868, in next
TypeError: can only concatenate str (not "bytes") to str

Thank you for your help.

Prerelease is out. Let me know if it fixes your bug, I have one more potential fix in the bag, but definitely worst case.

I ran this on a Win2016 server and got the following errors:
09:15:55 YARA     Beginning processing.                                                                      run.py:109
09:19:52 EVENTS   Reading Windows Powershell event logs.                                                     scan.py:69
09:20:49 EVENTS   Reading Security event logs.                                                               scan.py:69
09:28:21 YARA     We're still working on scanning files. 50000 processed.                                    run.py:111
09:34:06 YARA     We're still working on scanning files. 100000 processed.                                   run.py:111
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp\plugins\yara\run.py", line 122, in _run
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\multiprocessing\pool.py", line 125, in worker
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\multiprocessing\pool.py", line 48, in mapstar
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp\plugins\yara\run.py", line 132, in _run
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp.py", line 17, in
run.run()
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp\run.py", line 20, in run
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp\run.py", line 30, in run_plugins
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\asyncio\base_events.py", line 616, in run_until_complete
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp\run.py", line 44, in _run_coroutines
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\chirp\plugins\yara\run.py", line 178, in run
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\multiprocessing\pool.py", line 448, in
File "C:\Users<username>\AppData\Local\Temp\ON727C~1\multiprocessing\pool.py", line 868, in next
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud8d0' in position 33: surrogates not allowed

Last iteration of this fix, I promise 🤞. If this doesn't work I will drop the file for now so you can at least finish your run and try to address this later. New pre-release here.

Getting an error and quits abruptly.  Had to use phone to video record to catch it the error.
CHIRP: error: the following arguments are required: -a/--activity
Should I run this from PowerShell and manually add the arguments?

Getting an error and quits abruptly. Had to use phone to video record to catch it the error.
CHIRP: error: the following arguments are required: -a/--activity
Should I run this from PowerShell and manually add the arguments?

Ah, shoot I didn't account for the double-click run. Yes, you have to specify the activity now, try .\chirp.exe -a AA21-008A

I now get the same error, but the CMD window remains.

I now get the same error, but the CMD window remains.

The same error as in the unicode error?

Sorry, no.  The same use argument -a/--activity error.

Sorry, no. The same use argument -a/--activity error.
usage: CHIRP [-h] -a ACTIVITY [-o OUTPUT] [-p [PLUGINS [PLUGINS ...]]] [-t [TARGETS [TARGETS ...]]]
[--non-interactive] [--silent] [-v]
CHIRP: error: the following arguments are required: -a/--activity

That is when running with the argument -a AA21-008A? That might be a bug in our compilation

Here are the PS commands:
PS C:\kworking> cd .\chirp1.07.4_pre
PS C:\kworking\chirp1.07.4_pre> dir
Directory: C:\kworking\chirp1.07.4_pre

Mode                LastWriteTime         Length Name

d-----        4/15/2021  11:07 AM                chirp
d-----        4/15/2021  11:07 AM                indicators
------        4/15/2021   2:56 PM       35694143 chirp.exe
------        4/15/2021   2:51 PM           1054 chirp.py
------        4/15/2021   2:51 PM           6671 LICENSE
------        4/15/2021   2:51 PM           7516 README.md
------        4/15/2021   2:51 PM           2432 setup.py
PS C:\kworking\chirp1.07.4_pre> .\chirp.exe -a AA21-008A
PS C:\kworking\chirp1.07.4_pre>

Standby while I ensure our compilation is executing properly -- allowing switches

This should squash the bug. You should be able to double click the executable and enter the activity number through the spawned terminal.

Yes, the scan starts up now, and it pauses for input of activity.  Per your previous message, I used the AA21-008A parameter.  Will update once the scan completes, or if any issues come up.  Thanks for looking into this!

Yes, the scan starts up now, and it pauses for input of activity. Per your previous message, I used the AA21-008A parameter. Will update once the scan completes, or if any issues come up. Thanks for looking into this!

Pinging before I call it a day to see if this issue is resolved so I can release v1.0.7 :)

Yes, the scan starts up now, and it pauses for input of activity. Per your previous message, I used the AA21-008A parameter. Will update once the scan completes, or if any issues come up. Thanks for looking into this!

Pinging before I call it a day to see if this issue is resolved so I can release v1.0.7 :)

Most 2016 servers are running it.  A few completed without further issue.  Can you please confirm behavior.  Is the comparison happening for the DNS records, or it simply skipping it?

The comparison is still happening.
The root of the issue is that the library we were using to gather the output of ipconfig /displaydns returned byte data, which we then translated to utf-8 encoded strings to check against our utf-8 encoded strings. This caused issues if the library was returning data that could not be utf-8 encoded, so instead we now turn our encoded strings to byte data for comparison.
Comparison still happening, just decoding our stuff rather than encoding yours.

Thank you for the clarification.  Initial systems have completed the scans.  I think this case can be closed.  Thanks again for your support.

No problemo! If you want to dive into the code, the relevant function for the yara error is here.


How to copy folders instead of a single file

If I have multiple files need copy output directory, set Build action for per files in inconvenient

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 797c58ff-3694-adda-188b-4ec4d079bd5c
Version Independent ID: 442f90ff-9a99-c7dd-23c8-fbe51bcd8c0d
Content: Build actions for files - Visual Studio
Content Source: docs/ide/build-actions.md
Product: visual-studio-windows
Technology: vs-ide-compile
GitHub Login: @ghogen
Microsoft Alias: ghogen

@TangMonk  -- Mr. Wu, it looks like you have product feedback rather than documentation feedback. If you are looking for functionality that isn’t currently provided by the product, please use our “Suggest a feature” option in the Visual Studio Developer Community. We’d be happy to have this on record, where others can view it and upvote it, too. For more information about this works in the dev community, see our Suggest a feature page.


Elasticsearch support and Shade plugin for assembly

Hi @DmitryKey
The reason I replaced assembly plugin with shade, because it provide out of the box support for merging META-INF/services file - that's required for Elasticsearch support.

@apakulov hey, thanks a bunch for the pull request! I have updated lucene to 4.10.4, that is used in es 1.5.0. Will commit it shortly!


odo dev fails with unable push files to pod

/kind bug

What versions of software are you using?
odo v3.7.0 (262d11e3e)
Operating System:
Ubuntu 22.04
Output of odo version:
$ odo version
odo v3.7.0 (262d11e3e)
How did you run odo exactly?
odo init --name java-maven --devfile java-maven --devfile-registry DefaultDevfileRegistry --starter springbootproject
odo dev

Actual behavior
Error occurred on Push - watch command was unable to push component: failed to sync to component with name java-maven: failed to sync to component with name java-maven: unable push files to pod: error while streaming command: command terminated with exit code 2
Expected behavior
Syncing successfully
Any logs, error output, etc?
✗  Syncing files into the container [2s]

Error occurred on Push - watch command was unable to push component: failed to sync to component with name java-maven: failed to sync to component with name java-maven: unable push files to pod: error while streaming command: command terminated with exit code 2

 ◑  Syncing files into the container ✗  Command 'tar xf - -C /projects --no-same-owner' in container failed.

 ✗  stdout:

 ✗  stderr: tar: devfile.yaml: Cannot open: Permission denied
tar: src: Cannot mkdir: Permission denied
tar: src/main/java/com/example/demo/DemoApplication.java: Cannot open: No such file or directory
tar: mvnw.cmd: Cannot open: Permission denied
tar: src: Cannot mkdir: Permission denied
tar: src/test/java/com/example/demo/DemoApplicationTests.java: Cannot open: No such file or directory
tar: pom.xml: Cannot open: Permission denied
tar: mvnw: Cannot open: Permission denied
tar: .gitignore: Cannot open: Permission denied
tar: HELP.md: Cannot open: Permission denied
tar: src: Cannot mkdir: Permission denied
tar: src/main/resources/application.properties: Cannot open: No such file or directory
tar: Exiting with failure status due to previous errors

 ✗  err: error while streaming command: command terminated with exit code 2

Hi @stormshaun !
Thanks for reporting this issue.
Which kind of cluster are you using? We have similar issues reported when using odo on EKS (https://github.com/redhat-developer/odo/issues/6588) or AKS (https://github.com/redhat-developer/odo/issues/6585).

Hi @stormshaun ! Thanks for reporting this issue. Which kind of cluster are you using? We have similar issues reported when using odo on EKS (#6588) or AKS (#6585).

Hi @rm3l - We are experiencing this on OCP and also on ARO

Thanks for the info.
Can you please run odo preference set Ephemeral true prior to running odo dev, to see if it fixes the issue? Thanks.

Hi @stormshaun ! Thanks for reporting this issue. Which kind of cluster are you using? We have similar issues reported when using odo on EKS (#6588) or AKS (#6585).

Hi @rm3l - We are experiencing this on OCP and also on ARO

Thanks for the info.
Can you please run odo preference set Ephemeral true prior to running odo dev, to see if it fixes the issue? Thanks.

Thank you @rm3l, I'll try
Would you mind briefly sharing what the command does
odo preference set Ephemeral true

Would you mind briefly sharing what the command does odo preference set Ephemeral true

Sure! We suspect an issue with persistent volumes.
So with Ephemeral set to true, odo will not use a Persistent Volume Claim (PVC) to store source code and sync files, but an EmptyDir volume instead.
The default behavior is to use PVCs, but we noticed similar issues depending on the storage class.

Sure! We suspect an issue with persistent volumes. So with Ephemeral set to true, odo will not use a Persistent Volume Claim (PVC) to store source code and sync files, but an EmptyDir volume instead. The default behavior is to use PVCs, but we noticed similar issues depending on the storage class.

Hi @rm3l - Thank you! I was able to proceed after setting up the preference. If I  understand correctly, the issue lies between odo and PVC and not specifically in the cluster, as it looks like to be occurring across multiple platforms (EKS, AKS, OCP, ARO) and the work on odo side is in the board to address this behavior

Related: https://github.com/redhat-developer/odo/issues/5093

Another solution is setting sourceMapping inside your container and then using ${PROJECT_SOURCE} wherever the need be, in this case instead of mounting m2 on /home/user/.m2, you mount it on ${PROJECT_SOURCE}/.m2 and then change the build command to also use the same.
So your final devfile will look something like this

commands:
- exec:
    commandLine: mvn -Dmaven.repo.local=${PROJECT_SOURCE}/.m2/repository package   #<----
    component: tools
    group:
      isDefault: true
      kind: build
    workingDir: ${PROJECT_SOURCE}
  id: mvn-package
- exec:
    commandLine: java -jar target/*.jar
    component: tools
    group:
      isDefault: true
      kind: run
    workingDir: ${PROJECT_SOURCE}
  id: run
- exec:
    commandLine: java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=${DEBUG_PORT},suspend=n
      -jar target/*.jar
    component: tools
    group:
      isDefault: true
      kind: debug
    workingDir: ${PROJECT_SOURCE}
  id: debug
components:
- container:
    command:
    - tail
    - -f
    - /dev/null
    endpoints:
    - name: http-maven
      targetPort: 8080
    - exposure: none
      name: debug
      targetPort: 5858
    env:
    - name: DEBUG_PORT
      value: "5858"
    image: registry.access.redhat.com/ubi8/openjdk-11:latest
    memoryLimit: 512Mi
    mountSources: true
    sourceMapping: something #<----- setting this will assign the value to $PROJECT_SOURCE
    volumeMounts:
    - name: m2
      path: ${PROJECT_SOURCE}/.m2     #<----
  name: tools
- name: m2
  volume: {}
metadata:
  description: Java application based on Maven 3.6 and OpenJDK 11
  displayName: Maven Java
  icon: https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/java-maven.jpg
  language: Java
  name: java-maven
  projectType: Maven
  tags:
  - Java
  - Maven
  version: 1.2.0
schemaVersion: 2.1.0
starterProjects:
- git:
    remotes:
      origin: https://github.com/odo-devfiles/springboot-ex.git
  name: springbootproject

Hi @stormshaun ,
We have put together a troubleshooting guide with possible suggestions to fix the issue you reported: https://odo.dev/docs/user-guides/advanced/using-odo-with-other-clusters
I'm closing this issue then, but please feel free to reopen if the issue still persists after applying the suggestions provided.
Thanks again for reporting this issue.


feat: migrate to react-native-reanimated

Thank you very much for your helpful response. https://github.com/quidone/react-native-wheel-picker/pull/6#issuecomment-2336710950
I really appreciate it, as it has been very educational for a React Native beginner like myself.
I have removed the renderScrollView and confirmed that the callbacks in useAnimatedStyle and useAnimatedScrollHandler are working correctly within the worklet.
Please don't hesitate to let me know if there are any issues or further improvements needed for this PR.

@rozhkovs
Thank you for the review! If you prefer, I can switch the React Native version that example is using to an older one to avoid using the experimental feature. If so, the assertion in example/src/pickers/customized/Overlay.tsx will no longer be necessary.

@shoma-mano
Good job! I'll check locally later how it works.

If you prefer, I can switch the React Native version that example is using to an older one to avoid using the experimental feature. If so, the assertion in example/src/pickers/customized/Overlay.tsx will no longer be necessary.

Can you send the link about what is it about? I still don't get it.

@rozhkovs
I'm sorry for lack of context, I meant New Architecture (Fabric).Document says only the latest minor release react native is supported when it comes to New Architecture.
https://docs.swmansion.com/react-native-reanimated/docs/guides/troubleshooting/#outdated-version-of-react-native-for-new-architecture
But I was mistaken, only updating react native doesn't seem enough to use new architecture.It requires some expo settings.
So for now, I think it is ok to use 0.71.8 to avoid assertion in example/src/pickers/customized/Overlay.tsx.
If so, I have to down grade reanimated  to version that is compatible with 0.71.8.

@shoma-mano
Thanks for the clarification!
But to be honest, I still don't understand about "to avoid assertion in example/src/pickers/customized/Overlay.tsx.". Maybe I need to run the code and it will be clear.
I think it's a good idea to use "0.71.8" because updating expo and others to a newer version is already a bit beyond the scope of the current PR.

@rozhkovs
Yes, It may be a little difficult to understand without confirming error on IDE.
Without assertion, type error like this happens due to react native version difference between example(0.75.3) and library(0.71.8).

I think it's a good idea to use "0.71.8" because updating expo and others to a newer version is already a bit beyond the scope of the current PR.

I think so too! But after trying several versions of React Native Reanimated greater than 3.0.0, I found that the build fails if the React Native version remains at "0.71.8". Updating react native may be inevitable.

@shoma-mano

Without assertion, type error like this happens due to react native version difference between example(0.75.3) and library(0.71.8).

We need to synchronize the devDependencies of the library with the example. Then we will have the types of the same versions. That is, it should be enough to update the package' dependencies.json "react-native": "0.71.6" -> " react-native": "0.75.3" and the problem should disappear.

@rozhkovs
Thanks for response. I will update  update the package' dependencies.json "react-native": "0.71.6" -> " react-native": "0.75.3".
I have one more question: The existing android and ios folders are preventing the build from succeeding. Would it be okay to delete these folders?

@shoma-mano
Why are they in the way?
In theory, on iOS and Android, for the new version of Expo, these folders should be generated using the npx expo prebuild command. At the same time, the example uses native modules that are not included in the standard Expo SDK (@quidone/react-native-wheel-picker-feedback,...). That is, for the example to work correctly, we need to generate the ios and android folders and build the apps natively.
The commands expo run:android and expo run:ios should work. For the sake of experiment, you can delete these folders and re-generate them, perhaps the problem will disappear.
Perhaps we still need to take some steps to solve the problem when switching to the new version of Expo.
I honestly don't know if it's worth keeping track of the android and ios folders in Git in this case. Probably not, because this is the result of code generation, and you shouldn't edit something there yourself.

Hey there !
FYI in an expo project, ios & android folders are usually in the gitignore.

@rozhkovs

The commands expo run:android and expo run:ios should work. For the sake of experiment, you can delete these folders and re-generate them, perhaps the problem will disappear.

Yes, I confirmed that once the folder is deleted, the build succeeds with the new android and ios folders afterward. I don't know detail reason, Maybe they are just too old and it is hard to refresh without deleting folders completely. So, As @Alaa-Ben  says I think it is ok to include these folders to gitignore.

@Alaa-Ben

Hey there !
FYI in an expo project, ios & android folders are usually in the gitignore.

Thank you for the information!
@shoma-mano
Let's then delete the ios and android folders and add these folders to the example/.gitignore

@rozhkovs
I’m done! If there are any other points of concern or areas that need to be revised, please let me know.

@rozhkovs I’m done! If there are any other points of concern or areas that need to be revised, please let me know.

@shoma-mano I finally got to check this PR! Thanks for waiting!
The changes work correctly, but there is unpleasant news: on iOS (iPhone 11) and especially on Android (Readme Node 11 Pro), the performance is not satisfactory in my opinion. You can see that the animation occurs in jerks and there is a loss of frames, this is very clearly visible on phones where 120 hz.  I believe this is due to the Reanimated library itself, and we are unlikely to be able to speed this up.
I'm sorry for the little data, but this can be checked on a real Android device in debug mode. My device dropped to 30fps, this was not observed with Aniamted from react-native. In the Release version, there is also a drop in frames.
There is a proposal to leave Animated from react-native by default, and make Reanimated as an option for now. For example, you can add the animateEngine={"react-native" | "reaniamted" property}
How do you see it?

I mean, I don't want to just switch to Reanimated and lose the animation performance that we already have.

Thank you for reviewing, @rozhkovs!
How can I check the FPS? I have a Pixel 7, and while there's an option to display the refresh rate, there isn't an option to confirm FPS. I tried using Takoslots, but it doesn’t seem to support Android 14.
Can I confirm fps drop with "Show Perf" option in the emulator instead?
Anyway, if there’s a performance drop, it is not  ideal to use Reanimated as the default.
I think it’s best to hold off on the migration for a while.

@shoma-mano
I think the performance issue is related to this issue.

Can I confirm fps drop with "Show Perf" option in the emulator instead?

Probably, yes. But it is better to have a real device that is in the lower or middle price segment. As one of the options, you can add more elements to the WheelPicker until the moment when scrolling performance starts to drop. Next, you should find that with Animated from react-native, this will not happen.

@rozhkovs
I have confirmed now!
I'm pleasant to hear your suggestion to have options for  selecting library, But I think it's best to wait migrating considering code complexity.

@rozhkovs
I will try this when time allows :)
https://expo.dev/changelog/2024/10-24-sdk-52-beta


superset not displaying progress and waiting for pyhive results

Make sure these boxes are checked before submitting your issue - thank you!

[ Y ] I have checked the superset logs for python stacktraces and included it here as text if any
[ Y ] I have reproduced the issue with at least the latest released version of superset
[ Y ] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version
0.18
Expected results
Results are not displayed after running the query in sql lab
Actual results
2017-05-15 06:13:34,049:DEBUG:pyhive.hive:TFetchResultsResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=None, statusCode=0), results=TRowSet(rows=[], columns=[TColumn(i32Val=None, byteVal=None, i16Val=None, i64Val=None, stringVal=TStringColumn(nulls='\x00', values=[]), boolVal=None, doubleVal=None, binaryVal=None)], startRowOffset=0), hasMoreRows=False)
2017-05-15 05:11:57,471:ERROR:root:
Traceback (most recent call last):
  File "/home/ubuntu/environments/superset/local/lib/python2.7/site-packages/superset/sql_lab.py", line 138, in get_sql_results
    db_engine_spec.handle_cursor(cursor, query, session)
  File "/home/ubuntu/environments/superset/local/lib/python2.7/site-packages/superset/db_engine_specs.py", line 724, in handle_cursor
    resp = cursor.fetch_logs()
  File "/home/ubuntu/environments/superset/local/lib/python2.7/site-packages/superset/db_engines/hive.py", line 34, in fetch_logs
    response.results.rows, 'expected data in columnar format'
AssertionError

Steps to reproduce
virtualenv superset
. ./superset/bin/activate
pip install --upgrade setuptools pip
pip install superset
pip install MySQL-python
pip install pythrifthiveapi
fabmanager create-admin --app superset
superset db upgrade
superset init
Run a query that will take more than 6-7 secs to give the output
The above error is displayed, though the query is actively running in the background.

Notice: this issue has been closed because it has been inactive for 343 days. Feel free to comment and request for this issue to be reopened.

set async execute option of data source = False,then execute success


[5.2] Adds idType to model in order prevent assumptions about ids

Hi there,
We recently upgraded to Laravel 5.2 and noticed some weird behavior when trying to save models and tracked it down to an incorrect assumption made in a recent change to \Illuminate\Database\Model, namely in the getCasts() function:
    /**
     * Get the casts array.
     *
     * @return array
     */
    public function getCasts()
    {
        if ($this->getIncrementing()) {
            return array_merge([
                $this->getKeyName() => 'int', // This was our culprit
            ], $this->casts);
        }

        return $this->casts;
    }

Our codebase uses string uuids generated at the database layer (postgres) rather than the application layer, so technically we needed $incrementing to be set to true (since the database was handling "incrementing" in our case), however our ID type was string rather than int.
As an interim solution we just added the following to a BaseModel class we created which all of our models inherit from:
public function getCasts()
{
    return $this->casts;
}

this just overrides Laravel's getCasts function to get rid of the int typecast on the primary key. It worked but felt kind of jank, and we're sure we're not the only ones who have this use case. This PR proposes a solution which keeps Laravel's defaults, but adds the option to set $idType on models and override the default type (int) which is currently assumed to be true.
This shouldn't break anything for anyone else, it just adds this extra option which helps people with non-int ID types that are generated by the database rather than the application.
Let me know if you have any thoughts, questions, etc. I'd love to get this merged in (or at least start a discussion on how to best handle the issue). Thanks!

Any idea why the build is failing here? It's throwing a bunch of these:
Error: Call to a member function connection() on null

but I'm not entirely sure how this patch would be affecting anything to do with connections...

Any idea why the build is failing here? It's throwing a bunch of these:

Yeh, I've left a comment inline.

but I'm not entirely sure how this patch would be affecting anything to do with connections...

Yeh, the error is a bit cryptic due to our __call usage.

doh! it's probably something to do with that bad getter call.

doh! it's probably something to do with that bad getter call.

Yeh, that was exactly it. ;)

Looks like it should work now. Could you add a couple of tests please, then it'll be a :+1: from me.

I wasn't entirely sure how to unit test this at first but I think I figured something out. Let me know what you think of the two added in the latest commit (or just merge if they seem 👍 ).

@stevenleeg If your model doesn't have an auto-incrementing integer key you should set public $incrementing = false; in your model. I don't see any good use case for $idType.

@acasar setting $incrementing = false won't allow us to get the id generated by the database back (see here) therefore we need to set $incrementing = true.
Currently this also causes ids to be typecasted toint, hence this PR. $idType is used to specify this type in order to prevent a bad typecast when an id column doesn't use ints.

If your model doesn't have an auto-incrementing integer key you should set:

It can be auto-incrementing, but not an integer.

Ah yes, you're right :)

@acasar glad we could clear things up :)
@GrahamCampbell is there anything else I can do/clear up to get this merged? also mega thanks for the quick feedback!

@stevenleeg It looks good to me. It's up to Taylor to merge this though, if he likes it. ;)

@stevenleeg Thanks for the contribution. :heart:

@taylorotwell @GrahamCampbell @stevenleeg I could be wrong... but I think only the tests for this fix got merged into 5.2
Merge Commit -https://github.com/laravel/framework/commit/81a848285f05a4f05b4af86e3fc0f3ffa70873d4
(The tests pass because of the stub...maybe worth adding failure case tests?)

oops nvm I'm wrong, I just noticed $idType was renamed to $keyType for anyone else who runs into the same issue


DirectX interop question

According to your COM Interop Guide, I see that the best practice is to use the new interop API (IObjectReference, MarshalInspectable, and so on) even if we need to access non-WinRT types.

Generally, the RCW/CCW functions in Marshal should be avoided, as they are incompatible with the new ComWrappers support in .NET 5.

In our case, we work with both WinRT objects and non-WinRT (DirectX) COM objects, so we're interested in the common mechanism to marshal these COM types.
We tried to marshal DirectX objects using the MarshalInspectable<T> API as follows:
int[] featureLevels = { (int)D3D_FEATURE_LEVEL.D3D_FEATURE_LEVEL_9_1 };

D3D11CreateDevice(
    IntPtr.Zero,
    (uint)D3D_DRIVER_TYPE.D3D_DRIVER_TYPE_HARDWARE,
    IntPtr.Zero,
    (uint)D3D11_CREATE_DEVICE_FLAG.D3D11_CREATE_DEVICE_BGRA_SUPPORT,
    featureLevels,
    (uint)featureLevels.Length,
    7,
    out var device,  // we pass all of the COM objects as IntPtr to prevent the RCW creation by CLR mechanisms
    out var outFeatureLevel,
    out var context);

var d3dDevice = MarshalInterface<ID3D11Device>.FromAbi(device);

But we get an exception when calling the MarshalInterface<T>.FromAbi(...); method.
You can find a sample with the exception in the attachment (App4.zip). Simply run it and click on the Click Me button.

So my question is: What is the proper way to marshal non-WinRT COM objects?

@alexserov For marshaling a COM interop interface, you can continue to use the CLR's support:
var d3dDevice = (ID3D11Device)Marshal.GetTypedObjectForIUnknown(device, typeof(ID3D11Device));
The methods mentioned in the COM Interop Guide are for WinRT interfaces.

Closing the issue as it's been answered. Feel free to open a new issue if there is an issue with the proposed solution.


Docs not loading when using Router.mountApp('/api', server)

It seems like the the hard-coding of /docs as the base path for the swagger router is preventing mounting an application using Router.mountApp('/api', server) from platform. To be more specific, the docs .html loads fine, but all of the assets fail to load at /docs/{filename} when I'd really want for them to load from /api/docs/{filename}
I'm attempting to create a full-stack implementation of the "realworld" application, and the specification requires the API portion to run at /api/*, while the client rendering happens at /*.
I think just adding some kind of base path configuration would be totally sufficient, or maybe there's a way to adjust it with SwaggerFiles already?

Hey, yes, I can reproduce. I was solving a similar issue already but using HttpServer.router.mount (my use case is that I want to mount the swagger router without the whole effect-http machinery). With HttpServer.router.mount, the prefix is propagated within the HttpServer.router.RouteContext which the swagger router uses to generate the correct path in the index.html. The HttpServer.router.mountApp doesn't seem to do so.

Thanks for the quick response @sukovanej, I could definitely see being able to split it out into it's own handler so you can compose things as you see fit.
Regarding mount vs mountApp, it kinda sounds like a bug if mountApp doesn't propagate all relevant route information, no?

I started a thread in the effect discord. Let's confirm with Tim whether this is something to be fixed in the @effect/platform. If yes, let's do the adjustment in the @effect/platform and the fix here will be to update. If no, maybe there will be a way to check the mount path or something to make it work with mountApp. Otherwise, we'll need provide a way to configure the base path as you proposed.

Hey, I resolved it by introspecting the originalUrl on the request and calculating the prefix based on that. Please try to update to the latest version, it should start working with both mount and mountApp now.


hotfix: Remove cache-loader from webpack fonts rule

Jira
http://vjira2:8080/browse/WWWD-4467
Summary
Restores font files when creating multiple builds at once
Details
Steps to reproduce the problem

Check out the bolt-release-build repo
Run npx gulp --bolt-version=v2.13.1 --additional-components="@bolt/components-icon" (FYI, the additional-components option is a temporary workaround for a separate problem)
Observe that there is a builds/v2.13.1/builds/en--dev/fonts directory, but not a builds/v2.13.1/builds/en/fonts directory.

Note: it's possible this is an intermittent bug and may not always be reproducible the same way
Discussion
The fix in this PR is based on the discussion in https://github.com/webpack-contrib/cache-loader/issues/82
Why it works, I'm less sure.  The comments there mention a few things:

it should not be placed before any loader that has side effects like emitFile

I believe that's the case in our config, though I don't understand why it should be avoided.

You don't need cache-loader on file-loader, why you do this?

This brings up a bigger question-- are we even using cache-loader correctly?  Specifically, is there any benefit to using cache-loader before something like file-loader or url-loader (as opposed to, for example, babel-loader which has potentially heavy transpiling work to do?)
How to test
Confirm that the change in this PR resolves the bug described in "Steps to reproduce", above

PR was released with v2.13.2


Is there any documentation for creating an expectation with complex api request?

Describe the issue
I am an Android developer and I don't have all the services ready yet, so I thought to use any mock-server. Someone suggested me this. So my plan is like recording a demo by just mocking some APIs with mocked data.
Is mock-server the right choice for this? I have spent much time and I am still struggling to understand this. Anyway, I have created a mock-server with a simple API but in my actual case, I have some complex APIs with parameters, file attachments, OAuth2.0 login, etc.

How to create an expectation for a complex API for example-

GET /mysite/api/public/v1/pets/{petId}/categories/users/{userId}

How to mock OAuth2.0 with our mock-server? I didn't find any documentation.

What you are trying to do
I want to set some expectations/ mock apis so that I can post/ get/ petch some data using my android app for the demo purpose.
MockServer version
I am trying with latest version - 5.11.0
To Reproduce
I tried with node js code -
mockServerClient("localhost", 1080)     .mockSimpleResponse('/somePath', { name: 'value' }, 203)     .then(         function(result) {             // do something next             console.log("Sample success ", result);         },         function(error) {             // handle error             console.log("Sample Error ", error);         }     );
Expected behaviour
I have API endpoints with JSON request/ response, I want when I do POST it save the data, and when I use GET it should return the recently posted data.

For documentation please see the main website which has lots of examples: https://mock-server.com/mock_server/creating_expectations.html#request_body_matchers

Thanks @jamesdbloom for your quick response.
When I do POST it is not saving the data. What I want is when I use GET it should return the recently posted data.
Is it possible?

MockServer doesn't save data when you POST.  You need to set expectations in MockServer as per the documentation.  https://mock-server.com/


Prompt Search mode doesn't allow backspace

Right now when I press Ctrl-R to go into the prompt search mode, I'm unable to correct my search term using backspace or Ctrl-W or Ctrl-U.
But all those keys work fine if I use the SearchToolbar mode.

Fixed in: https://github.com/jonathanslenders/python-prompt-toolkit/commit/e6660b5010bd1e7feacc1e65c6a924090087890d
Does GNU readline support Ctrl-W/U in incremental search?

No, I was wrong.
I will try the patch.
Although readline does support C-E, C-A, C-F, C-B, which basically kicks you out of the search mode and puts the cursor on the matching character and does the appropriate movement (C-A: home, C-E, end etc).


Review stack variable descriptions

Review stack variable descriptions: https://github.com/terraform-ibm-modules/stack-ibm-core-security-services/blob/main/ibm_catalog.json
Maybe take a look at https://github.com/terraform-ibm-modules/stack-retrieval-augmented-generation/blob/main/ibm_catalog.json as a reference?

closed by #20


Create shared library for multiple microservices failed on Linux platform

Problem: We have multiple microservices written in Python, and shared some third-party packages.
We could create standalone executables for different microservices, but the size might be huge, compared with using shared library. So we plan to create a shared library which would include all dependent third-party packages with the following command:

python3.10 -m nuitka --module shared-env/lib/python3.10/site-packages/ --include-package=grpcio --include-package=grpcio-tools --include-package=protobuf --include-package=fastapi --include-package=pydantic --include-package=starlette --include-package=uvicorn --include-package=requests --include-package=pandas --include-package=pika --include-package=black --include-package=click --include-package=mypy-extensions --include-package=nuitka --include-package=packaging --include-package=pathspec --include-package=platformdirs --include-package=tomli --include-package=numpy --include-package=python-dateutil --include-package=pytz --include-package=setuptools --include-package=six --include-package=toml --include-package=tzdata --include-package=anyio --include-package=certifi --include-package=charset-normalizer --include-package=h11 --include-package=idna --include-package=sniffio --include-package=typing-extensions --include-package=urllib3 --standalone shared.py

, and then build each of the microservices using the following command:

python3.10 -m nuitka --include-module shared.so --standalone service_a.py

But when we built the shared.so, it report the following error message:

FATAL:
Error, specify only one positional argument unless "--run" is specified to
pass them to the compiled program execution.

We follow the instructions suggested by ChatGPT to modify the command of generating shared library as follows:

python3.10 -m nuitka --module --run shared-env/lib/python3.10/site-packages/ --include-package=grpcio --include-package=grpcio-tools --include-package=protobuf --include-package=fastapi --include-package=pydantic --include-package=starlette --include-package=uvicorn --include-package=requests --include-package=pandas --include-package=pika --include-package=black --include-package=click --include-package=mypy-extensions --include-package=nuitka --include-package=packaging --include-package=pathspec --include-package=platformdirs --include-package=tomli --include-package=numpy --include-package=python-dateutil --include-package=pytz --include-package=setuptools --include-package=six --include-package=toml --include-package=tzdata --include-package=anyio --include-package=certifi --include-package=charset-normalizer --include-package=h11 --include-package=idna --include-package=sniffio --include-package=typing-extensions --include-package=urllib3 --standalone shared.py

Then it reported error:

Nuitka-Options:WARNING: You did not specify to follow or include anything but main module. Check options and make sure that is intended.
Traceback (most recent call last):
File<EMAIL_ADDRESS>line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File<EMAIL_ADDRESS>line 86, in _run_code
exec(code, run_globals)
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/tools/data_composer/main.py", line 40, in
main()
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/tools/data_composer/DataComposer.py", line 412, in main
assert name not in names, name
AssertionError
Traceback (most recent call last):
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/main.py", line 155, in
main()
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/main.py", line 141, in main
MainControl.main()
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/MainControl.py", line 1103, in main
_main()
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/MainControl.py", line 972, in _main
result, options = compileTree()
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/MainControl.py", line 882, in compileTree
runDataComposer(source_dir)
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/build/DataComposerInterface.py", line 33, in runDataComposer
blob_filename = _runDataComposer(source_dir=source_dir)
File "/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/build/DataComposerInterface.py", line 54, in _runDataComposer
subprocess.check_call(
File<EMAIL_ADDRESS>line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/xxxx/xxxx/shared-env/bin/python3.10', '/Users/xxxx/xxxx/shared-env/lib/python3.10/site-packages/nuitka/tools/data_composer', 'artifacts/.build', 'artifacts/.build/__constants.bin']' returned non-zero exit status 1.

Nuitka version, full Python version, flavor, OS, etc. as output by this
command.

1.6.3
Commercial: None
Python: 3.10.12 (main, Jun  7 2023, 00:38:32) [Clang 14.0.3 (clang-14<IP_ADDRESS>.1)]
Flavor: Homebrew Python
Executable: /Users/danielwong/PycharmProjects/OculusManufacturing/arvr/projects/manufacturing/columbus/columbus_app/columbus_app_venv/bin/python3.10
OS: Darwin
Arch: arm64
Version C compiler: /usr/bin/clang (clang).

How did you install Nuitka and Python

I installed Python in a virtualenv, and install Nuitka with python3.10 -m pip install -r pip_requirements.txt

Provide in your issue the Nuitka options used

--module --run shared-env/lib/python3.10/site-packages/ --include-package=grpcio --include-package=grpcio-tools --include-package=protobuf --include-package=fastapi --include-package=pydantic --include-package=starlette --include-package=uvicorn --include-package=requests --include-package=pandas --include-package=pika --include-package=black --include-package=click --include-package=mypy-extensions --include-package=nuitka --include-package=packaging --include-package=pathspec --include-package=platformdirs --include-package=tomli --include-package=numpy --include-package=python-dateutil --include-package=pytz --include-package=setuptools --include-package=six --include-package=toml --include-package=tzdata --include-package=anyio --include-package=certifi --include-package=charset-normalizer --include-package=h11 --include-package=idna --include-package=sniffio --include-package=typing-extensions --include-package=urllib3 --standalone shared.py

Please don't waste my time with AI phantasies. Instead, read the docs. This amount of included packages makes no sense, esp. including Nuitka itself is funny. And in module mode, you only include your own code, many of the packages you listed will not work or coexist in any way. And then using site-packages for the module code is outright funny.
This is a very good illustration, on why AI bot destroy things for good people. But I think I will use you for some very nice reflection on how bad things can get.
As for size issues, you may want to read up on multidist, which allows to deploy multiple programs from one binary.


[raymath] Added macros for EPSILON on each function it's used in

This is so the functions can be easily copied and used. This was discussed in a separate pull:
https://github.com/raysan5/raylib/pull/3319#issuecomment-1722461911

I have noticed a difference between my addition and the one recommended by ray, and that's that i used "ifndef" while he used "#if !defined(RAYMATH_EPSILON)".
I didn't notice it on pull, i'll fix it in an hour when i'm free

I am now done

@Brian-ED thanks for this review!


erro noo me deja mandar prompts

Submit any bug at https://github.com/Simatwa/python-tgpt/issues/new
╭─A@pyTGPT~[🕒22:36:06-💻00:00:00-⚡0.0s]
╰─>hola
22:36:10 - ERROR : 'choices'
╭─A@pyTGPT~[🕒22:36:10-💻00:00:04-⚡0.8s]
╰─>

Thank you for raising this issue. A similar one is already addressed at #50


local api CRUD operations return imprecise typing

Bug Report
the local api CRUD operations return imprecise typing.
Steps to Reproduce
  const res = await payload.find({
    collection: "users",
    depth: 2,
  });

  // org will be typed `string | Organization`, but it should
  // just be `Organization`, because depth is 2.
  // The types do not seem to correctly infer the type.
  const org = res.docs[0].organizations

Other Details
    "payload": "1.6.10",

As of now depth is not taken into account for the inferred types. Therefore, they must be a union with string. We plan on improving this in the future.

Just a quick update for anyone else coming here:
Because there are a few complexities with how documents either will or will not get populated, completely outside of the depth argument, we are actually planning on keeping these relationship / upload field type unions as-is.
For example - access control, hooks, draft vs published, document deletion - these are just a few of the cases that could prevent a doc from being populated via depth.
For this reason it's probably best in code to always just ensure that you do have properly populated docs. This has actually helped us write better code in our own projects and is ultimately the safest type we can deliver.


listings not supported

Hello
thanks a lot for this work
I have a small issue, I try to use tagged pdf with a file including listings.
But there is a compile error
% !Mode:: "TeX:UTF-8:Main"
\DocumentMetadata{
 testphase={
     phase-III,
%     math,table
     },
 pdfversion=2.0,
 pdfstandard=ua-2,
}
%%\tagpdfsetup{math/mathml/write-dummy}
\tagpdfsetup{listings}
\documentclass{article}

\usepackage{amsmath}
\usepackage{listings}

\title{Listing Examples}
\author{Dom}
\date{\today}

\begin{document}

\section{Python code}

%\begin{lstlisting}[language=python,frame=single,numbers=left,basicstyle=\footnotesize]
\begin{lstlisting}[language=python]
import os

os.listdir()
\end{lstlisting}

\end{document}

compiled with lualatex-dev, (This is LuaHBTeX, Version 1.18.0 (TeX Live 2024)/Development id: 7611), this produces :
! Package tagpdf Error: The number of automatic begin (4) and end (5)
(tagpdf)                text-unit para hooks differ!

br

You point an interesting issue here, thank you. how should it be tagged ?
Did not really think about it initially since I was first trying to have it compiling (could be with a warning saying somth about the tagging not there/not defined)
I'd say at least a the language and then the content as text, numbering probably tagged (as page numbers are, would be interesting to do it). but this is quick first thought, need more work - not saying it can/should be fixed soon. but at least on the roadmap somewhere

well it is on the roadmap :-). Concerning how such things should be tagged ... in many cases that is wide open. In https://github.com/latex3/tagging-project/discussions/7 I made a first stab at a possible tagging structure for verbatim blocks (like listings are) but that is for from being the last word on the topic.


Sodium crash Render overlay

Describe the Bug
game goes into a loading loop then crashes with Create and other mods who use create enabled
Reproduction Steps
1.enabled integrated stronghold and create mod
2. start game and then it goes into a boot loop before crashing shortly after
3.
...
Expected Result
tested with waystones and owo-lib and it launches with no issue it should launch normally
Screenshots and Videos
No response
Crash Report or Log
https://pastebin.com/WFigKTPh
Operating System
windows 10
Mod Version
0.4.0
Minecraft Version
1.19.2
Other Mods
Integrated API
Integrated Strongholds
Create Fabric
owo-lib
Additional Context
I am using an archived version of sodium and sodium extras that i found on mondrith

downgrade fabric api to 0.76

downgrade fabric api to 0.76

issue resolved by doing this and disabling the flywheel compact mod for iris


Add link to "full-time form part-time plan" warning to upgrade account

Businesses that have signed up for the $99/month part-time plan are not allowed to hire full-time developers.
When reaching out to a developer, if the business is on the part-time plan and the developer is not looking for part-time contract roles...

[ ] Add link to Stripe Billing to upgrade account in warning/info message

I'll try and give this one a shot over the weekend if I have a couple of hours. If not I'll work on it on monday.

Awesome! I forgot to mention, but this is dependent on #358.

I see.  Stop them from completing the transaction(outside of their limited account)  and redirect them to the upgrade page instead. seems like a single issue to me!

Yes, I'll take a pass at 358. thank you

Closed via #380.


'--output': bool, in command.py, while agent.py expects a file :  file_out = args['--output']

Maybe, this is a problem of no documentation of the process or misuse of a variable name. The current version of the parser that variable has different name 'generate_files' and it is a boolean value to indicate if we want to generate the output files or not.