Ansible Common Task Failure

Hey there. I was just curious if you were running into Ansible task failures during your beta testing. The repo does a great job automating everything for me until the Ansible roles start. Specifically, the Ansible task that seems to fail each time is the
TASK [common : Upgrade module PowerShellGet to fix accept license issue on last windows ansible version] task.
I'm pretty sure it's pointing to https://github.com/Orange-Cyberdefense/GOAD/blob/4dfbbf637577f21c5918c3d36ff439fd0439077a/ansible/roles/common/tasks/main.yml#L23
Since it seems to pull the Ansible tasks/roles from the original GOAD repo, I wasn't sure if I should raise the issue here or there. Am happy to close and repost to the GOAD repo if you prefer!
The error output in its entirety is the following snippet:
fatal: [dc03]: FAILED! => {"changed": true, "cmd": "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12\nInstall-PackageProvider -Name NuGet -Force\nInstall-Module PowerShellGet -Force", "delta": "0:00:10.296897", "end": "2023-12-27 16:11:41.700072", "msg": "non-zero return code", "rc": 1, "start": "2023-12-27 16:11:31.403174", "stderr": "Install-PackageProvider : No match was found for the specified search criteria for the provider 'NuGet'. The package \r\nprovider requires 'PackageManagement' and 'Provider' tags. Please check if the specified package has the tags.\r\nAt line:2 char:1\r\n+ Install-PackageProvider -Name NuGet -Force\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvider:InstallPackageProvider) [Install-Pac \r\n   kageProvider], Exception\r\n    + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackagePro \r\n   vider\r\n \r\nException calling \"ShouldContinue\" with \"2\" argument(s): \"Windows PowerShell is in NonInteractive mode. Read and \r\nPrompt functionality is not available.\"\r\nAt C:\\Program Files\\WindowsPowerShell\\Modules\\PowerShellGet\\<IP_ADDRESS>\\PSModule.psm1:7392 char:8\r\n+     if($Force -or $psCmdlet.ShouldContinue($shouldContinueQueryMessag ...\r\n+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException\r\n    + FullyQualifiedErrorId : PSInvalidOperationException\r\n \r\nInstall-Module : NuGet provider is required to interact with NuGet-based repositories. Please ensure that '<IP_ADDRESS>' \r\nor newer version of NuGet provider is installed.\r\nAt line:3 char:1\r\n+ Install-Module PowerShellGet -Force\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : InvalidOperation: (:) [Install-Module], InvalidOperationException\r\n    + FullyQualifiedErrorId : CouldNotInstallNuGetProvider,Install-Module", "stderr_lines": ["Install-PackageProvider : No match was found for the specified search criteria for the provider 'NuGet'. The package ", "provider requires 'PackageManagement' and 'Provider' tags. Please check if the specified package has the tags.", "At line:2 char:1", "+ Install-PackageProvider -Name NuGet -Force", "+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~", "    + CategoryInfo          : InvalidArgument: (Microsoft.Power...PackageProvider:InstallPackageProvider) [Install-Pac ", "   kageProvider], Exception", "    + FullyQualifiedErrorId : NoMatchFoundForProvider,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackagePro ", "   vider", " ", "Exception calling \"ShouldContinue\" with \"2\" argument(s): \"Windows PowerShell is in NonInteractive mode. Read and ", "Prompt functionality is not available.\"", "At C:\\Program Files\\WindowsPowerShell\\Modules\\PowerShellGet\\<IP_ADDRESS>\\PSModule.psm1:7392 char:8", "+     if($Force -or $psCmdlet.ShouldContinue($shouldContinueQueryMessag ...", "+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~", "    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException", "    + FullyQualifiedErrorId : PSInvalidOperationException", " ", "Install-Module : NuGet provider is required to interact with NuGet-based repositories. Please ensure that '<IP_ADDRESS>' ", "or newer version of NuGet provider is installed.", "At line:3 char:1", "+ Install-Module PowerShellGet -Force", "+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~", "    + CategoryInfo          : InvalidOperation: (:) [Install-Module], InvalidOperationException", "    + FullyQualifiedErrorId : CouldNotInstallNuGetProvider,Install-Module"], "stdout": "", "stdout_lines": []}

Make sure the Windows VMs have internet access for the ansible configuration part. I think I ran into this same issue when the Windows VM running the ansible scripts didn't have internet access. The pfSense rules should have allowed access from LAN to WAN during the setup process. Check if they have internet access, and if they do I will try to help from there. Also, I have run into ansible issue where I had to run the setup process twice before the ansible portion took.

Awesome, I appreciate the fast reply. I'll run the script again and check internet access and the pfSense rules and report back.

Yep found the culprit, the VMs could ping out but DNS resolution was failing. Once I enabled Enable Forwarding Mode for the DNS Query Forwarding setting under Services -> DNS Resolver -> General Settings, the Ansible task stopped failing!

Awesome!
In the next few days I am going to add the Elastic EDR Ubuntu 22.04 machine as an option for those wanting to have a EDR setup on each machine. I have some automation to work out, but it should be added before to much longer.

Solved.


This one's actually intentional, as you can use Stasis as a one-time Exhaust effect to get rid of a curse, etc.
It's not always ideal, but I think it's probably better to allow it than disallow it.

Agreed.


Error processing image locally with openai api

Had to change the process_page() function to this for it to work (didnt seem to provide the correct "message" syntax to openai):

            def process_page(page_num):
                page = doc[page_num]
                text = page.get_text()
                pix = page.get_pixmap()
                image = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)

                base64_image = make_image_url(image, host_images=HOST_IMAGES)  # Assuming it returns a base64-encoded result

                messages = [
                    {
                        "role": "user",
                        "content": [
                            {"type": "text", "text": "```{} ```\n{}".format(text, SCRAPING_PROMPT)},
                            {"type": "image_url", "image_url": {"url": base64_image}},
                        ]
                    },
                ]

same here


[GOBBLIN-245] Create topic specific extract of a WorkUnit in KafkaSource

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title.

https://issues.apache.org/jira/browse/GOBBLIN-245

Description

[x] Here are some details about my PR, including screenshots (if applicable):
Current KafkaSource ignores topic specific configurations on creating Extract of a WorkUnit. The task is to create the extract with topic specific configurations if any or else job level configurations.

Tests

[x] My PR does not need testing for this extremely good reason:
Currently, there is no unit test for KafkaSource. It's testability is pretty bad. We need a task to make it testable.

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@htran1 Please review


Add interface point for RNGs

Replaces #164

Codecov Report

Merging #166 (148349e) into dev (6c48032) will decrease coverage by 0.26%.
The diff coverage is 86.66%.

@@            Coverage Diff             @@
##              dev     #166      +/-   ##
==========================================
- Coverage   88.94%   88.67%   -0.27%
==========================================
  Files           8        8
  Lines         208      212       +4
==========================================
+ Hits          185      188       +3
- Misses         23       24       +1

Impacted Files
Coverage Δ

src/types.jl
100.00% <ø> (ø)

src/builders.jl
55.55% <33.33%> (-6.95%)
:arrow_down:

src/classifier.jl
100.00% <100.00%> (ø)

src/common.jl
96.55% <100.00%> (+0.18%)
:arrow_up:

src/image.jl
100.00% <100.00%> (ø)

src/regressor.jl
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c48032...148349e. Read the comment docs.

@ayush-1506 Could you please review?

Sure, will review this later today.


Pre-calculated launch measurement doesn't match the attestation report one

As I mentioned in the last SVSM call I have some issues with the pre-calculated launch measurement
I restarted with a clean build, I'm using the following components:
igvm: commit 9b27f1b173cbbb6558663c83b62cbd0398e25853 (tag: igvm_defs-v0.2.1, tag: igvm-v0.2.1)
edk2: commit d965a1bd3d331e4b422526368e832f3d327fd0da (HEAD -> svsm, origin/svsm)
qemu: commit 260df2b36be5e31e1ffe6b97e7c846f1d8cd3c52 (HEAD -> svsm-igvm, origin/svsm-igvm)
svsm: commit fc982434231cd1835efc16f6e101afe8023d3dee (origin/main, origin/HEAD, main)

On SVSM I applied this patch (you can find the branch here: https://github.com/stefano-garzarella/svsm/tree/get_report):
diff --git a/kernel/src/greq/pld_report.rs b/kernel/src/greq/pld_report.rs
index dc8ca69..29924c7 100644
--- a/kernel/src/greq/pld_report.rs
+++ b/kernel/src/greq/pld_report.rs
@@ -73,7 +73,7 @@ pub struct SnpReportResponse {
     /// Reserved
     _reserved: [u8; 24],
     /// The attestation report generated by firmware
-    report: AttestationReport,
+    pub report: AttestationReport,
 }

 /// Supported values for SnpReportResponse.status
@@ -164,7 +164,7 @@ pub struct AttestationReport {
     /// Guest-provided data
     report_data: [u8; 64],
     /// The measurement calculated at launch
-    measurement: [u8; 48],
+    pub measurement: [u8; 48],
     /// Data provided by the hypervisor at launch
     host_data: [u8; 32],
     /// SHA-384 digest of the ID public key that signed the ID block
diff --git a/kernel/src/svsm.rs b/kernel/src/svsm.rs
index 486d43d..dc85ea5 100755
--- a/kernel/src/svsm.rs
+++ b/kernel/src/svsm.rs
@@ -57,6 +57,12 @@ use svsm::vtpm::vtpm_init;

 use svsm::mm::validate::{init_valid_bitmap_ptr, migrate_valid_bitmap};

+extern crate alloc;
+
+use alloc::{format, vec};
+use svsm::greq::pld_report::SnpReportResponse;
+use svsm::greq::services::get_regular_report;
+
 extern "C" {
     pub static bsp_stack_end: u8;
 }
@@ -463,6 +469,22 @@ pub extern "C" fn svsm_main() {

     create_kernel_task(request_processing_main).expect("Failed to launch request processing task");

+    let mut buf = vec![0; size_of::<SnpReportResponse>()];
+
+    let size = get_regular_report(&mut buf).unwrap();
+    assert_eq!(size, buf.len());
+
+    let response = SnpReportResponse::try_from_as_ref(&buf).unwrap();
+
+    if response.validate().is_ok() {
+        let measurement_string = response
+            .report
+            .measurement
+            .map(|v| format!("{v:02x}"))
+            .join("");
+        panic!("SNP Launch Measurement: {measurement_string}");
+    }
+
     #[cfg(test)]
     crate::test_main();

As you can see the one in the attestation report is different from the pre-calculated one (I put the panic to simplify printing):
$ make
...
"target/x86_64-unknown-linux-gnu/debug/igvmbuilder" --sort --policy 0x30000 --output bin/coconut-qemu.igvm --stage2 bin/stage2.bin --kernel bin/svsm-kernel.elf --filesystem bin/svsm-fs.bin  qemu
"target/x86_64-unknown-linux-gnu/debug/igvmmeasure" --check-kvm bin/coconut-qemu.igvm measure

===============================================================================================================
igvmmeasure 'bin/coconut-qemu.igvm'
Launch Digest: E29837FB4B3B78FD4D3935A06BD83A92B334DD380D4165B4FC52ACD166DAEFB253C23B0E66F510C23C549F86F7B213E5
===============================================================================================================

"target/x86_64-unknown-linux-gnu/debug/igvmbuilder" --sort --output bin/coconut-hyperv.igvm --stage2 bin/stage2.bin --kernel bin/svsm-kernel.elf --comport 3 hyper-v --native
"target/x86_64-unknown-linux-gnu/debug/igvmmeasure" bin/coconut-hyperv.igvm measure

===============================================================================================================
igvmmeasure 'bin/coconut-hyperv.igvm'
Launch Digest: 87E288C09865BA8D9DB26FFE05E9D7208C7753DF55CB17CBE8994741ABE3918C0A14C3D841748C20A0A498C57679F87B
===============================================================================================================

$ ./scripts/launch_guest.sh --qemu ../qemu-svsm/build/qemu-system-x86_64
...
[SVSM] VTPM: Microsoft TPM 2.0 initialized
[SVSM] [CPU 0] Virtual memory pages used: 0 * 4K, 0 * 2M
[SVSM] Launching request-processing task on CPU 0
[SVSM] ERROR: Panic: CPU[0] panicked at kernel/src/svsm.rs:485:9:
SNP Launch Measurement: aab55e933e0ac602da92a971976c20e4af65149be3955d91bd76ae7b4042bf08e09eb985d31bb699fd1a8f531b2081eb

I will try to debug it, @roy-hopkins any suggestion?

The problem was due to incorrect init-flags on the QEMU command line in the script. I've fixed it and raised a PR.

@joergroedel can you re-open this issue, or should I create a new one?
As I reported in https://github.com/coconut-svsm/svsm/pull/353#issuecomment-2125126506 we have another issue related to SevFeatures::SevFeatureDebugSwap

Re-opened as there is another problem to fix.


[Question] Remote URL Upload?

From my complete ignorance, I wonder...
It's possible to add a Remote URL Upload function or the INDEX only replicate the same functions that G. Drive have?

I think it is possible and someone has already suggested. But I have to do some researches in order to figure out what is the best approach. Since I have much prioritized tasks on the to-do list, It may take some time for me to look into it.
Another thing is I'm always open to PRs so you can have a try and make a PR if you get succeed.

I think it is possible and someone has already suggested. But I have to do some researches in order to figure out what is the best approach. Since I have much prioritized tasks on the to-do list, It may take some time for me to look into it.
Another thing is I'm always open to PRs so you can have a try and make a PR if you get succeed.

Awesome man, an option for URL Upload would be amazing, (i think at the moment there is no INDEX with that feature).
Thanks for the great job you do with this INDEXER.

@ZeroCool22 Hi, yes this feature is nice but I think there's no need for this, there's a lot of way to remote upload in G Drive, like google collab scripts, telegram bots, try this bot in telegram @gdriveupme_bot and *@GdriveXbot".
@cheems You are amazing!


resolve [#370] to remove globals

removed some unused imports and removed global variables whose uses have already been removed. @pkdash @selimnairb could you review to make sure they don't cause any issues? I have already done a global search and the removed don't appear to be used anywhere.

Unfortunately, I can't tell if these changes will cause problems just by looking at the code.  I suggest making sure we have unit tests that cover the modified code, and then running the tests to see if they pass.

agreed unit tests is the best. I did my best to make sure these globals are not used and removal of them will not cause issues, but to be 100% certain, unit tests are objective measure. @pkdash, is the test coverage you recently implemented sufficient at this stage to run for this purpose? If so, could you merge this branch in and run coverage test to make sure coverage score does not go down?

Used Pycharm "Find usage" as well as "Find across project" and made sure the removed globals are not used or can be safely replaced with a local variable. Also run coverage test and made sure the coverage percentage does not go down (stayed the same as expected). Also talked this with @selimnairb, @pkdash , and @ironfroggy, and think this pull request can be safely merged into develop.

if we don't have tests already we can't really be confident that any new tests adequately test this change. If everything runs, I think we should merge, but I do agree that tests should be added for the affected functions if they do not already exist, before merging.

This pull request removed three global variables. Two global variables are not used in any functions any more, so they should be safe to remove. The other global variable is replaced by a local variable, and is not really used outside of that function get_resource_types(), so it should be safe to replace it with a local variable and return the local resource type variable value from the function. So this pull request should be safe to merge, and from this point on we should avoid to use global variables if at all possible.


Gradle: Upgrade PostgreSQL to version 42.2.9

See https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md#4229-2019-12-06.
Signed-off-by: Sebastian Schuberth<EMAIL_ADDRESS>

@mnonnenmacher, mind merging this despite the unrelated ApVeyor failures?