Dependency Updates, Documentation Enhancements, New Features, and Code Refactoring

So, I have the first big truck of changes for you. I mainly did formatting (those are separate commits) and clippy changes, improving alot of things efficiency-wise. I also fixed a MRSV of minimum supported Rust version, dictated to us by our dependencies anyway.
I compressed the helper/crypt.rs part and pulled away all the static values we currently do not allow to be changed from user side anyway. Makes the code alot more readable.

Chore

 update zip dependency to version 2.2.2

 add categories to Cargo.toml

 allow multiple versions of thiserror
This fixes the following clippy warning:

multiple versions for dependency thiserror: 1.0.69, 2.0.8
for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#multiple_crate_versions

thiserror and thiserror-impl are transitive dependencies:
cargo tree --invert --package<EMAIL_ADDRESS>
thiserror-impl v1.0.69 (proc-macro)
└── thiserror v1.0.69
    ├── html_parser v0.7.0
    │   └── umya-spreadsheet v2.2.0 (/home/mxsrm/code/umya-spreadsheet)
    └── rav1e v0.7.1
        └── ravif v0.11.11
            └── image v0.25.5
                └── umya-spreadsheet v2.2.0 (/home/mxsrm/code/umya-spreadsheet)

cargo tree --invert --package<EMAIL_ADDRESS>
thiserror-impl v2.0.8 (proc-macro)
└── thiserror v2.0.8
    ├── pest v2.7.15
    │   ├── html_parser v0.7.0
    │   │   └── umya-spreadsheet v2.2.0 (/home/mxsrm/code/umya-spreadsheet)
    │   ├── pest_derive v2.7.15 (proc-macro)
    │   │   └── html_parser v0.7.0 (*)
    │   ├── pest_generator v2.7.15
    │   │   └── pest_derive v2.7.15 (proc-macro) (*)
    │   └── pest_meta v2.7.15
    │       └── pest_generator v2.7.15 (*)
    └── zip v2.2.2
        └── umya-spreadsheet v2.2.0 (/home/mxsrm/code/umya-spreadsheet)

 remove unused js feature from Cargo.toml
This commit removes the unused js feature from the [features]
section in Cargo.toml. The js feature is no longer utilized in
the project, and its removal helps to maintain a cleaner configuration.
Previously, this feature led to the inclusion of the js feature from
the getrandom dependency, which was removed in a prior commit.

Deleted the js feature entry from Cargo.toml.

 set minimum supported Rust version to 1.79.0
This commit sets the minimum supported Rust version in Cargo.toml
to 1.79.0, as determined by the cargo-msrv tool. This version is
dictated by the bitstream-io dependency, which is a transitive
dependency of the image crate. Setting the minimum version ensures
compatibility with the required features and functionality.

Added rust-version = "1.79.0" to Cargo.toml.

 move lint configurations to lib.rs
This commit removes the lint configurations from Cargo.toml and
transfers them to lib.rs using attribute macros. This change
centralizes lint settings within the codebase, making it easier to
manage and understand the linting rules applied to the project.

Removed lint settings from Cargo.toml.
Added lint attributes in lib.rs for better backwards compatibility.

 sort dependencies in Cargo.toml
This commit sorts the dependencies in Cargo.toml for improved
readability and organization. Sorting helps maintain a consistent
format and makes it easier to locate specific dependencies.

Documentation

 enhance documentation for encryption-related functions

Add detailed parameter descriptions and return values for all public functions
Improve documentation formatting and clarity for encryption methods
Add specific documentation for AES-256-CBC implementation details
Include clippy allow attributes for possible truncation warnings
Document HMAC and IV generation processes
Add comprehensive documentation for password-to-key conversion logic

New Features

 enhance Excel timestamp conversion with additional calendar systems

add num-traits dependency for numeric casts
introduce DEFAULT_TIMEZONE constant for consistency
implement excel_to_date_time_object for converting Excel timestamps to NaiveDateTime
add convert_date_windows_1900 and convert_date_mac_1904 for specific calendar systems
enhance convert_date_crate to support both Windows 1900 and Mac 1904 systems
include detailed documentation for new functions with examples and panic behavior

Refactor

 update is_address function for regex initialization

Refactored the is_address function to use OnceLock for lazy initialization of the regex pattern, improving performance by compiling the regex only once.
Enhanced documentation for is_address to clarify its parameters, return values, and potential panics.
Removed the previous implementation of is_address to streamline the code.
Added examples in the documentation to demonstrate usage.

 remove unnecessary qualification

 clippy lints

 simplify make_buffer function and improve worksheet processing

Reorganized the make_buffer function for better readability and maintainability.
Replaced manual iteration with try_for_each for processing worksheets.
Consolidated worksheet processing logic to handle both deserialized and raw data more cleanly.
Improved comments for clarity on each processing step.
Streamlined the addition of various worksheet components (charts, drawings, comments, etc.).
Ensured proper handling of printer settings and other worksheet relationships.

 remove leading underscores from method names and allow dead_code at crate level

Rename private methods across multiple modules by removing leading underscores to adhere to Rust naming conventions.
Additionally, add #![allow(dead_code)] at the crate level to silence dead code warnings.

 centralize constants into a dedicated module
Extract multiple static variables into the constants module, eliminating the need to pass them as parameters in function calls. This refactor simplifies function signatures, centralizes configuration values, and enhances maintainability. Future updates may revert this approach to accommodate support for additional algorithms as the crate evolves.

 abstract random byte generation with macro and enhance RNG security

Introduced the generate_random_bytes! macro to eliminate repetitive code for generating random bytes.
Replaced rand::thread_rng with rand::rngs::OsRng to utilize a cryptographically secure random number generator.
Updated cryptographic key constants to use fixed-size arrays instead of slices for improved type safety.
Enhanced error handling in random byte generation by adding expect messages.
Removed unused imports and commented-out code to clean up the codebase.
Improved consistency in hash function implementations and key management.

This refactoring improves code maintainability, readability, and security by centralizing random byte generation and ensuring the use of secure RNG sources.

 replace is_none_or with match statements
This commit refactors the code to replace the use of the is_none_or
method with match statements. This change improves compatibility with
a lower minimum supported Rust version, since is_none_or is only
available in stable since Rust 1.82.0.

Updated is_support method in TwoCellAnchor to use match.
Refactored _has_vertical and has_horizontal methods in MergeCells
to use match for start and end row/column checks.

 replace hex decoding/encoding with hex-literal
This commit simplifies the test code by replacing custom hex decoding
and encoding functions with the hex-literal crate. This change
improves readability and reduces boilerplate code in the tests.

Removed decode_hex and encode_hex helper functions.
Updated test cases to use the hex! macro for byte literals.

 replace OsRng with thread_rng for random byte generation
This commit refactors the random byte generation in the gen_random_bytes function by replacing the use of OsRng with thread_rng. This change simplifies the code and improves performance by using the thread-local random number generator.

Updated the gen_random_bytes function to utilize thread_rng() for filling the byte vector.
Removed the OsRng import as it is no longer needed.

 replace ahash with std::hash::DefaultHasher
This commit removes the ahash dependency and replaces its usage with the standard library's DefaultHasher. This change simplifies the dependency tree and reduces external dependencies while maintaining the same hashing functionality.

Removed the ahash crate from Cargo.toml.
Updated the SharedStringItem struct to use std::hash::DefaultHasher instead of AHasher from ahash.

 rename cell_collection to cells, row_dimensions to rows, and column_dimensions to columns

Renamed cell_collection to cells for consistency and clarity.
Renamed row_dimensions to rows for improved readability.
Renamed column_dimensions to columns for better understanding.
Updated all method calls and references accordingly.

 replace ThinVec with rust-lang Vec

 replace lazy_static with built-in std::sync::OnceLock

 remove lazy_static, add comments

 fix typos, clippy lints

 reimplement and cleanup helper/crypt.rs

 replace getrandom with rand for random byte generation

Replaced getrandom crate with rand to simplify and modernize random number generation.
Updated gen_random_* functions to use rand::thread_rng().fill() for generating random bytes of specified lengths.
Adjusted encryption functions to use gen_random_bytes for generating random values.

Style

 fix many clippy lints
 fix formatting and improve check_sheet_name logic

Corrected the documentation comment for get_sheet_by_name to fix the syntax error.
Refactored the check_sheet_name method to use a more concise if-else structure for clarity.
Improved readability of the code while maintaining the same functionality.

 add modern rustfmt.toml and apply it
 enable more lints

Commit Statistics

33 commits contributed to the release.
11 days passed between releases.
31 commits were understood as conventional.
1 unique issue was worked on: #249

Commit Details

view details

#249

Optimize code (1f81738)

Uncategorized

Fix many clippy lints (e60fa40)
Fix formatting and improve check_sheet_name logic (6b3c774)
Update zip dependency to version 2.2.2 (ea1d8a2)
Update is_address function for regex initialization (4eab5d7)
Remove unnecessary qualification (505d766)
Add categories to Cargo.toml (37d1139)
Allow multiple versions of thiserror (de9f38f)
Clippy lints (8d41bc1)
Enhance documentation for encryption-related functions (f8d9659)
Simplify make_buffer function and improve worksheet processing (72faf4f)
Remove leading underscores from method names and allow dead_code at crate level (4dcea0e)
Centralize constants into a dedicated module (26e7021)
Add modern rustfmt.toml and apply it (98fcf7f)
Abstract random byte generation with macro and enhance RNG security (b1916f7)
Enhance Excel timestamp conversion with additional calendar systems (92a1af9)
Remove unused js feature from Cargo.toml (995f40d)
Set minimum supported Rust version to 1.79.0 (f76be7a)
Replace is_none_or with match statements (9511418)
Move lint configurations to lib.rs (6b2c88e)
Sort dependencies in Cargo.toml (517cf8c)
Replace hex decoding/encoding with hex-literal (739446e)
Replace OsRng with thread_rng for random byte generation (b65b035)
Replace ahash with std::hash::DefaultHasher (12926d6)
Rename cell_collection to cells, row_dimensions to rows, and column_dimensions to columns (cb80120)
Replace ThinVec with rust-lang Vec (714bf07)
Replace lazy_static with built-in std::sync::OnceLock (aaaec5f)
Remove lazy_static, add comments (f3de769)
Fix typos, clippy lints (71f0572)
Reimplement and cleanup helper/crypt.rs (30e3e9a)
Replace getrandom with rand for random byte generation (a317cce)
Enable more lints (48b864b)
#248 (5e20000)

Two suggestions:

I would suggest leaving ThinVec alone unless it is absolutely ok to add 16 bytes to each of those structures.

ahash's hasher is much faster than Rust's default hasher. Therefore ahash is usually used when performance is concerned.  I suggest doing some profiling to check.

These two changes may affect CPU and RAM usage on low level hardware such as ARM chips or low power x86.

ahash was only used in a single place:
https://github.com/MathNya/umya-spreadsheet/blob/b58e16edb0bb11cca9561dd7db898183be551002/src/structs/shared_string_item.rs#L68-L81
All other Hashs were already using the default implementation. Do you really see that much difference in performance?
Regarding ThinVec: have you measured your use cases against this PR? Rust does a lot of clever optimization under the hood and ThinVec even tells you, that if you construct it, the allocations might be actually more than std::Vec:

Of course, any actually constructed ThinVec will theoretically have a bigger allocation, but the fuzzy nature of allocators means that might not actually be the case.

One big area one could think about improving would be error handling. Currently (this PR) we have 601 .unwrap() calls in this codebase.

Regarding ThinVec: have you measured your use cases against this PR? Rust does a lot of clever optimization under the hood and ThinVec even tells you, that if you construct it, the allocations might be actually more than std::Vec:

Well, Rust layout follows a very stable pattern and it it one of the things that Rust does not optimize away for you.  Therefore a large type stays a large type after all optimizations.  That's why you still need to consider size of types in Rust programs.  AFAIK Rust never touches the memory layout of types.
ThinVec is used in the FireFox browser pretty extensively so I suppose it would be stable enough.
An alternative is to use Option<Box<Vec>> and automatically puts in code to change empty Vec's into None. This has the same effect as using ThinVec.  Using Box<Vec> itself is also a solution, which also has the same size as using ThinVec, but then you have an extra allocation for each instance.  Memory usage does not change, but you pack the parent type more closely so you get better cache locality.
Also remember collections such as HashMap may contain up to 70% empty slots, meaning that if your type contains empty spaces then those empty slots just multiply.

ThinVec is used in the FireFox browser pretty extensively so I suppose it would be stable enough.

Firefox uses smallvec which is developed (in part) by Mozilla and is actively maintained. It has a different architecture and idea behind it then the (seemling unmaintained) thin-vec you refer to.

Currently this crate mostly exists to facilitate Gecko (Firefox) FFI, but it works perfectly fine as a native rust library as well.

Not sure myself but it says it is developed to interop with Gecko.
Anyway, I'm not particularly hung up on it. However I have maintained crates that using it makes a lot of sense, especially where there is an enum with a Vec that makes it 16 bytes larger and it kills caching. Can't move to Box<Vec> because it forces another allocation and redirection that slows things down.  In such cases ThinVec has a clear benefits in tight loop benchmarks.

I'm happy to reimplement it if @MathNya thinks it useful. Or implement smallvec instead.

I'm happy to reimplement it if @MathNya thinks it useful. Or implement smallvec instead.

SmallVec solves a different problem: avoiding allocations. It does not save memory.  A SmallVec can be larger than a Vec depending on your config.
You'd need to bemchmark to check if using SmallVec is worth it as it stores small number of items inline to avoid allocations, at the expense of larger memory footprint.

A bit more details on the two different crates.
SmallVec I suggest we use on transitory purposes, for example preparing a list for processing inside a function.  The list is destroyed after use and never goes outside of the function.  Using SmallVec this way keeps everything on the stack for direct access and cache locality, and avoids all allocations if the number of items is often small.  I have benchmarked a lot of such usages and it usually yields 10-15% speed improvements alone.
ThinVec should be used where every byte counts.  If using a Vec does not affect a type's size (e.g. when another variant is larger) or the Vec is usually empty (where you can use Box<Vec>), then there is no need.  I tend to use it in cases where having a Vec automatically makes a LOT of types larger, for example in a fundamental type which is embedded everywhere inside a lot of other data structures.

If you want to substantially change the API you would end in a near-total re-write of the whole crate. If you do that, you could implement many things more efficiently, e.g. you could abstract the whole Unit types into a single generic one using Cow<'_, str>, e.g.:
use std::{
    borrow::Cow,
    fmt::Display,
    str::FromStr,
};

#[derive(Clone, Default, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Value<T> {
    value: Option<T>,
}

impl<T> Value<T> {
    pub(crate) fn get_value(&self) -> Option<&T> {
        self.value.as_ref()
    }

    pub(crate) fn set_value(&mut self, value: T) -> &mut Self {
        self.value = Some(value);
        self
    }

    pub(crate) fn remove_value(&mut self) -> &mut Self {
        self.value = None;
        self
    }

    pub(crate) fn has_value(&self) -> bool {
        self.value.is_some()
    }
}

impl<T: Display + Default + Clone + FromStr> Value<T> {
    pub(crate) fn set_value_string<S: Into<String>>(&mut self, value: S) -> &mut Self {
        match value.into().parse::<T>() {
            Ok(parsed) => self.set_value(parsed),
            Err(_) => self.remove_value(), // Or handle the error differently, e.g., log it
        }
    }

    pub(crate) fn get_value_or_default(&self) -> T {
        self.value.clone().unwrap_or_default()
    }
}

impl<'a> Value<Cow<'a, str>> {
    pub(crate) fn get_value_string(&self) -> Cow<'a, str> {
        self.value.clone().unwrap()
    }
}

macro_rules! create_and_export_ValueType {
    ($t:ty, $i:ident) => {
        pub type $i = Value<$t>;

        impl $i {
            pub(crate) fn get_value_string(&self) -> Cow<'a, str> {
                self.value
                    .as_ref()
                    .map_or_else(|| Cow::Borrowed(""), |v| Cow::Owned(v.to_string()))
            }
        }
    };
}

create_and_export_ValueType!(bool, BoolValue);
create_and_export_ValueType!(u8, ByteValue);
create_and_export_ValueType!(i8, SignedByteValue);
create_and_export_ValueType!(f64, DoubleValue);
create_and_export_ValueType!(i16, I16Value);
create_and_export_ValueType!(i32, I32Value);
create_and_export_ValueType!(i64, I64Value);
create_and_export_ValueType!(u16, U16Value);
create_and_export_ValueType!(u32, U32Value);
create_and_export_ValueType!(u64, U64Value);
pub type StringValue<'a> = Value<Cow<'a, str>>;

I started on that adventure once or twice, but you just need to change too much to do it incrementally. Many functions want &str from the get_value_string method, others expect String etc.

Agree. Not recommended to change the current API too much as it'll be too much rewrites for existing code.

In terms of memory efficiency (@schungx) I did some profiling with valgrind and the memory consumption upon cloning a Worksheet is huge because of the way Color, Border and Borders are represented. This is v2.2.1:

This is the memory layout after re-implementing the structs, using the rgb crate as basis to represent colors:

I personally use umya to parse a plan from work, iterate over it, and color cells that contain specific names (to create personalized versions for many people at once).
Here massif data from Valgrind:
Master (v2.2.1)

This PR @ https://github.com/MathNya/umya-spreadsheet/pull/250/commits/89016c76ca92580bba3f0a27a7375239204f1054

This PR @ 3c26246

We could probably do alot more optimization, but this change was minimal-invasive and did not require changing alot of internal APIs.

It is interesting to see what happens if you remove all the #[inline] instructions and let the compiler decide for himself (otherwise same code as this PR @ 3c26246).
No #[inline], lto = false

No #[inline], lto = true

So letting the compiler decide for itself and using lto if possible in the final application is better than force-inlining that many functions. I will push a commit to this PR to remove all #[inline] instructions.

@schungx
@mxsrm
Thank you for your support through the end of the year.
I am not very knowledgeable about performance enhancement and have left it to you.
I would be grateful if you could continue to support us if you would like.
I look forward to working with you again next year.

I don't know if it's valid there but in CSS it's allowed. #333 is short for #333333
As per: https://www.w3schools.com/css/css_colors_hex.asp

3 Digit HEX Value
Sometimes you will see a 3-digit hex code in the CSS source.
The 3-digit hex code is a shorthand for some 6-digit hex codes.
The 3-digit hex code has the following form:
#rgb
Where r, g, and b represent the red, green, and blue components with values between 0 and f.

I will push a commit to this PR to remove all #[inline] instructions.

I would not suggest that.  My own benchmarks have #[inline] substantially speeding up the linking phase when LTO is turned on.
If LTO is turned off, any function without #[inline] will never be inlined, and there are a LOT of simple accessor functions.
#[inline] is never for memory, it is intended for speed.  Without #[inline], my release build links in 4 hours with LTO, while with #[inline] is it less than 1 hour.

I don't know if it's valid there but in CSS it's allowed. #333 is short for #333333
As per: https://www.w3schools.com/css/css_colors_hex.asp

3 Digit HEX Value
Sometimes you will see a 3-digit hex code in the CSS source.
The 3-digit hex code is a shorthand for some 6-digit hex codes.
The 3-digit hex code has the following form:
#rgb
Where r, g, and b represent the red, green, and blue components with values between 0 and f.

I implented it in that way. See a496cbe.

@schungx @mxsrm Thank you for your support through the end of the year. I am not very knowledgeable about performance enhancement and have left it to you.

Not my area of expertise either. In daily life I am an interventional neuroradiologist that enjoys coding in his free time. @schungx is more knowledgable in that regard I think.

Not my area of expertise either. In daily life I am an interventional neuroradiologist that enjoys coding in his free time. @schungx is more knowledgable in that regard I think.

Just different experiences that's all.
I happen to be maintaining an LOB application running on limitrd hardware, so memory and CPU count a lot.  For most applications I'd agree there may not be a big impact, unless we're dealing with massive spreadsheets with tens of thousands of rows, like some users who complained in issues.


Missing Withdraw Command?

Thanks to you I created and published the Smart Contract. Then how do we withdraw the matic or ethereum accumulated in the contract to our wallet? As far as I understand that withdraw command is not included in this smart contract. So how are we going to integrate this withdraw command into our contract that we've published?

Use ** to withdraw the full amount.


[CLI] Not showing remnux installed after successful install

SIFT VM and SIFT CLI at latest versions (1.7.1 CLI and v2018.35.0 saltstack).
After resolving previous issue https://github.com/REMnux/remnux-cli/issues/3, had successful install but no indication that REMnux is installed:

COMPLETED SUCCESSFULLY -- Success: 241, Failure: 0
red@siftworkstation -> ~
$ remnux version
<EMAIL_ADDRESS>remnux-version: notinstalled

This results in an inability to update remnux because it is 'not installed'. Running remnux upgrade does work to installed/upgrade, just not updating.

Definitely a bug, I'll take a look.


WIP: Remove only local sources

A tidier way to remove any local sources before trying to vendor.
(If you're ok with the idea give me a thumbs up and I'll write some tests to cover it.)

Could you add some more tests please?
Also I am keen to keep the code as robust as possible. If the set of valid source types evolves over time, then a future source type might be misclassified as ignorable or not. Can you be specific in the code as to how all source types are handled and fail if an unexpected source type is found?

Absolutely. Will do.

Am mulling over a good way to do the tests... I can redirect crates.io to the crates.io ending in .git in a test but proving that it was any different .... ah maybe I could peek into the generated lock file. Bingo that's what I was looking for.

FYI cargo vendor calls this option --respect-source-config


Improve css class gathering

Solves #28

This is the most simple implementation of the CSS class gathering tool.
It only finds strings inside a given macro call, girouette.core/css by default.


Update .mergify.yml

Codecov Report

Merging #111 (63329d7) into master (26875f9) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #111   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            5         5
  Lines          106       106
=========================================
  Hits           106       106

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


Subscription button overlaps dark-theme button in mobile screen

I didn't know whether this issue has been solved yet or not. But as the title said, the subscription button overlaps the dark-theme button when I switched to the mobile screen

I think the Portal button shouldn’t overlap the menu actually, the problem is that that button has a higher z-index value. You can fix this by injecting the following code:
<style>
.gh-portal-triggerbtn-iframe {
	z-index: 3;
}
</style>


Fix file mapping for New Haven

Summary:

Fixes URL for downloading the CSV file
Fixes the name of the output file

Tests:

I removed any new-haven files locally.
Next, ran npm run download which downloaded the csv file to data/raw
Then, ran npm run convert which created a geojson file in data/geojson
Finally, ran npm run normalize which outputted a geojson file in data/normalized

Example object from data/normalized:
(base) camillele @ Camilles-MBP ~/Desktop/waterthetrees/wtt_area/data/normalized
└─ $ ▶ gj new-haven.geojsons
{
  "type": "Feature",
  "properties": {
    "scientific": "__UNKNOWN",
    "genus": " ",
    "species": " ",
    "common": "*Unknown",
    "dbh": "2",
    "planted": "1/11/2017",
    "address": "308 Alden Av",
    "ref": "71",
    "id":<PHONE_NUMBER>067315,
    "sourceId": "new-haven",
    "city": "New Haven",
    "country": "United States",
    "email"<EMAIL_ADDRESS>    "download": "https://urban-resources-initiative.carto.com/api/v2/sql?filename=newhavenstreettrees_carto_mapping_05242022&q=select+*+from+public.newhavenstreettrees_carto_mapping_05242022&format=csv&bounds=&api_key=&skipfields=the_geom_webmercator,the_geom",
    "info": "https://urban-resources-initiative.carto.com/me",
    "lat": 41.3245735200001,
    "lng": -72.96377563,
    "count": 1
  },
  "geometry": {
    "type": "Point",
    "coordinates": [
      -72.96377563,
      41.3245735200001
    ]
  },
  "id":<PHONE_NUMBER>067315
}

We ran through the tests in our meeting today and everything looks good. 👍


Update workflows to build jdk15 instead of jdk14

Now that jdk14 is 'done' and replaced with jdk15, we should update the workflow files.

To update:
https://github.com/AdoptOpenJDK/build-jdk/blob/master/.github/workflows/test.yml#L19
and
https://github.com/AdoptOpenJDK/build-jdk/blob/master/.github/workflows/test.yml#L45

Closed by #14


I'm getting errors running this

I got:
/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/async.js:61
        fn = function () { throw arg; };
                           ^

TypeError: Cannot read property 'success' of undefined
    at Client.<anonymous> (/Users/alex/3module/authy-testing-sample/app.js:274:20)
    at Client.tryCatcher (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/util.js:16:23)
    at Promise.errorAdapter [as _rejectionHandler0] (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/nodeify.js:35:34)
    at Promise._settlePromise (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/promise.js:566:21)
    at Promise._settlePromise0 (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/alex/3module/authy-testing-sample/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:800:20)
    at tryOnImmediate (timers.js:762:5)
    at processImmediate [as _immediateCallback] (timers.js:733:5)

and then when I check on twilio it shows that I was sent back a Success! message.
Not sure what to do from here.

The 'Success!' message you see in the Twilio Console is probably coming from the response of the /verification webhook. Are you able to see the phone verification message body in the Twilio Console (i.e. Your  verification code is: XXXX)? Have you cloned the repo recently? What's the version of the authy-client you have in the package.json? I remember seeing some issues with that library before throwing similar error messages.


Bracket access to projection paramters

@djhoese wrote:

One feature that was brought up on that xarray/geoxarray issue was wanting to be able to take a CRS-like object and use the getitem method to use bracket access to projection parameters like my_crs['lon_0'] or something like that. Not sure if proj.4 parameter naming is the best thing, but yeah it was brought up and I think it could be a good idea. Not sure how to accomplish this with the current interfaces though given how everything is split up in to its own class.

So the requested feature here is perhaps about an easy way to access parameters. Right now how you would do it is by recursively accessing the correct nested sub-elements. E.g. metermultiplier = crs.toplevel.unit.unitmultiplier.value. In a way this would be the correct way to do it, as things would be logically grouped.
I could see the usefulness of also providing more convenient access in a way that kind of "searches" for the parameters for you based on the name, without the user needing to know about their nested structure. As you say, using the proj4 names may not be ideal here, but then again, all naming conventions come from one or another format spec, so it really just depends what you're used to I guess.
One way might be to implement a "get" and "set" method, where you could also supply which format you are using for your search and for the supplied and return values. That way, users more familiar with proj4 could use that, or those familiar with OGC WKT could use that, etc.

For now, one way around this for those used to proj4, I have added an as_dict arg which can be used to get proj4 as a dict. Something like this:


Test GitHub CI

This PR was opened just for test purposes, to understand how GitHub Actions workflow will act, when PR is from forked repo.


Updates of x/sys, x/tools, etc.
What did you see instead?
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/xerrors in REDACTED/pkg/mod/cache/vcs/3dae2e2383f75162530e523fa60c46a0150bec632a2a57ec3fee7fac3fc3648c: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/xerrors/': The requested URL returned error: 502
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/crypto in REDACTED/pkg/mod/cache/vcs/de5fd3af413a4f3f969455ae522b4002fcb7bb4c158f339396dfc77710c9007d: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/crypto/': The requested URL returned error: 502
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/crypto in REDACTED/pkg/mod/cache/vcs/de5fd3af413a4f3f969455ae522b4002fcb7bb4c158f339396dfc77710c9007d: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/crypto/': The requested URL returned error: 502
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/sys in REDACTED/pkg/mod/cache/vcs/76a8992ccba6d77c6bcf031ff2b6d821cf232e4ad8d1f2362404fbd0a798d846: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/sys/': The requested URL returned error: 502
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/sys in REDACTED/pkg/mod/cache/vcs/76a8992ccba6d77c6bcf031ff2b6d821cf232e4ad8d1f2362404fbd0a798d846: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/sys/': The requested URL returned error: 502
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/text in REDACTED/pkg/mod/cache/vcs/5b03666c2d7b526129bad48c5cea095aad8b83badc1daa202e7b0279e3a5d861: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/text/': The requested URL returned error: 502
go get: upgrading<EMAIL_ADDRESS>git ls-remote -q https://go.googlesource.com/net in REDACTED/pkg/mod/cache/vcs/4a22365141bc4eea5d5ac4a1395e653f2669485db75ef119e7bbec8e19b12a21: exit status 128:
        fatal: unable to access 'https://go.googlesource.com/net/': The requested URL returned error: 502

This seems to affect all the x/ repositories. I am able to go get modules from other repositories normally. This problem also occurs in other modules, so it is not limited to the one I happen to be working on right now.

go.googlesource.com is hosted by Google, which is having a large-scale outage right now. Since you're using modules, you might try setting GOPROXY=https://proxy.golang.org. proxy.golang.org is also hosted by Google, but seems to be tolerating the outage better.

FYI for @tparker @draftcode, who have helped us with Gerrit in the past, but I doubt they can do much in this case.

GOPROXY=https://proxy.golang.org

Confirmed that setting this makes the symptoms go away for me, thanks for the workaround. (And yes, I realize the proxy will eventually not be a workaround 🙂).

Outage is over.

Still facing the same issue
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
# cd .; git clone https://go.googlesource.com/sys /go/src/golang.org/x/sys
Cloning into '/go/src/golang.org/x/sys'...
fatal: unable to access 'https://go.googlesource.com/sys/': The requested URL returned error: 502
package golang.org/x/sys/unix: exit status 128
https://go.googlesource.com/sys/ is returning 502.
I have already added GOPROXY=https://proxy.golang.org to my env variables still same issue

Duplicate of https://github.com/golang/go/issues/36048. Seems to be a temporary blip.

I don't think you would have gotten that error if GOPROXY were set correctly. Either way, a months-old bug isn't the place to discuss it; please file a new one if you're having trouble.


[verbatim] Couldn't find face with no name

This is a very strange bug. Minimal file required for replication is included here, but the main part is here:
\begin[class=book]{document}
\script[src=packages/verbatim]
\font[filename=libertine-font/LinLibertine_R.otf]%Probably any font file
\define[command=verbatim:font]{\font[family="Menlo"]}%Other fonts may probably need more or less verbatim lines
\chapter{x }%Required, no idea why
\begin{verbatim}
x
x
x
[... and many more x's]
\end{verbatim}
\end{document}

This leads to error ! Couldn't find face  at fontbug.sil, or with --trace:
! Couldn't find face '' at fontbug.sil
  at verbatim on line: 6, column: 1
stack traceback:
	.../local/Cellar/sile/<IP_ADDRESS>/share/sile/core/utilities.lua:53: in function 'core/utilities.error'
	.../Cellar/sile/<IP_ADDRESS>/share/sile/core/harfbuzz-shaper.lua:64: in local 'callback'
	/usr/local/Cellar/sile/<IP_ADDRESS>/share/sile/core/font.lua:98: in field 'cache'
	.../Cellar/sile/<IP_ADDRESS>/share/sile/core/harfbuzz-shaper.lua:35: in method 'shapeToken'
	...ocal/Cellar/sile/<IP_ADDRESS>/share/sile/core/base-shaper.lua:53: in method 'measureChar'
	...cal/Cellar/sile/<IP_ADDRESS>/share/sile/core/measurements.lua:111: in function <...cal/Cellar/sile/<IP_ADDRESS>/share/sile/core/measurements.lua:110>
	(...tail calls...)
	/usr/local/Cellar/sile/<IP_ADDRESS>/share/sile/core/length.lua:9: in method 'absolute'
	...ocal/Cellar/sile/<IP_ADDRESS>/share/sile/core/base-shaper.lua:33: in local 'f'
	/usr/local/Cellar/sile/<IP_ADDRESS>/share/sile/core/settings.lua:48: in field 'temporarily'
	...ocal/Cellar/sile/<IP_ADDRESS>/share/sile/core/base-shaper.lua:29: in method 'measureSpace'
	...ocal/Cellar/sile/<IP_ADDRESS>/share/sile/core/base-shaper.lua:143: in method 'makeSpaceNode'
	...cal/Cellar/sile/<IP_ADDRESS>/share/sile/languages/unicode.lua:28: in method 'makeGlue'
	...cal/Cellar/sile/<IP_ADDRESS>/share/sile/languages/unicode.lua:109: in function <...cal/Cellar/sile/<IP_ADDRESS>/share/sile/languages/unicode.lua:84>

In generated pdf, last ~5 characters are missing (X's span two whole pages and then the document ends). I don't think that different fonts will fix it, but they may require different amount of verbatim characters. The fact that filename with family font specification is mixed seems significant, as adding
  measureSpace = function(self, options)
    local ss = SILE.settings.get("document.spaceskip")
    if ss then
      SILE.settings.temporarily(function()
        SILE.settings.set("font.size", options.size)
        SILE.settings.set("font.family", options.family)
        -- FOLLOWING LINE
        if options.filename then SILE.settings.set("font.filename", options.filename) end
        ss = ss:absolute()
      end)
      return ss
    end

to base-shaper.lua:32 seems to fix it. But I have no idea why, because the replication criteria seem extremely arbitrary, so it almost seems like there is a different underlying issue. But I may be mistaken, so instead of opening PR, I'd like someone else, with deeper knowledge of the internals, to look at it.

I'm unable to replicate this bug at all, so it's kind of hard to verify what affects it. Can you comment on whether the recent merge of #607 changed this at all?

It happens 100% regularly for me, even on master - OS X 10.11.6.
It is important that default font is set as filepath and verbatim:font is set as family. Omitting either font setting or reversing their types "fixes" the error. But it happens for each font file/family I have tested.
Try adding more lines (x's), what is included is the minimum required to trigger the bug on my computer, but if you have slightly different font it may need more verbatim lines. There is no upper bound of verbatim content to trigger this, only lower bound.


deer author: I dont know how does the cost volume form, and what is it means?

and why form a 4D cost volume it make a good result. Which paper can I refer to? thanks very much.

You can refer to GC-Net to find more details about the cost volume. But I still confused about that.
Why do they design this way? Besides, PSMNET cost volume looks a little different from GC-Net.


docs error more explanatory

Hey, I just made a Pull Request!
Try to explain a bit more about about what might have caused the docs not being found when building techdocs.
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Fixes #15697

bump. any feedback here is welcome. The current message can be quite confusing for users.


Race at client connection can lead to invalid client vs cluster state

A client can try to reconnect to the cluster after a disconnection/timeout and it might lead to an inconsistent state:

Client tries to connect to Server A and times out
Client tries to connect to Server B and succeeds
It is published that Client is connected to Server B
The first attempt at 1 finishes its process and it is published that Client is connected to Server A

In the end, the Client will be connected  to Server B but the cluster thinks it is connected to Server A. So no information about the events or replies to queries will ever be sent to the Client because Server B that is responsible for it has no connection. I have logs in case it is needed to confirm the issue.

Fixed by #276


Retarget project to portable class library

I want to use this library in Universal Windows Platform, so I retarget it in order to support more platform. Since the Task Parallel Library is supported in all the platforms after Windows 8, the conversion is direct.

Code contract (Contract.Requires) currently cannot work on .net core, so I must find someway instead.


distance has no effect

Even when taken directly from the example, distance no effect. I'm not sure what function it is supposed to serve exactly, but from what I can see it makes no difference on the color scheme at all, no matter what value I insert.

Check the example: http://c0bra.github.io/color-scheme-js/
It has no effect on mono or contrast schemes

Going to close since there's been no reply.


mac运行任务report报告也没有生成

贴上运行情况，Running面板的输出

贴上运行情况，Running面板的输出

Exception in thread "Thread-28" org.yaml.snakeyaml.error.YAMLException: java.nio.charset.MalformedInputException: Input length = 1
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:215)
at org.yaml.snakeyaml.reader.StreamReader.ensureEnoughData(StreamReader.java:173)
at org.yaml.snakeyaml.reader.StreamReader.ensureEnoughData(StreamReader.java:168)
at org.yaml.snakeyaml.reader.StreamReader.peek(StreamReader.java:123)
at org.yaml.snakeyaml.scanner.ScannerImpl.scanToNextToken(ScannerImpl.java:1204)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:320)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
at org.yaml.snakeyaml.parser.ParserImpl$ParseImplicitDocumentStart.produce(ParserImpl.java:210)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:161)
at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:152)
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:156)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:178)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:493)
at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:486)
at com.g3g4x5x6.nuclei.ultils.CommonUtil.getMapFromYaml(CommonUtil.java:157)
at com.g3g4x5x6.nuclei.ultils.CommonUtil.getTemplateInfoFromPath(CommonUtil.java:199)
at com.g3g4x5x6.nuclei.panel.tab.TemplatesPanel$12.visitFile(TemplatesPanel.java:538)
at com.g3g4x5x6.nuclei.panel.tab.TemplatesPanel$12.visitFile(TemplatesPanel.java:533)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
at com.g3g4x5x6.nuclei.panel.tab.TemplatesPanel.walkFiles(TemplatesPanel.java:533)
at com.g3g4x5x6.nuclei.panel.tab.TemplatesPanel.getAllTemplatesFromPath(TemplatesPanel.java:521)
at com.g3g4x5x6.nuclei.panel.tab.TemplatesPanel.lambda$refreshDataForTable$2(TemplatesPanel.java:430)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181)
at org.yaml.snakeyaml.reader.UnicodeReader.read(UnicodeReader.java:118)
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:180)
... 23 more

贴上运行情况，Running面板的输出

[WRN] Found 44 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[WRN] Found 35 templates with syntax warning (use -validate flag for further examination)
[INF] Current nuclei version: v3.1.1 (outdated)
[INF] Current nuclei-templates version: v9.7.1 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 0
[INF] Templates loaded for current scan: 7486
[INF] Executing 7341 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 163 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 9
[INF] Templates clustered: 1254 (Reduced 10998 Requests)
[ptr-fingerprint] [dns] [info] <IP_ADDRESS>.in-addr.arpa [<IP_ADDRESS>.broad.xm.fj.dynamic.163data.com.cn.]
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 30 times
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 32 times
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 32 times
[INF] Using Interactsh Server: oast.fun
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 30 times
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 30 times
[INF] Skipped <IP_ADDRESS>:6060 from target list as found unresponsive 30 times
[INF] Skipped <IP_ADDRESS>:81 from target list as found unresponsive 30 times
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 31 times
[INF] Skipped <IP_ADDRESS>:81 from target list as found unresponsive 31 times
bash-3.2$
e before v3 for continued support. loaded with deprecated protocol syntax, updat
bash: [WRN]: command not found
urther examination)nd 35 templates with syntax warning (use -validate flag for f
bash: syntax error near unexpected token (' bash-3.2$ [INF] Current nuclei version: v3.1.1 (outdated) bash: syntax error near unexpected token ('
bash-3.2$ [INF] Current nuclei-templates version: v9.7.1 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 0
[INF] Templates loaded for current scan: 7486
[INF] Executing 7341 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 163 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 9
[INF] Templates clustered: 1254 (Reduced 10998 Requests)
[ptr-fingerprint] [dns] [info] <IP_ADDRESS>.in-addr.arpa [<IP_ADDRESS>.broad.xm.fj.dynamic.163data.com.cn.]
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 30 times
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 32 times
[INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 32 times
[INF] Using Interactsh Server: oast.fun
[INF] Skipped bash: syntax error near unexpected token (' bash-3.2$ [WRN] Scan results upload to cloud is disabled. bash: [WRN]: command not found bash-3.2$ [INF] New templates added in latest release: 0 bash: [INF]: command not found bash-3.2$ [INF] Templates loaded for current scan: 7486 bash: [INF]: command not found plates.2$ [INF] Executing 7341 signed templates from projectdiscovery/nuclei-tem  bash: [INF]: command not found bash-3.2$ [WRN] Executing 163 unsigned templates. Use with caution. bash: [WRN]: command not found bash-3.2$ [INF] Targets loaded for current scan: 9 bash: [INF]: command not found bash-3.2$ [INF] Templates clustered: 1254 (Reduced 10998 Requests) bash: syntax error near unexpected token ('
27.broad.xm.fj.dynamic.163data.com.cn.]] <IP_ADDRESS>.in-addr.arpa [41.214.154.
bash: [ptr-fingerprint]: command not found
30 times$ [INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive
bash: [INF]: command not found
32 times$ [INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive
bash: [INF]: command not found
2 times2$ [INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 3
bash: [INF]: command not found
bash-3.2$ [INF] Using Interactsh Server: oast.fun
bash: [INF]: command not found
30 times$ [INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive
bash: [INF]: command not found
30 times$ [INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive
bash: [INF]: command not found
30 times060 from target list [IN30.77:81 from target list as found unresponsive
bash: [INF]: command not found
1 times2$ [INF] Skipped <IP_ADDRESS>:80 from target list as found unresponsive 3
bash: [INF]: command not found
31 times$ [INF] Skipped <IP_ADDRESS>:81 from target list as found unresponsive

能不能像这样子测试下几个简单info模板，截图我看看

能不能像这样子测试下几个简单info模板，截图我看看

不应该呀，命令都执行成功了，你看看执行的命令行的参数指定的报告目录对不对

不应该呀，命令都执行成功了，你看看执行的命令行的参数指定的报告目录对不对

我的mac好像都没有执行生产报告的命令

不应该呀，命令都执行成功了，你看看执行的命令行的参数指定的报告目录对不对

我执行命令他不会自动生成报告吗

不应该呀，命令都执行成功了，你看看执行的命令行的参数指定的报告目录对不对

需要手动加上执行生成报告的命令吗

不应该呀，命令都执行成功了，你看看执行的命令行的参数指定的报告目录对不对

我的报告位置没改过

你就把Running面板当普通cmd执行nuclei扫描试试，就是自己填充下参数，看看能不能扫完正常出报告。（这个程序也只是把nuclei的命令行参数拼接下执行的，没什么特别的呀，不知道为什么不生成报告，估计得等我弄个mac虚拟机复现下才行了）

无法复现，暂时关闭


ModuleNotFoundError

Hi @cudeso,
Receiving this error when running the script, have the latest version.
we will officially deprecate python versions below 3.10 on January 1st 2024.
Please update your codebase.
from pymisp import *
Traceback (most recent call last):
File "script.py", line 2, in
import config
ModuleNotFoundError: No module named 'config'

This means the config.py , the main configuration file, is not present in the directory where you execute the script.
Maybe it's not readable by the user you use to execute the sync, or it has been removed?

Hi I can see the config.py file in the directory, i have not changed any permissions and im executing the script as root user.

Hello. Are you running the script from the directory where the config.py file is stored?

Yes
Below is the contents of the directory i'm running out of:

Thank you, works now :)


feature mask problem

Hi, @husthuaan. I have a question about the Layer Norm. Why don't you apply mask in Layer Norm if you using adpative feature.

In LayerNorm, features won't affect others.


Configurable logger

Somewhat related to https://github.com/paritytech/substrate/issues/7019 but more than a cli flag to have logs as new line delimited JSON it would be nice to have more control of the logger and add custom filters/formatters and what not.
Logging is usually very "personal" isn't it? instead of being told I can only wear green or red pants I'd like to be able to go out with blue shorts or nothing at all :P

I think I need this too. Wanted to change logs to print time in UTC instead of local timezone, which is different on every machine, but can't find a way to configure it :confused:


[fix] Using the iterator of the new Gonum API

This change allows the v0.9.2-working2 branch to compile and work with the latest evolution of the Gonum API.
The implementation relies on the OrderedNode implementation of the iterator package.

I'll work on this after I finish submitting my last chapters to the publisher... sorry that it's taken solong


Improve seeds

Context
Its very annoying having to log in to the admin panel without an admin user so this creates an admin user plus PrivateChildcareProvider users. There are lots other things we can add to seeds but this provides benefit with getting easy access to the backend through the<EMAIL_ADDRESS>user which will only work in development at the moment.
Changes proposed in this pull request
[What has been changed?]

Closing in favour of changes here: https://github.com/DFE-Digital/npq-registration/pull/789


HTML

HTML
User Story
As a Bootcamp student
I want to keep a record of everything learned in the program
so that I can visit anytime I have a question
Acceptance Criteria
GIVEN a prework Study Guide
WHEN Visit the website in my browser
THEN I see four boxes titled HTML,CSS,Git and JavaScript with notes

Completed


feat(ex1): add WaitGroup to sync goroutes with main thread

Here is the solution for the first concurrence exercise.

Perfecto


Only works on Chrome

Through demos I have found this client only really works through Chrome.
We should support all major browsers, including the major mobile browsers (since the UX is now suitable for mobile).

This actually already works on MS Edge - it's just that the data URI is reaaaally slow.  Should solve for edge the same way as for legacy browsers - see issue #12

Edge is also crashing out in other interesting ways.

This was causing the slowness.  Should not need to use legacy mechanism... phew.

Works on Edge, Firefox, Safari, and iPad too now.


Add a scaffold operations worker for ansible tower

Adds a simple worker that waits for operations commands on the operations kafka topic.

it would be great to extract this code to common (ingress-api-client?)

Eh I would prefer not to put any operations code in the ingress-api-client.  I'm not a huge fan of some of the collector common code there in the first place and this has even less to do with ingress-api.  If anything collector-common and operations-common gems would be better.

Nice.  +1 for the shared code idea, but not in a client...we need some sort of shared topo gem...perhaps topo core?  Either way, that's for the future.

Yeah actually topo-core could be a good place :+1:


fix #21 update to use newer gradle APIs

Hey Eric, thank you so much for the pull request.
I just realized that Travis wouldn't build the JDK 8 project anymore, so I had to update it to openjdk11. Would you mind rebasing to the newest master, to build it cleanly?
Thank you!

Fixed the ITs that I completely missed in the first pass.
Happy to discuss with you regarding the changes, as I'm not sure I got them all 100% correct (despite tests now passing). There may be some confusion wrt the role of the jcstress configuration vs. the sourceset.

Hello again.
Thank you for fixing the tests. I saw that they were failing, but unfortunately I had little time to look into details.
I'm by no means a Gradle or Jcstress expert - but the test coverage is pretty okay, so I think it should be okay. I will try to find some time tomorrow to locally run some old jcstress suites - either mine, or some from the Internet. I'll make sure that they pass with the new plugin version, and then we're ready to go.
Thank you again for the contribution!
Jerzy

Hello Eric.
I've checked, approved and merged. Is there something else you'd like to add here before I release a new version?
Thanks,
Jerzy


The e2e test failed with the latest web debugger client

The latest version of web debugger client (client @ 572e3f41) cannot pass the e2e test: https://github.com/es-meta/esmeta/actions/runs/4083075600.
Thus, it is impossible to update its version, and we need to figure out what happens in this version.

It is resolved by #138.


Enable the graphpkgfetcher to pull build nodes from upstream repos if available V2

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

[x] The toolchain has been rebuilt successfully (or no changes were made to it)
[x] The toolchain/worker package manifests are up-to-date
[x] Any updated packages successfully build (or no packages were changed)
[x] Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
[x] Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
[x] All package sources are available
[x] cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
[x] LICENSE-MAP files are up-to-date (./SPECS/LICENSES-AND-NOTICES/data/licenses.json, ./SPECS/LICENSES-AND-NOTICES/LICENSES-MAP.md, ./SPECS/LICENSES-AND-NOTICES/LICENSE-EXCEPTIONS.PHOTON)
[x] All source files have up-to-date hashes in the *.signatures.json files
[x] sudo make go-tidy-all and sudo make go-test-coverage pass
[x] Documentation has been updated to match any changes to the build system
[ ] Ready to merge

Summary

For running fast builds it would be great to avoid rebuilding nodes if they are already available from a remote repo. We should scan each build node we find in the graph and try to match it to an exact .rpm file in the repo. If we can make an exact update the graph node's rpmPath to point into the cache instead of the expected out/RPMS location.
The scheduler will still treat the nodes in the same way as before, if all the expected files are present, and we don't want to rebuild it for another reason, skip it. If we do need to rebuild it for any reason the .rpms will end up in out/RPMS still.
To make sure delta packages are not confused with normal packages we will only place .rpm files into out/RPMS if they are rebuilt by the scheduler, regardless of if the user selects PACKAGE_BUILD_LIST or not. The built package summary lists these packages in different lists now.
Change Log

graphpkgfetcher tool will try and hydrate build nodes if an exact matching .rpm is available from a repo.
scheduler will know to check the delta nodes for prebuilt rpms if enabled, and will skip building new .rpms if all expected files are available in either out/RPMS or the cache.

Does this affect the toolchain?

NO
Test Methodology

Local builds only for now

This will require a rebase to drop the duplicate commit once https://github.com/microsoft/CBL-Mariner/pull/5587 is merged.

@dmcilvaney If the motivation is to avoid building nodes that are available in the upstream-repo,

graphPreProcessor can be updated to check upstream as well
All the RUN nodes with corresponding RPMs can be cloned into a PREBUILT node
As PACKAGE_REBUILD_LIST will create a goal node directly linking to RUN node, the build will happen
Additionally downloading the RPM will speed up tdnf-install time during pkgWorker. But IMHO this is a separate optimization.

I had considered it, but unfortunately prebuilt nodes will never be rebuilt. I want the delta nodes to be "normal" build nodes which can be rebuilt if the scheduler decides it is needed (due to missing sub-package, or rebuilt dependency)

I couldn't completely understand the scenario where the scheduler will decide to rebuild. If node-A is depending on node-B whereas node-B is a delta node, it shouldn't be rebuilt. Only if node-B is passed as part of PKG_REBUILD_LIST, it should be rebuilt. This scenario is already handled with the PREBUILT node method. Please correct me if I'm wrong. PS: My motivation here is to avoid additional node types.

I couldn't completely understand the scenario where the scheduler will decide to rebuild. If node-A is depending on node-B whereas node-B is a delta node, it shouldn't be rebuilt. Only if node-B is passed as part of PKG_REBUILD_LIST, it should be rebuilt. This scenario is already handled with the PREBUILT node method. Please correct me if I'm wrong. PS: My motivation here is to avoid additional node types.

There are to reasons a node might be rebuild here:

If a dependency rebuilds, we want to preserve the existing behaviour and rebuild the downstream packages. (ie foo -depends-> bar, foo is a delta node. If we for some reason rebuild bar, we also want foo to rebuild).
The other option is less common but will be more important going forward (I'd like to make the scheduler smarter about rebuilding packages). If for some reason PMC doesn't have foo-subpkg1 which is defined in the local spec (maybe you just added it, or the publish failed, etc...) we will need to rebuild the entire SRPM. Here we could add logic to say "no, you aren't a prepackaged node stay a build node!" but I'd like in the future for the scheduler to be able to tell based on source hashes etc. if a node needs to be rebuilt rather than just version-release.

Thanks for the patience. I understand the intention behind introducing the DELTA state now.
I think the logic of graphpkgfetcher is complex. If my understanding is correct, the logic is as follows:

If the node is REMOTE-UNRESOLVED, try to resolve it from upstream-repo, Download the RPM, Update the node state and RPMPath
If the node is BUILD-BUILD, try resolve it from upstream-repo, Download the RPM, update the node state to DELTA, Update the RPMPath in both BUILD and RUN nodes
Update all IMPLICIT nodes with RPMPath with cached RPMPath resolved from upstream-repo. Currently this is happening only for BUILD-DELTA nodes. But I think this should be done for all REMOTE nodes.
Then the graphpkgfetcher require a complete overhaul to rewrite the logic in a simpler way. I'm not sure if this is the right time to do it. But I'm not sure if we can avoid it either. [This sentence is from co-pilot's LLM. Nicely put :P]

I'm yet to completely understand the scheduler part. My expectation is,

If any package that is dependency for BUILD-DELTA node is built, that BUILD-DELTA node should be rebuilt.
The RPMPath of both BUILD and RUN nodes should be updated to newly built RPM path.
All the nodes depending on the BUILD-DELTA and corresponding RUN node should be rebuilt with newly built RPMs.
Let me go through the scheduler changes with above perception and get back to you.

Pardon me if I'm completely incorrect here. At this point, can't we determine what are the nodes to be re-built and what are not? Similar to the logic used by schedulerutils for sub-graphing. If we do this in graphpkgfetcher, we can avoid the scheduler changes. I'm not sure if this is possible. But I think it's worth exploring.

  }

Are there cases where a node is a delta one but not cached? If not, things are getting a bit messy here with the nested if statements. For the sake of code clarity, what do you think about breaking these 4 options apart to something like this:
switch {
    case buildState.IsNodeCached(node):
        prebuiltSRPMs[node.SrpmPath] = node
    case buildState.IsNodeDelta(node):
        prebuiltDeltaSRPMS[node.SrpmPath] = node
    case buildState.IsNodeAvailable(node):
        builtSRPMs[node.SrpmPath] = node
    default:
        <That remaining logic with `unbuiltSRPMs`>

Refers to: toolkit/tools/scheduler/schedulerutils/printresults.go:68 in d0d6b7d. [](commit_id = d0d6b7d2e621f7151a689a9679075e615e569bdf, deletion_comment = False)

OK, there's clearly something wrong here, that we need to repeat the same changes. I see the maps now map to bool instead of actual nodes, but it looks like we're using these maps as sets (i.e. we don't care about the values of the booleans), what do you think about extracting a common function here?
No push for that in this particular PR, though, since it's already big.

Refers to: toolkit/tools/scheduler/schedulerutils/printresults.go:194 in d0d6b7d. [](commit_id = d0d6b7d2e621f7151a689a9679075e615e569bdf, deletion_comment = False)

}

Referencing our offline discussion: we should probably remove that, since otherwise this is going to be causing some confusion about what is a delta build . Since we're going to allow rebuilds during delta builds, these lines should no longer be here.
Since this PR is already out, and removal of these lines needs to happen after we adjust our building process, I don't mind leaving this clean-up to a separate PR. Let's just not forget about it.

Refers to: toolkit/tools/scheduler/schedulerutils/preparerequest.go:100 in d0d6b7d. [](commit_id = d0d6b7d2e621f7151a689a9679075e615e569bdf, deletion_comment = False)

Member

Sorry for the delayed response, Your understanding of the fetcher looks right. Agreed this is getting complex, but I don't see a clean way to integrate this without a full re-write which I don't have time for right now.
As for the scheduler, my goal was to keep the behavior of the scheduler unchanged as much as possible. ie, a BUILD-DELTA node is "just" a build node, and will be treated exactly the same way as a normal build node. (if a dependency rebuilt lets rebuild, if a sub package is missing lets rebuild). We want to keep the delta RPMs out of ./out/RPMS, but that means our rpm paths are wrong IF we rebuild the node.

  }

Are there cases where a node is a delta one but not cached? If not, things are getting a bit messy here with the nested if statements. For the sake of code clarity, what do you think about breaking these 4 options apart to something like this:
switch {
    case buildState.IsNodeCached(node):
        prebuiltSRPMs[node.SrpmPath] = node
    case buildState.IsNodeDelta(node):
        prebuiltDeltaSRPMS[node.SrpmPath] = node
    case buildState.IsNodeAvailable(node):
        builtSRPMs[node.SrpmPath] = node
    default:
        <That remaining logic with 'unbuiltSRPMs'>
}

Refers to: toolkit/tools/scheduler/schedulerutils/printresults.go:68 in d0d6b7d. [](commit_id = d0d6b7d, deletion_comment = False)

I'll have to double check, but I believe a delta node that is selected for rebuild will be "delta", but not "cached". If this isn't the case I'll use a switch here.

Thanks for the patience. I understand the intention behind introducing the DELTA state now.
I think the logic of graphpkgfetcher is complex. If my understanding is correct, the logic is as follows:

If the node is REMOTE-UNRESOLVED, try to resolve it from upstream-repo, Download the RPM, Update the node state and RPMPath
If the node is BUILD-BUILD, try resolve it from upstream-repo, Download the RPM, update the node state to DELTA, Update the RPMPath in both BUILD and RUN nodes
Update all IMPLICIT nodes with RPMPath with cached RPMPath resolved from upstream-repo. Currently this is happening only for BUILD-DELTA nodes. But I think this should be done for all REMOTE nodes.
Then the graphpkgfetcher require a complete overhaul to rewrite the logic in a simpler way. I'm not sure if this is the right time to do it. But I'm not sure if we can avoid it either. [This sentence is from co-pilot's LLM. Nicely put :P]

I'm yet to completely understand the scheduler part. My expectation is,

If any package that is dependency for BUILD-DELTA node is built, that BUILD-DELTA node should be rebuilt.
The RPMPath of both BUILD and RUN nodes should be updated to newly built RPM path.
All the nodes depending on the BUILD-DELTA and corresponding RUN node should be rebuilt with newly built RPMs.
Let me go through the scheduler changes with above perception and get back to you.

Pardon me if I'm completely incorrect here. At this point, can't we determine what are the nodes to be re-built and what are not? Similar to the logic used by schedulerutils for sub-graphing. If we do this in graphpkgfetcher, we can avoid the scheduler changes. I'm not sure if this is possible. But I think it's worth exploring.

Sorry for the delayed response, Your understanding of the fetcher looks right. Agreed this is getting complex, but I don't see a clean way to integrate this without a full re-write which I don't have time for right now.
As for the scheduler, my goal was to keep the behavior of the scheduler unchanged as much as possible. ie, a BUILD-DELTA node is "just" a build node, and will be treated exactly the same way as a normal build node. (if a dependency rebuilt lets rebuild, if a sub package is missing lets rebuild). We want to keep the delta RPMs out of ./out/RPMS, but that means our rpm paths are wrong IF we rebuild the node.

Okay. I approve the PR. Please create a backlog or a Github-issue to clean-up the code in future. I would suggest documenting the logic and motivation. Otherwise, it will be harder for any fresh eyes.

}

Referencing our offline discussion: we should probably remove that, since otherwise this is going to be causing some confusion about what is a delta build . Since we're going to allow rebuilds during delta builds, these lines should no longer be here.
Since this PR is already out, and removal of these lines needs to happen after we adjust our building process, I don't mind leaving this clean-up to a separate PR. Let's just not forget about it.
Refers to: toolkit/tools/scheduler/schedulerutils/preparerequest.go:100 in d0d6b7d. [](commit_id = d0d6b7d, deletion_comment = False)

I think this logic is still needed, we just have a bad naming collision here. This "delta" means "allow us to use things we downloaded from PMC to resolve dynamic dependencies instead of waiting to find the real source from a local build".
I'm thinking I'll re-word and rename this flag to better reflect what it actually means.

// Write the graph to file (even if we are going to download delta RPMs, we want to save the graph with the resolved nodes first)

Now when I see both the refactoring and these changes: why do we want to save the graph even if we may need to overwrite it later?

Refers to: toolkit/tools/graphpkgfetcher/graphpkgfetcher.go:115 in 8135dcc. [](commit_id = 8135dcc0b03a68e8153030c4bd8bb79d52f4774d, deletion_comment = False)

toolchainPackages, err := schedulerutils.ReadReservedFilesList(*toolchainManifest)

Nit-pick: the toolchainPackages variable seems to be only needed inside the if hasUnresolvedNodes block, when hasUnresolvedNodes is true. Could we move it into that scope?

Refers to: toolkit/tools/graphpkgfetcher/graphpkgfetcher.go:95 in 3141610. [](commit_id = 3141610eea2cc0e6627766a3408b2f148d4c5129, deletion_comment = False)


Expand support for jax_array in jnp.array.

This relates to the long discussion in #4725 and #10065.

If this is acceptable then I can use the __jax_array__ extension mechanism to duck type shape polymorphic dimensions as JAX arrays. This would be more load put on the experimental __jax_array__ API, from an experimental API (shape polymorphism).

I'd want to get @mattjj's input on this – I'm not clear on the plans for __jax_array__

I'd really like to delete __jax_array__, and replace it with a more systematic mechanism, as briefly outlined in this comment. We may need to have a replacement ready first.
What are the alternatives to this PR, for accomplishing the jax2tf shape polymorphism goals?

The goal for shape polymorphism is to support the following when the shape of x contains polynomials
x / x.shape[0]
jnp.array(x.shape[0])
jnp.array(x.shape)

With the existing __jax_array__ support I can already achieve the first of the above by defining __jax_array__ for dimension polynomials. This kind of makes sense because without shape polymorphism x.shape[0] is a constant so users know they can use it in operations that can take JAX arrays.
This PR is specifically to allow cases 2 and 3 above.
Note that when the real dynamic shapes land in JAX, the polynomials disappear and their __jax_array__ disappears with them, because x.shape[0] will be a tracer that can already be used in operations with JAX arrays.
If I cannot use __jax_array__ then my plan B is to not support case 1 above, and instead document that users have to call jax.array(x.shape[0]) explicitly. This is a bit awkward because they need to do this only if there is shape polymorphism. And then I plan to support cases 2 and 3 above by replacing the change in this PR with:
object = tree_map(lambda leaf: core.dimension_as_value(leaf) if core.is_special_dim(leaf) else leaf, object)

Note that when the real dynamic shapes land in JAX, the polynomials disappear and their jax_array disappears with them, because x.shape[0] will be a tracer that can already be used in operations with JAX arrays.

Maybe we're not too far from this. In fact, a lot might already work today! Can we draw up (or can you point me to) some test cases which define success here?

As a start, the test cases that I'd use are:
JAX_DYNAMIC_SHAPES=1 JAX2TF_DEFAULT_EXPERIMENTAL_NATIVE_LOWERING=1 JAX_ENABLE_X64=0 pytest --durations=10 -r a --verbosity 1 -s jax/experimental/jax2tf/tests/shape_poly_test.py

There are hundreds of tests. Maybe we should move this new thread elsewhere.

Thanks, @gnecula !
You're right that we should stay focused in this thread. FWIW the concrete changes in this PR are super minimal and so don't count as much tech debt on their own. My only concern would be if these small changes somehow generate a behavior that we'll then get locked into supporting (and which might be hard or undesirable to support in the future). For jax2tf, it's already marked as experimental so the risk is small. This change could in theory also increase others' reliance on the __jax_array__ mechanism, though that's a bit of an abstract concern.
Overall, I think we're on the same page and we all have the same concerns/cautions in mind, so I'd say that if this small change helps advance your work let's just do it.

I agree with you that this is a small change and to the extent that users will lean on it, it will be in a way that we plan to support in the future simply because dynamic shapes will have naturally the behavior that we are forcing here.


perf(VehiclesForRouteChannel): Use vehicles from PubSub instead of GenServer

Summary
Ticket: [Name](https://app.asana.com/0/1205732265579288/1208619681848095/f)
What is this PR for?
Addresses the bottleneck found in Load Testing when many users are joining vehicles channels at once.
Vehicles.PubSub is largely a copy of Predictions.PubSub - there is probably a good basis for extracting out common pieces, but I think it is worth saving that for a separate ticket.
This does switch VehicleChannel to use PubSub as well for consistency.
Testing:

Added & updated unit tests
Ran locally on phone pointing to dev-orange deployiment
Confirmed logs for new PubSub module in Splunk as expected

@EmmaSimon thanks for asking - definitely worth running again, and I hadn't yet! Added results to the Load testing doc - this got us to push past 3,300 users at a rate of 5 users / second. I think we'll still want to tackle static data caching to improve that further, or start with a higher base # of instances running.


Update iam-policy to v2.0.0

what

Update terraform-aws-iam-policy to version 2.0.0

why

Previous update to version 1.x inadvertently removed the ability to accept a list of statements in iam_policy_statements. This update restores that ability, and adds additional options for input supported by terraform-aws-iam-policy

references

terraform-aws-iam-policy v2.0.0 release notes

/terratest


stats: simplify the stats of buffer size

Commit Message:
Set current current size stat instead of add/dec.
Signed-off-by: Yuchen Dai<EMAIL_ADDRESS>Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

I could be wrong... close it for now


Set 'mountSources' to 'true' in some example SpringBoot multi-container Devfiles

What type of PR is this:
/kind bug
/area testing
What does this PR do / why we need it:
In those examples, the build and run commands are running in
different container components.
Without mounting sources in the container used for the 'run' command,
this command will not start because Maven would not be able
to resolve the 'spring-boot' plugin:
 ✓  Building your application in container (command: defaultbuild) [39s]
 •  Executing the application (command: defaultrun)  ...
 ✗  Finished executing the application (command: defaultrun) [151ms]
 ⚠  Devfile command "defaultrun" exited with an error status in 20 second(s)
 ⚠  Last 100 lines of log:
 ✗  Waiting for the application to be ready [1m]
 ⚠  Port forwarding might not work correctly: timeout while checking for ports; ports not listening: (8080 in container "runtime"): context deadline exceeded
 ⚠  Running `odo logs --follow` might help in identifying the problem.

 -  Forwarding from <IP_ADDRESS>:20001 -> 8080

This was not caught in the tests because failing to run the
'run' command currently does not prevent the Dev session from starting.
This change might actually make the test faster as we would not have to wait 1 minute
to find out that the application port is not reachable.
 ✓  Building your application in container (command: defaultbuild) [44s]
 •  Executing the application (command: defaultrun)  ...
 ✓  Waiting for the application to be ready [1s]
 -  Forwarding from <IP_ADDRESS>:20002 -> 8080

Which issue(s) this PR fixes:
Related to https://github.com/redhat-developer/odo/issues/6754
PR acceptance criteria:

[ ] Unit test

[ ] Integration test

[ ] Documentation

How to test changes / Special notes to the reviewer:
odo init --name test-java-springboot \
  --devfile-path /path/to/odo/tests/examples/source/devfiles/springboot/devfile-with-missing-projectType-and-language-metadata.yaml \
  --starter springbootproject

odo dev

  << Timeline

  [FAILED] Expected
      <*url.Error | 0xc000870c30>: {
          Op: "Post",
          URL: "http://<IP_ADDRESS>:60116/api/newuser",
          Err: <*errors.errorString | 0xc00008a130>{s: "EOF"},
      }
  to be nil
  In [It] at: C:/Users/Administrator.ANSIBLE-TEST-VS/3776/tests/e2escenarios/e2e_test.go:431 @ 04/24/23 08:13:25.417

  There were additional failures detected.  To view them in detail run ginkgo -vv

Flaky E2E test (#6582)
/override windows-integration-test/Windows-test

[...]

Ran 452 of 824 Specs in 2110.872 seconds
SUCCESS! -- 452 Passed | 0 Failed | 0 Pending | 372 Skipped

Ginkgo ran 1 suite in 36m16.200276075s
Test Suite Passed

[...]

  [odo] I0421 14:09:07.330501   20288 exec.go:49] ExecuteCommand returned an an err: error while streaming command: error sending request: Post "https://c100-e.eu-de.containers.cloud.ibm.com:31988/api/v1/namespaces/e2e-test197ilv/pods/kvvtbv-app-5786ddc7bd-ftx2h/exec?command=%2Fbin%2Fsh&command=-c&command=cat+%2Fopt%2Fodo%2F.odo_cmd_run.pid+%7C%7C+true&container=tools&stderr=true&stdout=true": dial tcp: lookup c100-e.eu-de.containers.cloud.ibm.com: operation was canceled. for command '[/bin/sh -c cat /opt/odo/.odo_cmd_run.pid || true]'
[...]

  [FAILED] Expected
      <[]string | len:1, cap:1>: ["m2-kvvtbv-app"]
  to be empty
  In [It] at: /go/odo_1/tests/e2escenarios/e2e_test.go:264 @ 04/21/23 14:10:09.351
------------------------------

Summarizing 1 Failure:
  [FAIL] E2E Test starting with non-empty Directory [It] should verify developer workflow from non-empty Directory
  /go/odo_1/tests/e2escenarios/e2e_test.go:264

Ran 4 of 4 Specs in 451.506 seconds
FAIL! -- 3 Passed | 1 Failed | 0 Pending | 0 Skipped

Network issue in E2E tests.
Overriding because all integration tests passed and because the Devfiles changed in this PR are not used in E2E tests.
/override OpenShift-Integration-tests/OpenShift-Integration-tests


contract on XDC using hardhat guide

Hello @ArpitaGanatra , for the how-to article we need access to the code in the .sol file so that people don't have to look at your screenshot and try to copy it.  Could you please refactor your how to article to use the simple solidity Pizza example found [here]?(https://github.com/XDC-Community/docs/tree/main/how-to/Pizza)

Hey @WalterBlueu
I have made the necessary changes. please take a look.

Thank you, I will take a look.

@ArpitaGanatra I am going through the how-to and have received an error on deploy:

I believe I am missing setup information for XDC Mainnet.  Here is the contents of my hardhat.config file:
require("@nomicfoundation/hardhat-toolbox");
require("dotenv").config({ path: ".env" });

const XINFIN_NETWORK_URL = process.env.XINFIN_NETWORK_URL;
const XINFIN_PRIVATE_KEY = process.env.XINFIN_PRIVATE_KEY;

module.exports = {
  solidity: "0.8.9",
  networks: {
    xinfin: {
      url: XINFIN_NETWORK_URL,
      accounts: [XINFIN_PRIVATE_KEY],
    },
  },
};

I am using a throwaway XDCPay account that I created for this how-to, here is my .env file contents:
XINFIN_NETWORK_URL=https://erpc.xinfin.network
WALLET_PRIVATE_KEY=myself pulse label credit penalty matrix citizen crowd proud place muffin kingdom

@WalterBlueu
In .env, WALLET_PRIVATE_KEY is suppose to have your private key and not the seed phrase.

@ArpitaGanatra can you please update the how-to with explicit steps on how to setup the .env and config files?  I want to make sure that someone without any past dev experience is able to follow this step-by-step and find success.  Thanks for working with me on this.

@WalterBlueu made the necessary changes.

I will take a look, thank you!

@ArpitaGanatra I am still seeing the same error:

Here is the project directory layout (incase my .env file is at the wrong location):

Here is the content of my .env file:
XINFIN_NETWORK_URL=https://erpc.xinfin.network
WALLET_PRIVATE_KEY=c46461a489a2a3b9330730560a8da9111b0001a7eb84bfcc1cb37a31dcb86054

Here is the content of my hardhat.config file:
require("@nomicfoundation/hardhat-toolbox");
require("dotenv").config({ path: ".env" });

const XINFIN_NETWORK_URL = process.env.XINFIN_NETWORK_URL;
const XINFIN_PRIVATE_KEY = process.env.XINFIN_PRIVATE_KEY;

module.exports = {
  solidity: "0.8.9",
  networks: {
    xinfin: {
      url: XINFIN_NETWORK_URL,
      accounts: [XINFIN_PRIVATE_KEY],
    },
  },
};

Please let me know if you can identify what I am doing wrong, and update the how-to to reflect.  I appreciate your patience.

@WalterBlueu
You are getting the error of being undefined.
Here are the possible reasons.
1)Have you done npm i dotenv as mentioned in the how-to?
2) check if spellings in your hardhat.config.js and .env match.

@ArpitaGanatra I found the typo, in your article you have:

it is supposed to be:

I am getting a new error message now:

Let me know if you know how to address this issue.

@WalterBlueu I will change that mistake. Also when I saw the wallet with the private key in your .env it does not have any balance.
XINFIN_NETWORK_URL=https://erpc.xinfin.network XINFIN_PRIVATE_KEY=202e3c9d30bbeca38d6578659919d4c3dc989ae18c16756690877fdc4dfa607f
could you use this as your env.

@WalterBlueu
I went through my tutorial building the whole project from scratch reading the tutorial.
I suggest you try the same, try to build a fresh project. please use the .env that I have provided to you above. I will try to add a video with the pizza.sol working example.

@ArpitaGanatra went back through it from scratch, using your .env file and the only issue I ran into was a solidity compiler version mismatch, which was easy enough to find and fix.  I was able to successfully deploy the contract!  Thank you for all of your support.  I am going to merge this pull request and promote this article.

@WalterBlueu Happy to help.
Let me know if you need help with anything else. :)


Added GeoIps to loadbalancers

@Gman98ish @rbibby this is approved, does this need merging in?

@Gman98ish @rbibby this is approved, does this need merging in?

The API for it hasn't been released yet


How to  add code highligthing for a  new langage?

Hi,
I wonder how to add support for a new langage.
I would like to give try to presenterm for creating slides for students.
The idea is to present the use of gmsh.
I would like to have basic support for the geo file format of gmsh just like the gmsh-vscode extension .
Thanks for your feedback.
Best

This uses bat under the hood to actually do all the syntax highlighting. If the language is supported by bat then it should be as easy as https://github.com/mfontanini/presenterm/pull/361, otherwise this is in the same boat as https://github.com/mfontanini/presenterm/issues/271

Closing, feel free to reopen if needed. See #388 also for a way to invoke bat directly.


Make trigonometric augmentation user friendly

Matlab's implementation augments the variables that represents angles to two new states that are simply their cos and sin.
This is important for the performance of the algorithm and is possible to be done in the current implementation by simply passing an augmented dataset to the PILCO object.
Investigate how to do this in a more user friendly way.

Hey Nikitas,
I can address this issue if you haven't started working on it.
I was thinking about passing to the PILCO object a set of indices similar to the ones used in the MATLAB PILCO version (angi, dyni, dyno...)

Sure, PRs are welcome!
While coding, keep in mind that the purpose of this package is to provide researchers/experts of the area of gaussian process dynamical systems with a basic & clean implementation for PILCO. So we usually opt for code readability/conciseness instead of user friendliness [that is, when we can't have both :) ].


This appears to be reporting vastly different download speeds than the web-based products

Extension
https://www.raycast.com/tonka3000/speedtest
Raycast Version
1.78.1
macOS Version
14.1
Description
I'm seeing download speeds that are 1/3 to 1/2 of what I'm getting on the speedtest.net website (and verified by other third party speed test software)
Steps To Reproduce

Run speed test
Compare results to web interface

Current Behaviour
No response
Expected Behaviour
No response

Just test it out. For me it works as expected. Could you provide any additional info?
Do you connected to VPN? Is the location of the server in the extension differ from the web site?


Load put and call contracts dynamically from OptionsFactory

Aside from the additional work, is there any reason why aren't puts and calls loaded dynamically from the OptionsFactory?
OptionsFactory can be iterated to get all contracts and all put/call attributes could be derived from those contracts.
By loading them dynamically, opynmonitor.xyz could allow people to transact with all contracts ever created on the protocol and not select few. This would make opynmonitor.xyz the same what uniswapdex.com (shows all tokens and not just whitelisted) is to uniswap.exchange

Ahh This is a great idea, I originally hardcoded all the options because of the request limitation from Infura, it would cost too much to query these constants every time.
Now it seems like we have all the information we need from the opyn subgraph so I guess this would no longer be an issue.
Another potential problem would be having too many options on the list (probably because more and more people are gonna create their own options for testing purpose). I'm thinking about having an optional filter to filter out all the contracts that's not from the opyn tean (contract.owner). Do you have any other suggestions on this?

Hiding all options made by anyone but the team opens removes the ability for more use cases. For example, I would love to write my own puts at specified strike prices and dates. Not for testing purposes but for someone to buy them. Even though the contract allows me to do that, nobody will ever discover them if they are not visible in some interface. And since these put contracts share the exact same code and their only difference is that I'm not on the team of Opyn, it would be against one of the principles of being blockchains and DeFi of being permissionless.
We could hide the empty or small option contracts in a similar way in which https://uniswapdex.com/ hides small exchange contracts. By showing some switch or customizable input to filter out contracts with have less than N amount of collateral on them.

To reduce the Infura calls, we could batch RPC calls by using https://github.com/pooltogether/etherplex or https://github.com/makerdao/multicall.js which both use https://github.com/makerdao/multicall
Or also by caching loaded options locally with IndexedDB along with the highest number (count) of the latest loaded option. Then compare the count of options stored locally vs the count in the factory and load only new options.
Further RPC call reduction could be done by only loading N number of latest options and showing "load more" button.

Yeah I'm not saying I don't want to show other custom options, just want to make sure we have proper filters for users to use so they won't get annoyed.
I think another problem this leads to is where to exchange those newly created options? Currently most people use Uniswap, I think we may need 0x if we want to bootstrap custom options. I have an 0x trading page here:
https://opynmonitor.xyz/#/trade/oeth-usdc
Any idea how to guide users to use it more intuitively?


Please support jupyter file ".ipynb" in vscode. ;P

Python support. But ipynb don't support yet.

I agree, here is my current work around for this issue.

Convert the .ipynb file into a .py file.
Generate docs for everything you need in the .py file.
Copy and paste the docstrings from the .py file back into the .ipynb file.


unable do junit testing of TableResult outside of package

I am using table to result to get the result from query job and then using a FieldValueList to iterate the list of row and is working fine, However when I do the junit testing for this code , I tried the TableResultTest and getting error that TableResult() not available publicly. Is there any way I can populate the FiledValueList in TableResult and do the unit test on different package
TableResult result = queryJob.getQueryResults();
for (FieldValueList row : result.iterateAll()) {
}

Thanks a lot a for fixing this issue

Thanks for fixing this issue


example-jack fails to build dsp

$ cargo build
...
error: failed to run custom build command for `example v0.1.0 (/home/user/rust-faust/examples/example-jack)`

Caused by:
  process didn't exit successfully: `/home/user/rust-faust/target/debug/build/example-bf683518de162587/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=dsp

  --- stderr
  cargo:rerun-if-changed=dsp/volume.dsp
  thread 'main' panicked at 'Failed to execute command: Os { code: 2, kind: NotFound, message: "No such file or directory" }', faust-build/src/lib.rs:31:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

What faust version do you have installed? Just tried to run it and it works fine locally.

Also the no such file or directory error sounds a bit like faust is not installed at all?

Yes, that was the problem. I thought the error said that the compiler could not find the path to the DSP file.


Flac transcoding leads to N/A bitrate and duration

Version information

fluent-ffmpeg version: 2.1.0
ffmpeg version: 3.2.4
OS: ArchLinux

Code to reproduce
const ffmpeg = require('fluent-ffmpeg');
const fs = require('fs');

ffmpeg(input)
  .format('flac')
  .on('start', function(commandLine) {
    console.log('Spawned Ffmpeg with command: ' + commandLine);
  })
  .on('codecData', function(data) {
    console.log('Input is ' + data.audio + ' audio ' +
      'with ' + data.video + ' video');
  })
  .on('progress', function(progress) {
    console.log('Processing: ' + progress.percent + '% done');
  })
  .on('end', function(stdout, stderr) {
    console.log('Transcoding succeeded !');
  })
  .on('stderr', function(stderrLine) {
    console.log(stderrLine);
  })
  .on('error', function (err) {
    console.log(err);
  })
  .output(fs.createWriteStream('./output.flac')).run();

Expected results
$ ffprobe fluent_output.flac
ffprobe version 3.2.4 Copyright (c) 2007-2017 the FFmpeg dev
  built with gcc 6.3.1 (GCC) 20170109
  configuration: --prefix=/usr --disable-debug --disable-sta
ss --enable-libbluray --enable-libfreetype --enable-libfribi
g --enable-libopus --enable-libpulse --enable-libschroedinge
libwebp --enable-libx264 --enable-libx265 --enable-libxvid -
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, flac, from 'fluent_output.flac':
  Metadata:
    GENRE           : Trailer Production Music
    RATING          : 100
    TITLE           : White Witch (Choir)
    ALBUM           : Legend
    ARTIST          : Two Steps From Hell
    DATE            : 2008
    track           : 37
    album_artist    : Two Steps From Hell
    ENCODER         : Lavf57.56.101
  Duration: 00:00:10.00, start: 0.000000, bitrate: 1488 kb/s
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s32 (24 bit)

Observed results
$ ffprobe fluent_output.flac
ffprobe version 3.2.4 Copyright (c) 2007-2017 the FFmpeg dev
  built with gcc 6.3.1 (GCC) 20170109
  configuration: --prefix=/usr --disable-debug --disable-sta
ss --enable-libbluray --enable-libfreetype --enable-libfribi
g --enable-libopus --enable-libpulse --enable-libschroedinge
libwebp --enable-libx264 --enable-libx265 --enable-libxvid -
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, flac, from 'fluent_output.flac'':
  Metadata:
    GENRE           : Trailer Production Music
    RATING          : 100
    TITLE           : White Witch (Choir)
    ALBUM           : Legend
    ARTIST          : Two Steps From Hell
    DATE            : 2008
    track           : 37
    album_artist    : Two Steps From Hell
    ENCODER         : Lavf57.56.101
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s32 (24 bit)

Logs
Spawned Ffmpeg with command: ffmpeg -ss 10 -i input.flac -acodec flac -ac 44100 -ar 44100 -aq 0 -vn -t 10 -f flac pipe:1
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.1 (GCC) 20170109
  configuration: [...]
Input #0, flac, from 'input.flac':
  Metadata:
    GENRE           : Trailer Production Music
    RATING          : 100
    TITLE           : White Witch (Choir)
    ALBUM           : Legend
    ARTIST          : Two Steps From Hell
    DATE            : 2008
    track           : 37
    album_artist    : Two Steps From Hell
  Duration: 00:02:41.17, start: 0.000000, bitrate: 1757 kb/s
    Stream #0:0: Audio: flac, 48000 Hz, stereo, s32 (24 bit)
Output #0, flac, to 'pipe:1':
  Metadata:
    GENRE           : Trailer Production Music
    RATING          : 100
    TITLE           : White Witch (Choir)
    ALBUM           : Legend
    ARTIST          : Two Steps From Hell
    DATE            : 2008
    TRACKNUMBER     : 37
    ALBUMARTIST     : Two Steps From Hell
    encoder         : Lavf57.56.101
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s32 (24 bit), 128 kb/s
    Metadata:
      encoder         : Lavc57.64.101 flac
Stream mapping:
Input is flac audio with  video
  Stream #0:0 -> #0:0 (flac (native) -> flac (native))
Press [q] to stop, [?] for help
[flac @ 0x555619be7aa0] unable to rewrite FLAC header.
size=    1817kB time=00:00:10.00 bitrate=1488.7kbits/s speed=71.9x
Processing: undefined% done
video:0kB audio:1809kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.457627%

Transcoding succeeded !

And I tried the command in my terminal and the audio had a valid duration
$ ffmpeg -ss 10 -i 'input.flac' -acodec flac -ac 44100 -ar 44100 -aq 0 -vn -t 10 -f flac file_from_terminal.flac
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.1 (GCC) 20170109
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, flac, from 'input.flac':
  Metadata:
    GENRE           : [...]
  Duration: 00:02:41.17, start: 0.000000, bitrate: 1757 kb/s
    Stream #0:0: Audio: flac, 48000 Hz, stereo, s32 (24 bit)
Output #0, flac, to 'testfile.flac':
  Metadata:
    GENRE           : [...]
    encoder         : Lavf57.56.101
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s32 (24 bit), 128 kb/s
    Metadata:
      encoder         : Lavc57.64.101 flac
Stream mapping:
  Stream #0:0 -> #0:0 (flac (native) -> flac (native))
Press [q] to stop, [?] for help
size=    1817kB time=00:00:10.00 bitrate=1488.7kbits/s speed=68.6x
video:0kB audio:1809kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.457627%

$ ffprobe file_from_terminal.flac
ffprobe version 3.2.4 Copyright (c) 2007-2017 the FFmpeg developers
  built with gcc 6.3.1 (GCC) 20170109
  configuration: [...]
Input #0, flac, from 'file_from_terminal.flac':
  Metadata:
    GENRE           : Trailer Production Music
    RATING          : 100
    TITLE           : White Witch (Choir)
    ALBUM           : Legend
    ARTIST          : Two Steps From Hell
    DATE            : 2008
    track           : 37
    album_artist    : Two Steps From Hell
    ENCODER         : Lavf57.56.101
  Duration: 00:00:10.00, start: 0.000000, bitrate: 1488 kb/s
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s32 (24 bit)

Please note this looks more specific to flac than this particular file. The same code ran with mp3 format (and the corresponding codec ofc) works and I am able to use all the chunks in the browser. I tried quickly with different source files (some mp3) and same bug...

Edit: since this is a stream, flac is unable to rewrite header as opposed to a file. Is there a way to change this behavior?

This is indeed probably due to using a stream. There may be some options in the flac codec to put metadata at the end, but that's out of the scope of this lib, you will have to look at ffmpeg docs for that.
But why would you do .output(fs.createWriteStream("...")) in the first place ? Just because you can use streams doesn't mean you should :)

Didn't find any in the docs, i'll try to take a look at the source code.
This was to get a minimum example working, in my application the output is a node Response object (from http)

Do you found a solution ??
I'm facing same issue

@vinz243
Thank you for quick reply.
I have been at this issue 5 days, but today I could resolve it by downloading the file before converting it, in this way could copy all metadata to the audio file

@ghyath5
did you find a solution to do the convertion with pipes (stream) but the output file to has the duration and bitrates?

So how does one extract the bitrate of a FLAC file with ffmpeg?

Feels like this makes sense to be in the ffmpeg docs? Spent hours trying to figure this out until I stumbled across this ticket