Feature: Rust polars import / export

SDK
Rust
Description
In the Rust SDK we should be able to go to/from polars more easily.  For example, something like the following should work when adding data:
let my_df: DataFrame = ...
tbl.add(data_source::from_polars(my_df)).execute();

The above should also work with LazyFrame, although, for today at least, this will probably involve the consumption / execution of the LazyFrame:
let my_df: LazyFrame = ...
tbl.add(data_source::from_polars_lazy(my_df).execute();

Also note that DataSource doesn't exist yet.  In my brainstorming its just a module with a bunch of static methods (enabled by feature flags) for creating Box<dyn RecordBatchReader + Send> from various inputs.  As a result, this technique should also be usable in other methods that need data (e.g. create_table, merge_insert)

I can take this

okay i'm scoping some of this out so we get a nice conversion. forgive my slowness, still learning Rust

Wait we should close this issue. If we need a deeper integration with Polars, I'm willing to start on that instead.
The solution here would just be:
Type conversions to LanceDB
If we made the deeper integration:
Conversions should be native to Lance

Wait we should modify this issue or work on the Lance repo instead.

Now that add is based on a trait we don't even need the data_source::from_polars part.  We should be able to do something like:
let my_df: DataFrame = ...
tbl.add(my_df).execute();

This will require adding something like...
#[cfg(feature = polars)]
impl IntoArrow for polars::DataFrame {
  ...
}

The solution here would just be:
Type conversions to LanceDB

I think we just need polars <-> arrow which should already be a part of polars.

If we need a deeper integration with Polars, I'm willing to start on that instead.
If we made the deeper integration:
Conversions should be native to Lance

What do you imagine this would look like?  What is your goal with a deeper integration?  Can you give some example use cases?

Agh, have been pulled away with other projects. Sorry, will have to hand this off to someone else

No sweat, I'm a bit tied down with encodings at the moment but will try and snag this next week.

Can I take a stab at this @westonpace?

Can I take a stab at this @westonpace?

Yes.  Please feel free.  I'm still working on encodings.

Some quick findings:
It looks like Polars uses a different arrow crate (polars-arrows) than the official one.
Polars exposes iter_chunks, which seems to be the way to access an iterator of RecordBatches.
I see that the polars-arrow crate has a way of converting to and from the different array implementations, which may make it possible to adhere to the IntoArrow interface, but this will require adding a dependency on it.

Hmm, thanks for the investigation so far.  This certainly complicates things :frowning_face:
However, we should be able to achieve zero-copy with the FFI / C data interface.
https://docs.rs/polars-arrow/0.38.3/polars_arrow/ffi/fn.export_iterator.html will create an ArrowArrayStream (https://docs.rs/polars-arrow/0.38.3/polars_arrow/ffi/struct.ArrowArrayStream.html).
We should be able to reinterpret that as a arrow-rs FFI_ArrowArrayStream (https://docs.rs/arrow/latest/arrow/ffi_stream/struct.FFI_ArrowArrayStream.html).
That being said, I haven't had to do this for rust<->rust yet, so there always could be surprises :)
Here is where we do this to convert pyarrow into arrow-rs: https://github.com/apache/arrow-rs/blob/master/arrow/src/pyarrow.rs
Something similar is probably done in polars to convert to python / pyarrow.

I see that the polars-arrow crate has a way of converting to and from the different array implementations, which may make it possible to adhere to the IntoArrow interface, but this will require adding a dependency on it.

Adding a dependency on polars is inevitable I think.  We should do this with a feature (e.g. lancedb should have a non-default feature polars which adds the polars dependency).  Or are you saying we may need a dependency on polars-arrow?  That would be fine too as long as we guard it with the same polars feature.
So I'm not too worried about the dependency here as much as I am about the fact that going polars-arrow -> Vec -> arrow-rs is probably going to introduce a copy or a bunch of boilerplate or tedious and error-prone code (the C data conversion is more of a "hard to get it right but you only have to get it right once" thing).

ok, will give it a go.

Thinking out loud. Still learning Rust/Arrow so not 100% sure...
I don't actually think we need to use the FFI/C data interface directly to get zero copy. It seems like the polars-arrow Arrow2Arrow trait implementations are zero copy - they clone the underlying buffers when converting to Polar's Arrow representation to Arrow.rs's representation, but the buffers either contain raw pointers or Arc which are both zero copy.
I wrote a simple script to test the convertor:
use arrow_array::RecordBatch;
use polars::prelude::*;

type ArrowSchema = arrow_schema::Schema;

fn main() {
    // Load the Parquet file into a LazyFrame.
    let df = LazyFrame::scan_parquet("test.parquet", ScanArgsParquet::default())
        .expect("Failed to read Parquet file")
        .collect()
        .expect("Failed to collect LazyFrame");

    // Process each chunk in the LazyFrame and convert it into RecordBatch.
    let batches: Vec<RecordBatch> = df
        .iter_chunks(false)
        .zip(df.schema().iter_names())
        .map(|(chunk, name)| {
            // Collect the arrays in the chunk into Arrow arrays.
            let columns: Vec<arrow_array::ArrayRef> = chunk
                .arrays()
                .into_iter()
                .map(|polars_array| arrow_array::ArrayRef::from(&**polars_array))
                .collect();

            // Create Arrow schema fields for each column.
            // TODO: figure out how to handle nullability and other field metadata
            let fields: Vec<arrow_schema::Field> = columns
                .iter()
                .map(|array| {
                    arrow_schema::Field::new(name.to_string(), array.data_type().clone(), true)
                })
                .collect();

            // Create an Arrow schema from the fields.
            let arrow_schema = Arc::new(ArrowSchema::new(fields));

            // Create a RecordBatch from the Arrow schema and columns.
            RecordBatch::try_new(arrow_schema, columns).expect("Failed to create RecordBatch")
        })
        .collect();

    dbg!(batches);
}

The main issue I ran into is that the polars-arrow crate does not compile with the "arrow-rs" feature enabled when using the latest version (0.38.3). They've addressed this on the main branch and if I point the dependency to their GitHub repo, everything works as expected.


New name for natural mortality with updated SS version

Just copying this over from (nwfsc-assess/nwfscDiag) as it likely pertains to our workflow once we update:
SS version <IP_ADDRESS> changed the parameter label from "NatM_p_1_Fem_GP_1" to "NatM_uniform_Fem_GP_1"

@cgrandin I searched the repo for the old name and found it in the following two locations. Should we change those and then close this issue?
https://github.com/pacific-hake/hake-assessment/blob/9a9b9c64ad95bb036f85fead221145063aff6c69/R/tables-parameters.R#L351
https://github.com/pacific-hake/hake-assessment/blob/be6a0c53928dfc999fd42ed3d835d035eca964b9/beamer/SRG/Requests/requests-day1/beamer-hake-requests-day1.rnw#L210

Looks like Andy changed that first one so that it still works with last year's model, if the new name is found, use it otherwise use the old name. If that's not there then Table 25 will have last year's M missing.
 ifelse("NatM_uniform_Fem_GP_1" %in% names(.x$mcmc),
   f(median(.x$mcmc$`NatM_uniform_Fem_GP_1`), digits),
   f(median(.x$mcmc$`NatM_p_1_Fem_GP_1`), digits)),

I changed the second one. Thanks for checking!


Add custom menu item

Anyway to customise the PDFView by adding Highlight features in the UIMenuItem during text selected?

text cannot be selected because there is none, the image of the rendered page is displayed as the result


Fix versioning of packaged Rust components

The behavior of cargo-deb was changed to generate version numbers with
tildes instead of dashes. This is causing newly built packages to sort
below older releases.
https://github.com/mmstick/cargo-deb/pull/102
Signed-off-by: Richard Berg<EMAIL_ADDRESS>

This sed was overly broad, so I restricted it to the first match.


[BD-6] Add python 3.8 tests

This PR:

Add tests in python3.8
Separate doc tests in their own travis worker
Upgrade dependencies
Quality fixes for new pylint version

The failure of tests is a known issue: See https://openedx.atlassian.net/browse/ARCHBOM-1061
I separated the problematic tests in their own worker (doc tests)
Reviewers

[ ] @ericfab179
[ ] @awais786
[ ] Ready for edX review

Thanks for the pull request, @morenol! I've created OSPR-4461 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.
Feel free to add as much of the following information to the ticket:

supporting documentation
edx-code email threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.

build for doc is failing under both python versions, maybe that needs to be fixed and commits should be squashed.

build for doc is failing under both python versions, maybe that needs to be fixed and commits should be squashed.

Yes, I am going to squash them. The explanation of the failure is described in the link that I shared in the description of the PR. It is not clear to my which of these options I should take to solve the problem. And also I think that I can fix that in other PR.

@mraarif I squashed and then added a commit that solves the test, but that may fail in the future if github decides to change their search page again

It should be fine for now I think if the tests fail and get reported I believe the team owning this will take care of this.

Perfect, @awais786 please review and merge if you think that this is ready.

@morenol 🎉 Your pull request was merged!
Please take a moment to answer a two question survey so we can improve your experience in the future.


When is it planned to move the docker build container from ubuntu 20.04 to 22.04?

good question. i think out-of-the-box this would change which glibc versions we support
if that is correct then we need to think in terms of supported distros before making a decision
cc @lizan

@sambercovici was there a specific reason for asking ?

I am looking on compiling for arm64 and especially looking for gcc support for SIMD like SVE and SVE2
Ubuntu 20.04 gcc does not support it. Ubuntu 22.04 does.


Support for extracting CSS

I'd like to ask are you going to add support for extracting CSS?

Hey, in short: yes 😸
The long answer: yes, but it's something complicated 🤔 I will provide more technical details once will have time, but it's more or less the same problem as in https://github.com/atlassian-labs/compiled/issues/722.

Thanks for the answer!
I’d glad when it happened. 👍🏻

Hi @layershifter,
I guess 300Kb file isn’t possible in case with the atomic CSS, that’s why I think it’d be perfect to add optional extracting of CSS. It would give the opportunity to process styles with post processors and browse caching, so the styles loading only at first visit the page. If you have another way to work with post processors I would like to know.

Thanks a lot!
I saw your pull request and I’m testing it now in my project. I’ll keep you updated.

Thanks a lot, I really appreciate your help.
I successfully connected the package with the feature.
Extracting CSS from components works fine, but extracting styles from the page doesn’t work.
Print out the argument sourceCode of the function shouldTransformSourceCode in the file griffelWebpackExtractionLoader.ts:74:
import { makeStyles } from '@griffel/react';

const useClassNames = makeStyles({
  page: {
    color: '#768494',
  },
});

export function ContactUsPage() {
  const classNames = useClassNames();

  return
ContactUsPage
;
}

export default ContactUsPage;

Used packages:
"@nrwl/next": "13.8.6",
"next": "^12.1.0",
"react": "^17.0.2”
If you need more details I’d glad to help.

@Dwlad90 this might be a problem with config (that is totally unclear without docs 🙄 ), I made a configured repo with latest Next.js: https://github.com/layershifter/griffel-extraction-test

Thanks a lot, great job!
The bug reason was in the order of webpack loaders. I set order like in the example and now everything works perfect!

Implemented for Webpack, check @griffel/webpack-extraction-plugin.

Implemented for Webpack, check @griffel/webpack-extraction-plugin.

Thanks a lot for a package, its a great job!
I was trying to use this package with nextJS using the mini-css-extract-plugin built-in but in its construction I get the following error:
Global CSS cannot be imported from files other than your Custom <App>. Due to the Global natureof stylesheets, and to avoid conflicts, Please move all first-party global CSS imports to pages/_app.js. Or convert the import to Component-Level CSS (CSS Modules).
Sandbox:
I have to add the plugin with certain parameters to the pipeline so it’ll work properly:
new MiniCssExtractPlugin({     // without these Next.js will look for the generated stylesheets from the wrong place     filename: "static/chunks/[chunkhash].css",     chunkFilename: "static/chunks/[chunkhash].css",     ignoreOrder: true,   })
Sandbox:
Maybe you have an idea how can I use a built-in plugin?

@Dwlad90 TBH I haven't tested it with Next.js, but I think that we can handle it. Would you mind to create a separate issue for this problem?

@layershifter - i am experimenting with css extraction. First attempt shows that it works for makeStyles, but the makeStaticStyles seems to not be extracted, atleast javascript have to run before those styles are applied. Trying to figure out why

@pksorensen indeed, makeStaticStyles() is not supported for AOT/CSS extraction. We don't use it in products, so it was a lot priority for us. Can you please an issue?


bcachefs: can't mount multi-device FS by UUID

Describe the bug
Steps To Reproduce
Steps to reproduce the behavior:

add "bcachefs" to supportedFilesystems and reboot

bcachefs format --background_compression=zstd --errors=ro --replicas=2 --foreground_target=ssd --promote_target=ssd --discard --label=ssd.ssd1 /dev/sda --discard --label=ssd.ssd2 /dev/sdb --discard --label=ssd.ssd3 /dev/sdc

mkdir -p /mnt
mount -t bcachefs "UUID=$(bcachefs show-super /dev/sda | grep Ext | awk '{print$3}')" /mnt

mount doesn't work, dmesg: bch2_fs_open() bch_fs_open err opening /dev/sdb: insufficient_devices_to_start

Expected behavior
Mount works
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
My nixpkgs branch (based on early 23.11): https://github.com/chayleaf/nixpkgs/tree/dbe92cbf44795892d585afc540a9324d773553c8
$ bcachefs version
bcachefs tool version 1.3.3
$ uname -a
Linux server 6.7.0-rc2 #1-NixOS SMP Tue Jan  1 00:00:00 UTC 1980 aarch64 GNU/Linux

Notify maintainers
@YellowOnion

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Priorities
Add a :+1: reaction to issues you find important.

insufficient_devices_to_start

This looks like an upstream issue, do only some of your devices show up in blkid or lsblk with that UUID?
maybe udev needs a trigger for non-partition disks.
try  udevadm settle before calling mount, or just use "/dev/sda:/dev/sdb"

This looks like an upstream issue, do only some of your devices show up in blkid or lsblk with that UUID?

# blkid
...
/dev/sda: UUID="29ea4586-1f72-4227-8663-60e40f0e87c9" BLOCK_SIZE="512" UUID_SUB="60c90ab6-90b3-4c14-b3bf-4c41ee9907ba" TYPE="bcachefs"
/dev/sdb: UUID="29ea4586-1f72-4227-8663-60e40f0e87c9" BLOCK_SIZE="512" UUID_SUB="cefe3e33-6cf5-428b-b393-045390eb9fc1" TYPE="bcachefs"
/dev/sdc: UUID="29ea4586-1f72-4227-8663-60e40f0e87c9" BLOCK_SIZE="512" UUID_SUB="f2b325a6-e474-47c7-a3b4-2cddd4146251" TYPE="bcachefs"

try udevadm settle before calling mount, or just use "/dev/sda:/dev/sdb"

udevadm settle doesn't work. "/dev/sda:/dev/sdb" does, but it wouldn't work in initrd, because the initrd currently only uses the first device (/dev/sda in this case)

actually there seem to be no initrd problems as it correctly only uses the first devices for unlocking, so I'll close this

this doesn't work with systemd, because systemd says for Timed out waiting for device /dev/sda:/dev/sdb:/dev/sdc

@chayleaf bcachefs mount is really dumb it just creates the multi device string internally by looking up the UUIDs, this is specifically something with your system so it's hard for me to debug, guys here want to drop the initrd scripts and use systemd so I don't know how much energy I want to spend on an ad-hoc assemble script, and with the way you're talking it looks like it might be something else other than a race condition soon after boot too. Can you report it upstream? If you're any good with gdb maybe hook it up and see what's happening. If you're any good with rust remove the gag code and splatter around prints to see what's happening.

I've tested it with systemd - it can neither mount UUID=... (probably because it uses bcachefs-tools which doesn't detect all 3 devices) nor /dev/...:/dev/... (for obvious reasons of there not being such a device). If it's a bcachefs-tools issue, I'll dig into its code.

@chayleaf yeah systemd doesn't support the multidevice string stuff, been an open issue for a few years, I boot multidevice, non-encrypted using UUID= syntax and haven't seen any issues yet, so it's more of an issue with encryption, unlock probably needs UUID support, I would recommend using bcachefs on-top of LUKS for now, bcachefs hasn't been audited, and there's a few holes like the superblock is plain and has a simple crc not a hmac applied, and nocow doesn't encrypt because it's fundimentally at odds with how bcachefs does encryption. make sure you've cherry picked all of the bcachefs patches for nixpkgs, from either master or not. bcachefs-tools needs heaps of work, I already pushed a patch to the kernel adding a replacement for that god-awful gag hack that doesn't work.

I'm not using any encryption at the moment (see the command in the initial post) and it still doesn't work. Thanks for the LUKS tip though.
I'll report back after checking bcachefs-tools

so the actual issue seems to be that mount.bcachefs UUID=088a3d70-b54c-4437-8e01-feda6bfb7236 /mnt works, bcachefs mount UUID=088a3d70-b54c-4437-8e01-feda6bfb7236 /mnt works, but mount -t bcachefs UUID=088a3d70-b54c-4437-8e01-feda6bfb7236 /mnt doesn't. I guess this is an util-linux issue then

@chayleaf Sorry late reply, Could you report this to util-linux guys? they might need a dump of your superblock to figure out why it can't read it.


[windows] Support capture the entire screen

Target file "lib\main.dart" not found. i wonder why

Target file "lib\main.dart" not found. i wonder why

Please change directory to example and execute flutter run .

Supported, please update to v0.1.2


⚠ Bofrost.de has degraded performance

In a78df40, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1347 ms

Resolved: Bofrost.de performance has improved in a2f64b3 after 9 minutes.


spring boot 3.2 not working

Describe the bug
A clear and concise description of what the bug is and stack trace.
Expected behavior
spring boot 3.2 not working in 3.0.6
Enviroment (please complete the following information):

OS name : [e.g. MacOS]
Docker Version: [e.g. 20.10.24]

Module (please complete the following information):

Module Name [e.g. embedded-aerospike]
Module Version [e.g. 3.0.0]
Docker Image Name (if use custom):  [e.g. aerospike:ce-<IP_ADDRESS>]

Additional context
Parameter 0 of method kafkaConfigurer in com.playtika.testcontainer.kafka.configuration.KafkaContainerConfiguration required a single bean, but 5 were found:
- embeddedMySQL: defined by method 'mysql' in class path resource [com/playtika/testcontainer/mysql/EmbeddedMySQLBootstrapConfiguration.class]
- embeddedRedis: defined by method 'redis' in class path resource [com/playtika/testcontainer/redis/EmbeddedRedisBootstrapConfiguration.class]
- minio: defined by method 'minio' in class path resource [com/playtika/testcontainer/minio/EmbeddedMinioBootstrapConfiguration.class]
- kafka: defined by method 'kafka' in class path resource [com/playtika/testcontainer/kafka/configuration/KafkaContainerConfiguration.class]
- embeddedMockServer: defined by method 'mockServerContainer' in class path resource [com/playtika/testcontainer/mockserver/EmbeddedMockServerBootstrapConfiguration.class]

@liuruibin Spring Boot 3.2.0 upgraded to Testcontainers 1.19.3, in my tests testcontainers-spring-boot 3.0.6 wasn't compatible since 1.19.2.
Please try again by overriding the dependency management with
<testcontainers.version>1.19.1</testcontainers.version>
if you're using Maven (or gradle -Ptestcontainers.version=1.19.1).
References:
testcontainers 1.18.3: https://github.com/spring-projects/spring-boot/blob/v3.1.6/spring-boot-project/spring-boot-dependencies/build.gradle#L1519
testcontainers 1.19.3: https://github.com/spring-projects/spring-boot/blob/v3.2.0/spring-boot-project/spring-boot-dependencies/build.gradle#L1642

We are waiting for the release of the Spring Cloud 2023.0 (code name: Leyton) to begin adapting to the new version.

Makes sense, but their milestone looks... empty and finished at the same time 😅 https://github.com/spring-cloud/spring-cloud-release/milestone/132
They just switched to Spring Boot 3.2.0, if you have time to spend maybe you could build 2023.0.0-SNAPSHOT yourself.
Can your community help, is there a TODO list for the upgrade process? I can't seem to find such a commit for 3.1, those gazillion dependabot commits surely don't make the history easier to grasp 😄

Resolves #1632
We have released version 3.1.0 with String Boot 3.2 support. Let's check it together :)

@Fameing Thank you for supporting Spring Boot 3.2 👍
I can confirm that testcontainers >= 1.19.2 still doesn't work with embedded-postgresql 3.1.0 (the same as 3.0.6), 1.19.1 works with both versions.

I can ask you to provide an example of a project where it does not work for you.

I can ask you to provide an example of a project where it does not work for you.

I can confirm that with the below setup embedded-postgresql is broken. :
<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.1</version>
    </parent>

<dependencies>
    <dependency>
        <groupId>com.playtika.testcontainers</groupId>
        <artifactId>embedded-postgresql</artifactId>
        <version>3.1.2</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-bootstrap</artifactId>
        <version>4.1.0</version>
    </dependency>
</dependencies>

It broke when we went from Spring Boot 3.1.5 -> 3.2.0. The issue is that the application fails to detect when the database is up and running, it's related to startupLogCheckRegex and the log message database system is ready to accept connections.
If I apply the workaround from @reneleonhardt it starts working again:
<properties>
    <testcontainers.version>1.19.1</testcontainers.version>
</properties>

I tried to reproduce this case and I was not successful. Everything works fine for me.
demo-testcontainer.zip

I apologize for the late answer @Fameing, and also for being a bit unclear. I does work on the first startup, but not the second if we have persisted the data with a bootstrap config like this:
embedded:
  postgresql:
    mountVolumes:
      -
        hostPath: '/dev/docker-data/myservice'
        containerPath: '/var/lib/postgresql/data'
        mode: READ_WRITE

Here's your example modified with this addition that breaks the application on the second startup:
demo-testcontainer-broken.zip
I'm guessing that there is some difference in how the startup message is logged when the database bootup from existing data, to my naked eye it seems to be exactly the same but the code times out waiting for it.
This is the error message from the second startup:
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*database system is ready to accept connections.*\s'
	at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47) ~[testcontainers-1.19.3.jar:1.19.3]
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52) ~[testcontainers-1.19.3.jar:1.19.3]

And this is the complete database log from the container:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2024-01-25 08:33:29.288 UTC [31] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2024-01-25 08:33:29.563 UTC [37] LOG:  starting PostgreSQL 16.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
2024-01-25 08:33:29.563 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-25 08:33:29.564 UTC [40] LOG:  database system was shut down at 2024-01-25 08:33:29 UTC
2024-01-25 08:33:29.566 UTC [37] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2024-01-25 08:33:29.682 UTC [37] LOG:  received fast shutdown request
2024-01-25 08:33:29.682 UTC [37] LOG:  aborting any active transactions
2024-01-25 08:33:29.685 UTC [37] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2024-01-25 08:33:29.686 UTC [38] LOG:  shutting down
2024-01-25 08:33:29.686 UTC [38] LOG:  checkpoint starting: shutdown immediate
2024-01-25 08:33:29.690 UTC [38] LOG:  checkpoint complete: wrote 925 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.001 s, total=0.004 s; sync files=0, longest=0.000 s, average=0.000 s; distance=4269 kB, estimate=4269 kB; lsn=0/191BA30, redo lsn=0/191BA30
2024-01-25 08:33:29.694 UTC [37] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2024-01-25 08:33:29.795 UTC [1] LOG:  starting PostgreSQL 16.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
2024-01-25 08:33:29.795 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
2024-01-25 08:33:29.795 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-01-25 08:33:29.795 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-25 08:33:29.796 UTC [53] LOG:  database system was shut down at 2024-01-25 08:33:29 UTC
2024-01-25 08:33:29.799 UTC [1] LOG:  database system is ready to accept connections

@reneleonhardt @JohanNordlinder I found the problem in wait strategy. In next release is fixed this situation and not needs to downgrade on testcontainers.version=1.19.1.
Waiting version 3.1.4

@JohanNordlinder @reneleonhardt try version 3.1.4.

Works like a charm @Fameing, thanks and well done!

Springboot 3.2.4 not working.
PFB LOGS:
2024-05-17 19:04:54.632] - 25596 INFO [main] --- org.apache.catalina.core.StandardService: Starting service [Tomcat]
[2024-05-17 19:04:54.638] - 25596 INFO [main] --- org.apache.catalina.core.StandardEngine: Starting Servlet engine: [Apache Tomcat/10.1.19]
[2024-05-17 19:04:54.979] - 25596 INFO [main] --- org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]: Initializing Spring embedded WebApplicationContext
[2024-05-17 19:04:56.678] - 25596 INFO [main] --- org.apache.catalina.core.StandardService: Stopping service [Tomcat]
Any help would be highly appreciated


[BUG] $$typeToString ellipsizes type information

Describe the bug
For complex types, $$typeToString replaces parts of the type description with ...
Code to reproduce
These are the same examples as in #73. I am using the following macro to reflect the types of arrow functions that I am exporting:
export function $implement<Impl>(name: string, expr: Impl) {
   $$define!(name, expr, false, true); // Final `true` is export
   $$ident!(name).reflectedType = $$typeToString!<Impl>();
}

It works fine on
$implement!('square',
   <T>(dep: Dependencies<'multiply', T>) => (z:T) => dep.multiply(z, z))

where square.reflectedType ends up getting the very reasonable value <T>(dep: { multiply: (a: T, b: T) => T; }) => (z: T) => T. However, on
$implement!('sqrt',
   <T>(dep: Dependencies<'equal' | 'conservativeSqrt' | 'unaryMinus', RealType<T>>
       & Dependencies<'zero' | 'complex', T>
       & Dependencies<'absquare' | 're' | 'divideReal', Complex<T>>
       & {
         addTR: Signature<'addReal', T>,
         addRR: Signature<'add', RealType<T>>,
         addCR: Signature<'addReal', Complex<T>>
      }):
      Signature<'sqrt', Complex<T>> =>
   z => {
      const myabs = dep.conservativeSqrt(dep.absquare(z))
      const r = dep.re(z)
      const negr = dep.unaryMinus(r)
      if (dep.equal(myabs, negr)) {
         // pure imaginary square root; z.im already zero
         return dep.complex(
            dep.zero(z.re), dep.addTR(z.im, dep.conservativeSqrt(negr)))
      }
      const num = dep.addCR(z, myabs)
      const denomsq = dep.addRR(dep.addRR(myabs, myabs), dep.addRR(r, r))
      const denom = dep.conservativeSqrt(denomsq)
      return dep.divideReal(num, denom)
   })

it produces sqrt.reflectedType equal to
<T>(dep: { unaryMinus: (a: RealType<T>) => RealType<T>; conservativeSqrt: (a: RealType<T>) => RealType<T>; equal: (a: RealType<T>, b: RealType<...>) => boolean; } & { ...; } & { ...; } & { ...; }) => (a: Complex<...>) => Complex<...>
so we have lost much of the type information to those ellipses.
Expected behavior
I would like there to be a way to force $$typeToString to produce the entire description of the typescript type of its template argument, without any ellipsization.
Additional context
Typescript 5.1.6, ts-macros 2.4.1, ts-patch 3.0.2

Can you send me the types used in the snippet that causes the ellipsization? I can't replicate it.

OK, I have tried to collect them from the various imports of that source file:
type Deps<T> =  T extends unknown ? { [K in keyof T]: T[K] } : never;
type Dependencies<Name extends SignatureKey<T>, T> = Deps<{[K in Name]: Signature<K, T>}>
type Signature<Name extends SignatureKey<T>, T> = Signatures<T>[Name]
type SignatureKey<T> = keyof Signatures<T>
type AliasOf<Name extends string, T> = T & {aliasOf?: Name}
interface Signatures<T> {
   zero: (a: T) => ZeroType<T>
   one:  (a: T) => OneType<T>
   nan:  (a: T | NaNType<T>) => NaNType<T>
   re:   (a: T) => RealType<T>
   equal:   (a: T, b: T) => boolean
   add:              (a: T, b: T) => T
   unaryMinus:       (a: T) => T
   conj:             (a: T) => T
   subtract:         (a: T, b: T) => T
   multiply:         (a: T, b: T) => T
   square:           (a: T) => T
   absquare:         (a: T) => RealType<T>
   reciprocal:       (a: T) => T
   divide:           (a: T, b: T) => T
   conservativeSqrt: (a: T) => T
   sqrt: (a: T)=> T extends Complex<unknown> ? T : T | Complex<T>
   complex: ((re: T) => Complex<T>) | ((re: T, im: T) => Complex<T>)
   addReal: AliasOf<'add', (a: T, b: RealType<T>) => T>
   divideReal: AliasOf<'divide', (a: T, b: RealType<T>) => T>
}
type ValueIntersectionByKeyUnion<T,  TKey extends keyof T> = {
  [P in TKey]: (k: T[P])=>void
} [TKey] extends ((k: infer I)=>void) ? I : never

export interface AssociatedTypes<T> {
   undefined: {
      type: undefined
      zero: undefined
      one: undefined
      nan: undefined
      real: undefined
   }
   number: {
      type: number
      zero: 0
      one: 1
      nan: typeof NaN
      real: number
   }
   Complex: T extends Complex<infer R> ? {
      type: Complex<R>
      zero: Complex<ZeroType<R>>
      one:  Complex<OneType<R> | ZeroType<R>>
      nan:  Complex<NaNType<R>>
      real: RealType<R>
   } : never
}
type AssociatedTypeNames = keyof AssociatedTypes<unknown>['undefined']
type ALookup<T, Name extends AssociatedTypeNames> = ValueIntersectionByKeyUnion<{
   [K in keyof AssociatedTypes<T>]:
   T extends AssociatedTypes<T>[K]['type'] ? AssociatedTypes<T>[K][Name] : unknown},
   keyof AssociatedTypes<T>>

type ZeroType<T> = ALookup<T, 'zero'> & T
type OneType<T> = ALookup<T, 'one'> & T
type NaNType<T> = ALookup<T, 'nan'>
type RealType<T> = ALookup<T, 'real'>
type Complex<T> = { re: T; im: T; }

I think I have extracted everything so that the snippet should compile; let me know if I have missed anything. Thanks!

Awesome! Looking forward to the next release ;-)

It's already available in v2.4.2!

Thanks! Did you actually publish to npm? npm still reports v2.4.1 as latest version.

That's really weird... the action which handles publishing errored with You cannot publish over the previously published versions: 2.4.2 however there's no trace of version 2.4.2 ever existing before this. On top of that, if you go to npm and search for this package, you'll see the latest version as 2.4.2 but if you go to the package page you'll see it as 2.4.1.
I'll be releasing a minor version really soon (most likely by the end of the week), all changes from 2.4.2 will be included in that release!

👍 For the time being I'm using the code straight from Github instead of npm, so I'm not blocked.


Bump oauth2.jwt.bearer.grant version

$subject
To capture changes from fix for https://github.com/wso2/product-is/issues/17775

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/6925793645

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/6925793645
Status: failure


Added support for field Components

Is it an option to re-use the ProjectComponent class instead of creating a new Component class? I assume they do the same?

ProjectCompnent seems to contain a description, while component doesn't have one at least on the issues I looked at.

Thank you for your contribution!


Fix Single Channel Images

As written single channel images (grayscale) will not work as some of the functions are hard coded to accept 3D arrays.   Converting single image to 3-channel images is one option. Maybe we can think of something that is more flexible.

This should be fixed in the new parameter space. Closing


IE9 support for max-width with workaround

Yeah, I agree with @dstorey. Since there's info on "initial" support now there's no need to expressly include it here, so I'll just remove the whole thing.


Implemented: functionality to reorder export threshold jobs or change the runTime of jobs in bulk(#2adm0cu)

Related Issues

Closes #
Short Description and Why It's Useful

Screenshots of Visual Changes before/after (If There Are Any)

IMPORTANT NOTICE - Remember to add changelog entry
Contribution and Currently Important Rules Acceptance

[x] I read and followed contribution rules

Will be closed by https://github.com/hotwax/threshold-management/pull/167


[BUG]: The batch that keeps on batching

Description of changes
This is an exploratory PR to verify the following scenario which are found to be failing:

Add (n>batch_size)
Update/Upsert (i in HNSW)
Delete (i)
Get (i)
Query (i)
Add (i)
Get (i)

Summarize the changes made by this PR.

Improvements & Bug fixes

...

New functionality

...

Test plan
How are these changes tested?

[ ] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#2113  👈
#2062
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @tazarov and the rest of your teammates on  Graphite


GH-14945: [Ruby] Add support for macOS 12 / Xcode 14

Xcode 14 shows a warning for "-undefined dynamic_lookup".
We need to use "-U${external_symbol_in_dynamic_loaded_library}" explicitly for all external symbols in dynamic loaded library without "-undefined dynamic_lookup".
In Red Arrow case, we use some symbols in Ruby/GLib2.

Closes: #14945

+1

Thanks for the super quick follow up @kou


Support turbo tasks inside of mod blocks

Description
I noticed this wasn't implemented when trying to write some unit tests inside of mod tests.
The same visibility requirements apply (must be at least pub(crate)).
Testing Instructions
Tested as part of https://github.com/vercel/turbo/pull/8529

#8530  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bgw and the rest of your teammates on  Graphite

Merge activity

Jun 24, 2:41 PM EDT: @bgw started a stack merge that includes this pull request via Graphite.


Pure client side upload

Currently the file upload is exclusively handled by the server (you need to provide the option.url).
I'm looking for a simple solution to send the file(s) to the function so I could use FileReader to process the file. I don't want to call any URL... Modern browsers support local databases that can store files and that use case is very common!

+1


[ENH] Pivot_wider column selection fix

PR Description
Please describe the changes proposed in the pull request:

Column selection works for all forms, including slicing

df

   Author  Article  Year  First  Second
0       1       11  2017      1       0
1       1       11  2018      0       1
2       2       22  2017      0       1
3       2       22  2018      0       1

df.pivot_wider(index = slice('Author','Article'),
               names_from = 'Year',
               flatten_levels = False)

               First      Second
Year            2017 2018   2017 2018
Author Article
1      11          1    0      0    1
2      22          0    0      1    1

df.pivot_wider(index = 'A*',
               names_from = 'Year',
               flatten_levels = False)

               First      Second
Year            2017 2018   2017 2018
Author Article
1      11          1    0      0    1
2      22          0    0      1    1

This PR resolves #1134.

PR Checklist

Please ensure that you have done the following:

[ ] PR in from a fork off your branch. Do not PR from <your_username>:dev, but rather from <your_username>:<feature-branch_name>.

[x] If you're not on the contributors list, add yourself to AUTHORS.md.

[x] Add a line to CHANGELOG.md under the latest version header (i.e. the one that is "on deck") describing the contribution.

Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

Automatic checks
There will be automatic checks run on the PR. These include:

Building a preview of the docs on Netlify
Automatically linting the code
Making sure the code is documented
Making sure that all tests are passed
Making sure that code coverage doesn't go down.

Relevant Reviewers

Please tag maintainers to review.

@ericmjl

🚀 Deployed on https://deploy-preview-1135--pyjanitor.netlify.app

Do you want to add a line to the change log? @Zeroto521 please do the honors of merging once the change log is updated!

Here we go.


remove django from base requirements

JIRA Issue: BOM-2418
Details:
Removing the Django requirement from base.in because:

Django requirement in base.in causes the django version in testeng-ci repo to upgrade to django==3.1.6 which causes failures in different jobs.
It upgrades the django version to 3+ in any repo that uses this package without applying constraint on django versions.

How can we achieve this when code-annotations does actually depend on Django? https://github.com/edx/code-annotations/blob/master/code_annotations/find_django.py#L10

How can we achieve this when code-annotations does actually depend on Django? https://github.com/edx/code-annotations/blob/master/code_annotations/find_django.py#L10

@regisb thanks for pointing it out. I didn't look in detail as I thought tests would fail but it turns out I was wrong. @jmbowman since django is required in this package, then should I add the constraint in the base.in file or in the setup.py file? so it doesn't affect the other packages/repos which are using this package.

My preference would be to make those imports (and the code using them) fail gracefully when Django is not installed, since there shouldn't be any Django models in a package or service that doesn't depend on Django.  Then Django would be an optional dependency of code-annotations rather than a required one.  The goal here is to avoid introducing Django as a dependency to all our Python repositories that don't already use it (via the dependency of edx-lint on this package).


qlog: bump version

So that we can cut a new release

I think the builds are failing for the chicken-egg dependency problem


docs(input): correct the autocomplete and autocorrect description

Short description of what this resolves:
Currently, the documentation for autocomplete and autocorrect is incorrectly set to the same as the placeholder documentation with the incorrect description:
@input {string} Instructional text that shows before the input has a value.

Changes proposed in this pull request:

Correct the documentation for both @Input attributes.
autocomplete: @input {string} Set the input's autocomplete property. Values: "on", "off".
autocorrect: @input {string} Set the input's autocorrect property. Values: "on", "off".

Ionic Version: 3.x
Fixes: #12609

I've just realized this has committed some incorrect files! Closing this PR and opening a new one with the correct pushes.


I wrote a hello world applet, but it can't run

hi，friend：
I wrote a hello world applet, but it can't run；
I generated a hello world applet from GCC in the cross tool chain compiled by myself, but it failed to run. The specific errors are as follows:
The steps are as follows:
（1）x86_64-linux-musl-gcc 11.c -o main
（2）./main
Errors are reported as follows:

Execute the following command ： ./main       Error reporting：  -bash: ./main: No such file or directory
Use the following command to view the dependent dynamic libraries: ldd main     Error reporting：./main: error while loading shared libraries: /lib64/libc.so: invalid ELF header
my host： Use this command to view my hosts  ：uname -m   result：x86_64

I tried some methods, but the problem still hasn't been solved. Can you help me see what caused the problem?

bash: ./main: No such file or directory

clear indication that the dynamic link interpreter isn't in the expected path.
check readelf -a main | grep -i interp to see the expected path.
you need to either install the musl dynlinker to that location, or run path/to/musl-cross-install/lib/libc.so main. alternatively you can static link your binary. the reason the dynlinker isnt installed to your system /lib is because mcm is a cross-compiler intended to compile binaries for other systems (which have the proper filesystem layout), and since it would require additional privileges.

Thank you for your reply. According to your method, both the static link method and the dynamic link method can make the program run.
I record my method. If other people encounter similar problems, you can refer to the following:

static link library: running: x86_ 64 Linux musl CC -static 11 C -o Jing
dynlinker ：### readelf -a main | grep -i interp ###  to see the expected path，result：[Requesting program interpreter: /lib/ld-musl-x86_64.so.1].   Run the LS -al command to check whether the current library has a soft connection?
Finally, copy the required library to the corresponding directory. My copy library commands are:
sudo cp -r libc.so /usr/lib
sudo cp -r ld-musl-x86_64.so.1 /usr/lib
Good wishes！


Add answer submission as middleware

Current answer submission logic is directly within router code, which should be separated.

Resolved at 3707d0422a6c0a751804e118a93c73f9696f8184


Cronline#next_time hangs on crossing dst boundary wiht ActiveSupport::TimeWithZone

The America/Santiago timezone switches to DST on Saturday 4th of September 2021. THis causes a hang when attempting to get the next time while crossing the DST boundary:
# Demonstrates the DST Crossing problem
require 'active_support'
require 'delayed_cron_job'

Time.zone = 'America/Santiago'

Timeout.timeout(10) { DelayedCronJob::Cronline.new('* * * * 1 ').next_time(Time.utc(2021, 9, 1)) }
puts "Time with utc works!"

Timeout.timeout(10) { DelayedCronJob::Cronline.new('* * * * 1 ').next_time(Time.new(2021, 9, 1, 0, 0, 0, TZInfo::Timezone.get('America/Santiago'))) }
puts "Time with timezone works!"

Timeout.timeout(10) { DelayedCronJob::Cronline.new('* * * * 1 ').next_time(Time.new(2021, 9, 1, 0, 0, 0, '-04')) }
puts "Time with timezone offset works!"

Timeout.timeout(10) { DelayedCronJob::Cronline.new('* * * * 1 ').next_time(Time.zone.local(2021, 1, 1)) }
puts "ActiveSupport::Timezone without dst crossing works!"

Timeout.timeout(10) { DelayedCronJob::Cronline.new('* * * * 1 ').next_time(Time.zone.local(2021, 9, 1)) }
# This is never reached :(
puts "ActiveSupport::Timezone with dst crossing works!"

The last line fails.
Instrumenting next_time to print the dates as it moves them:
        unless date_match?(time)
          otime = time
          dst = time.isdst
          time += (24 - time.hour) * 3600 - time.min * 60 - time.sec
          time -= 3600 if time.isdst != dst # not necessary for winter, but...
          puts time
          if time == otime
            raise 'time didnt move!'
          end
          next
        end

Gives the following output:
2021-09-02 00:00:00 UTC
2021-09-03 00:00:00 UTC
2021-09-04 00:00:00 UTC
2021-09-05 00:00:00 UTC
2021-09-06 00:00:00 UTC
Time with utc works!
2021-09-02 00:00:00 -0400
2021-09-03 00:00:00 -0400
2021-09-04 00:00:00 -0400
2021-09-05 01:00:00 -0300
2021-09-06 00:00:00 -0300
Time with timezone works!
2021-09-02 00:00:00 -0400
2021-09-03 00:00:00 -0400
2021-09-04 00:00:00 -0400
2021-09-05 01:00:00 -0300
2021-09-06 00:00:00 -0300
Time with timezone works!
2021-09-02 00:00:00 -0400
2021-09-03 00:00:00 -0400
2021-09-04 00:00:00 -0400
2021-09-05 00:00:00 -0400
2021-09-06 00:00:00 -0400
Time with timezone offset works!
2021-01-02 00:00:00 -0300
2021-01-03 00:00:00 -0300
2021-01-04 00:00:00 -0300
ActiveSupport::Timezone without dst crossing works!
2021-09-02 00:00:00 -0400
2021-09-03 00:00:00 -0400
2021-09-04 00:00:00 -0400
2021-09-04 23:00:00 -0400
2021-09-04 23:00:00 -0400
Traceback (most recent call last):
        9: from test-delayed.rb:21:in `<main>'
        8: from /home/felipe/.rbenv/versions/2.7.4/lib/ruby/2.7.0/timeout.rb:110:in `timeout'
        7: from /home/felipe/.rbenv/versions/2.7.4/lib/ruby/2.7.0/timeout.rb:33:in `catch'
        6: from /home/felipe/.rbenv/versions/2.7.4/lib/ruby/2.7.0/timeout.rb:33:in `catch'
        5: from /home/felipe/.rbenv/versions/2.7.4/lib/ruby/2.7.0/timeout.rb:33:in `block in catch'
        4: from /home/felipe/.rbenv/versions/2.7.4/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
        3: from test-delayed.rb:21:in `block in <main>'
        2: from /home/felipe/src/satelabs/buk/gems/ruby/2.7.0/gems/delayed_cron_job-0.7.4/lib/delayed_cron_job/cronline.rb:131:in `next_time'
        1: from /home/felipe/src/satelabs/buk/gems/ruby/2.7.0/gems/delayed_cron_job-0.7.4/lib/delayed_cron_job/cronline.rb:131:in `loop'
/home/felipe/src/satelabs/buk/gems/ruby/2.7.0/gems/delayed_cron_job-0.7.4/lib/delayed_cron_job/cronline.rb:139:in `block in next_time': asdf (RuntimeError)

It looks to me as if the time.isdst check should change sign depending on whether we are entering or exiting DST, but I'm not sure I understand the logic correctly. Interestingly, the native ruby Time never triggers the isdst check (Time.new(2021, 9, 5, 1, 0, 0, TZInfo::Timezone.get('America/Santiago')).isdst is still false).

Thank you for reporting the issue. Seems like the cron algorithm is not entirely solved yet ;)
Probably relates to https://github.com/floraison/fugit/issues/60
I will look into using Fugit directly or inline its Cron class, as this is the source for our Cronline class anyways.

I released version 0.8.0 depending on the fixed fugit version.


TypeError: cannot pickle 'mmap.mmap' object

🐛 Bug Description
When run rl_order_execution, I get the error "TypeError: cannot pickle 'mmap.mmap' object". According to the same error post(https://github.com/stamparm/maltrail/issues/11823), it seems to be a problem with MacOs.  But I can’t find the way to resolve it in Qlib
To Reproduce
Steps to reproduce the behavior:
run "python -m qlib.rl.contrib.train_onpolicy --config_path exp_configs/train_opds.yml --run_backtest" in MacOs(M1max, Ventura 13.2.1)
Expected Behavior
Screenshot
Environment
Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.

Qlib version: latest rl_order_execution
Python version:  3.8
OS (Windows, Linux, MacOS): MacOS
Commit number (optional, please provide it if you are using the dev version):

Additional Notes

Hi,could you show me the train_opds.yml which can help me to reproduce the problem?

I just follow the example , train_opds.yml is the same as https://github.com/microsoft/qlib/blob/main/examples/rl_order_execution/exp_configs/train_opds.yml
The complete log is:
[27036:MainThread](2023-03-29 15:01:40,515) INFO - qlib.Initialization - [config.py:416] - default_conf: client. [27036:MainThread](2023-03-29 15:01:40,947) INFO - qlib.Initialization - [__init__.py:74] - qlib successfully initialized based on client settings. [27036:MainThread](2023-03-29 15:01:40,947) INFO - qlib.Initialization - [__init__.py:76] - data_path={'__DEFAULT_FREQ': PosixPath('/Users/Meteor/.qlib/qlib_data/cn_data')} [27036:MainThread](2023-03-29 15:01:41,010) INFO - qlib.rl.trainer.trainer - [trainer.py:211] -  2023-03-29 15:01:41	Train iteration 1/500 [27036:MainThread](2023-03-29 15:01:41,010) INFO - qlib.rl.trainer.vessel - [vessel.py:144] - Training initial states collection size: 36002 [27036:MainThread](2023-03-29 15:01:41,010) INFO - qlib.rl.utils.data_queue - [data_queue.py:70] - Automatically set data queue maxsize to 10 to avoid overwhelming. [27036:MainThread](2023-03-29 15:01:42,042) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[TypeError: cannot pickle 'mmap.mmap' object].   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/contrib/train_onpolicy.py", line 261, in <module>     main(config, run_training=not args.no_training, run_backtest=args.run_backtest)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/contrib/train_onpolicy.py", line 234, in main     train_and_test(   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/contrib/train_onpolicy.py", line 155, in train_and_test     train(   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/trainer/api.py", line 63, in train     trainer.fit(vessel)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/trainer/trainer.py", line 223, in fit     vector_env = self.venv_from_iterator(iterator)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/trainer/trainer.py", line 302, in venv_from_iterator     return vectorize_env(   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/utils/finite_env.py", line 369, in vectorize_env     return finite_env_cls(logger, [env_factory for _ in range(concurrency)])   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/utils/finite_env.py", line 128, in __init__     super().__init__(env_fns, **kwargs)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 412, in __init__     super().__init__(env_fns, worker_fn, **kwargs)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 80, in __init__     self.workers = [worker_fn(fn) for fn in env_fns]   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 80, in <listcomp>     self.workers = [worker_fn(fn) for fn in env_fns]   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 410, in worker_fn     return SubprocEnvWorker(fn, share_memory=True)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/worker/subproc.py", line 160, in __init__     self.process.start()   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/process.py", line 121, in start     self._popen = self._Popen(self)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/context.py", line 224, in _Popen     return _default_context.get_context().Process._Popen(process_obj)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/context.py", line 284, in _Popen     return Popen(process_obj)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__     super().__init__(process_obj)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__     self._launch(process_obj)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch     reduction.dump(process_obj, fp)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/reduction.py", line 60, in dump     ForkingPickler(file, protocol).dump(obj)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/utils.py", line 18, in __getstate__     return cloudpickle.dumps(self.data)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps     cp.dump(obj)   File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 632, in dump     return Pickler.dump(self, obj) TypeError: cannot pickle 'mmap.mmap' object

I just follow the example , train_opds.yml is the same as https://github.com/microsoft/qlib/blob/main/examples/rl_order_execution/exp_configs/train_opds.yml
The complete log is:

[27036:MainThread](2023-03-29 15:01:40,515) INFO - qlib.Initialization - [config.py:416] - default_conf: client.
[27036:MainThread](2023-03-29 15:01:40,947) INFO - qlib.Initialization - [init.py:74] - qlib successfully initialized based on client settings.
[27036:MainThread](2023-03-29 15:01:40,947) INFO - qlib.Initialization - [init.py:76] - data_path={'__DEFAULT_FREQ': PosixPath('/Users/Meteor/.qlib/qlib_data/cn_data')}
[27036:MainThread](2023-03-29 15:01:41,010) INFO - qlib.rl.trainer.trainer - [trainer.py:211] -
2023-03-29 15:01:41	Train iteration 1/500
[27036:MainThread](2023-03-29 15:01:41,010) INFO - qlib.rl.trainer.vessel - [vessel.py:144] - Training initial states collection size: 36002
[27036:MainThread](2023-03-29 15:01:41,010) INFO - qlib.rl.utils.data_queue - [data_queue.py:70] - Automatically set data queue maxsize to 10 to avoid overwhelming.
[27036:MainThread](2023-03-29 15:01:42,042) ERROR - qlib.workflow - [utils.py:41] - An exception has been raised[TypeError: cannot pickle 'mmap.mmap' object].
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/contrib/train_onpolicy.py", line 261, in
main(config, run_training=not args.no_training, run_backtest=args.run_backtest)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/contrib/train_onpolicy.py", line 234, in main
train_and_test(
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/contrib/train_onpolicy.py", line 155, in train_and_test
train(
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/trainer/api.py", line 63, in train
trainer.fit(vessel)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/trainer/trainer.py", line 223, in fit
vector_env = self.venv_from_iterator(iterator)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/trainer/trainer.py", line 302, in venv_from_iterator
return vectorize_env(
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/utils/finite_env.py", line 369, in vectorize_env
return finite_env_cls(logger, [env_factory for _ in range(concurrency)])
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/qlib/rl/utils/finite_env.py", line 128, in init
super().init(env_fns, **kwargs)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 412, in init
super().init(env_fns, worker_fn, **kwargs)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 80, in init
self.workers = [worker_fn(fn) for fn in env_fns]
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 80, in
self.workers = [worker_fn(fn) for fn in env_fns]
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/venvs.py", line 410, in worker_fn
return SubprocEnvWorker(fn, share_memory=True)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/worker/subproc.py", line 160, in init
self.process.start()
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/tianshou/env/utils.py", line 18, in getstate
return cloudpickle.dumps(self.data)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/Users/Meteor/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 632, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle 'mmap.mmap' object


Handle 'or' in highlight search string

This allows using multiple patterns e.g. for Doxygen todos
Closes #30

It would also be nice to have the highlighted text align with the default (highlight leading space and don't highlight trailing space) but I wasn't sure how to make that happen. I could add that if you point me in the right direction.

Hi! I just pushed an update that allows setting multiple patterns as a table for highlight.pattern instead of a string.
This should be even easier.
pattern = { [[.*<(KEYWORDS)\s*:]], [[.*\@(KEYWORDS)\s*]] }

Thanks!

Sorry for necroposting, but that's not working for me in same case - chars around keyword (@, :) are skipped from highlighting


Unable to fork and pull-request the wiki

Hello,
Is it possible to move all the wiki pages into a 'docs' directory so that I can suggest edits?
https://stackoverflow.com/a/11481887
There is great content on this repository and I would love to contribute. 😊
Thanks,
Cameron

Oh I had no idea that you couldn't do a pull request on github wiki. Thanks for raising this and I'd love for you to contribute. Let me go through the link you shared and see what's the best way to do this.

The link you shared is a pretty good idea. But I guess you'll need to first create a repository on your own account and then let me know to merge from your repo to mine. I'm happy to try it out if you want to give it a try.

Sounds good to me! Thank you for following up. I will work on creating a pull-request that contains changes that bring exactly what is in the wiki pages into the repo.
I can follow-up with a second pull-request that contains my suggestions to improve the wiki pages.

I have attempted to submit the CLA and the form has given me an error message. I have attached a screenshot of the error to this comment. I attempted this from mobile & PC and from two different wifi connections.
Let me know if you have any recommendations to resolve this error.

@kirkster96 Thank you so much for the amazing PR and for filling out the CONTRIBUTING.md file. It looks great. I'm now debating whether I should just scrap the wiki and keep everything in the docs folder. Might be easier for other to contribute and less overall work for me in copying it from there to the wiki. Thoughts?

No problem, I am happy to see that the PR was accepted!
I think that removing the wiki is the direction to go. Yesterday, I was following along with the Building SoC Design page and I found a link in the page content that points to the wiki page. So there are still more links that need to be updated with another PR.
I will take some time to go through all the content carefully and make sure all the links are no longer pointing to the wiki. In the mean time, it may be a good idea to keep the wiki around in order to keep the links from breaking.

Yesterday, I was following along with the Building SoC Design page and I found a link in the page content that points to the wiki page. So there are still more links that need to be updated with another PR.

Haha yea - I noticed that after I approved the PR as well. I fixed a couple of them, but there are quite a few. Thanks for taking the time to review and fix them, really appreciate your help.
Here's a quick grep of all the files and all the places where it needs to be updated:
Filelist:
~/de10-nano master > grep -l -R -o -P '.{30}https:\/\/.*?de10-nano\/wiki\/.{40}'                                                                                mango@tree task 1
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-2.md
docs/Building-SoC-Design.md
docs/Creating-a-Bootscript.md
docs/Building-the-Kernel.md
docs/Simple-Hardware-Adder_-Setting-up-the-Adder.md
docs/Archlinux-ARM-Root-File-System.md
docs/Simple-Hardware-Adder_-Writing-the-Software.md
docs/FPGA-SDRAM-Communication_-Avalon-MM-Agent-Slave-Trigger-Component.md
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-1.md
docs/Simple-Hardware-Adder_-Initial-Project-Setup.md
docs/FPGA-SDRAM-Communication_-More-about-the-Avalon-Memory-Mapped-Interface.md
docs/Flash-FPGA-On-Boot-Up.md
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-3.md
docs/Debian-Root-File-System.md
docs/FPGA-SDRAM-Communication_-Introduction.md
docs/_Sidebar.md
docs/Building-the-SD-Card-image.md

Locations:

docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-2.md:` that we created over [here](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Initial-Project-S
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-2.md:steps as shown on [this page](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Initial-Project-S
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-2.md:plained in the [Simple Adder](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Custom-Avalon-MM-
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-2.md:he design as explained [here](https://github.com/zangman/de10-nano/wiki/Flash-FPGA-from-HPS-(running-Linux)) and
docs/Building-SoC-Design.md:` format as explained [here](<https://github.com/zangman/de10-nano/wiki/Flash-FPGA-from-HPS-(running-Linux)#crea
docs/Building-SoC-Design.md: the FPGA either on [boot-up](https://github.com/zangman/de10-nano/wiki/Flash-FPGA-On-Boot-Up) or from [the HPS]
docs/Building-SoC-Design.md:the [development environment](https://github.com/zangman/de10-nano/wiki/Setting-up-the-Development-Environment)
docs/Creating-a-Bootscript.md:bed in the [Building U-Boot](<https://github.com/zangman/de10-nano/wiki/Building-the-Universal-Bootloader-(U-Boo
docs/Creating-a-Bootscript.md:me>) section. [This section](<https://github.com/zangman/de10-nano/wiki/Building-the-Universal-Bootloader-(U-Boo
docs/Building-the-Kernel.md: for WIFI as explained [here](https://github.com/zangman/de10-nano/wiki/%5BOptional%5D-Setting-up-Wifi#compile-t
docs/Simple-Hardware-Adder_-Setting-up-the-Adder.md:. This is covered over [here](https://github.com/zangman/de10-nano/wiki/Configuring-the-Device-Tree) so I won't
docs/Simple-Hardware-Adder_-Setting-up-the-Adder.md:gh the steps described [here](https://github.com/zangman/de10-nano/wiki/Configuring-the-Device-Tree) and verify
docs/Simple-Hardware-Adder_-Setting-up-the-Adder.md: already been covered [here](<https://github.com/zangman/de10-nano/wiki/Flash-FPGA-from-HPS-(running-Linux)>) an
docs/Archlinux-ARM-Root-File-System.md:h all the steps listed [here](https://github.com/zangman/de10-nano/wiki/Building-the-SD-Card-image) all the way
docs/Archlinux-ARM-Root-File-System.md: the [kernel and device tree](https://github.com/zangman/de10-nano/wiki/Building-the-SD-Card-image#kernel-and-de
docs/Archlinux-ARM-Root-File-System.md:d back over to [this section](https://github.com/zangman/de10-nano/wiki/Building-the-SD-Card-image#cleanup) to c
docs/Simple-Hardware-Adder_-Writing-the-Software.md:up the path in [this section](https://github.com/zangman/de10-nano/wiki/Setting-up-the-Development-Environment#a
docs/Simple-Hardware-Adder_-Writing-the-Software.md: to `ON` as described [here](<https://github.com/zangman/de10-nano/wiki/Flash-FPGA-from-HPS-(running-Linux)#set-
docs/FPGA-SDRAM-Communication_-Avalon-MM-Agent-Slave-Trigger-Component.md:plained in the [Simple Adder](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Custom-Avalon-MM-
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-1.md:MM Read and Write interfaces](https://github.com/zangman/de10-nano/wiki/FPGA-SDRAM-Communication:-More-about-the
docs/Simple-Hardware-Adder_-Initial-Project-Setup.md:HRD to the working directory](https://github.com/zangman/de10-nano/wiki/Building-SoC-Design#copy-the-ghrd-to-wor
docs/FPGA-SDRAM-Communication_-More-about-the-Avalon-Memory-Mapped-Interface.md: to the [Primer on Avalon MM](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Primer-on-Avalon-
docs/FPGA-SDRAM-Communication_-More-about-the-Avalon-Memory-Mapped-Interface.md:n the [Simple Hardware Adder](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Custom-Avalon-MM-
docs/FPGA-SDRAM-Communication_-More-about-the-Avalon-Memory-Mapped-Interface.md:ed from this [9 minute video](https://www.youtube.com/watch?v=8GAqT3nzHeQ). I strongly recommend watching this one, it was a lightbulb moment for me. Below, I will try and expand a bit more over what the video covers. But if everything in the video makes sense to you, feel free to just skip to the [last section](https://github.com/zangman/de10-nano/wiki/FPGA-SDRAM-Communication:-More-about-the
docs/Flash-FPGA-On-Boot-Up.md:he U-Boot to flash the FPGA](<https://github.com/zangman/de10-nano/wiki/Building-the-Universal-Bootloader-(U-Boo
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-3.md: designs [without rebooting](<https://github.com/zangman/de10-nano/wiki/Flash-FPGA-from-HPS-(running-Linux)>) ev
docs/FPGA-SDRAM-Communication_-Avalon-MM-Host-Master-Component-Part-3.md:you can [create a bootscript](https://github.com/zangman/de10-nano/wiki/Creating-a-Bootscript) which will make i
docs/Debian-Root-File-System.md:ng up WIFI as detailed [here](https://github.com/zangman/de10-nano/wiki/%5BOptional%5D-Setting-up-Wifi#install-t
docs/FPGA-SDRAM-Communication_-Introduction.md:n the [Simple Hardware Adder](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Introduction) pro
docs/_Sidebar.md:niversal Bootloader (U-Boot)](https://github.com/zangman/de10-nano/wiki/Building-the-Universal-Bootloader-(U-Boo
docs/_Sidebar.md:[Custom Avalon MM Components](https://github.com/zangman/de10-nano/wiki/Simple-Hardware-Adder:-Custom-Avalon-MM-
docs/_Sidebar.md:nt/Slave - Trigger Component](https://github.com/zangman/de10-nano/wiki/FPGA-SDRAM-Communication:-Avalon-MM-Agen
docs/_Sidebar.md:* [Avalon MM Host/Master - 1](https://github.com/zangman/de10-nano/wiki/FPGA-SDRAM-Communication:-Avalon-MM-Host
docs/_Sidebar.md:* [Avalon MM Host/Master - 2](https://github.com/zangman/de10-nano/wiki/FPGA-SDRAM-Communication:-Avalon-MM-Host
docs/_Sidebar.md:* [Avalon MM Host/Master - 3](https://github.com/zangman/de10-nano/wiki/FPGA-SDRAM-Communication:-Avalon-MM-Host
docs/Building-the-SD-Card-image.md:, then jump to [this section](https://github.com/zangman/de10-nano/wiki/Archlinux-ARM-Root-File-System) to conti

Having just taken the easy route and flashed a ready made image, I have a suggestion to add to the part of the readme that relates to doing that, mostly just linking to the part of the relevant page that explains what to do with the image:
ON PAGE:
README.md
https://github.com/zangman/de10-nano#readme
UNDER:
https://github.com/zangman/de10-nano#can-i-just-get-the-image-i-dont-want-to-go-through-the-guide-to-get-debian-on-my-de10-nano
Can I just get the image? I don't want to go through the guide to get Debian on my DE10-Nano.
-> Sure, You can download the SD Card image from the releases page.
ADD:

See Creating the SD Card Image

( link this to "Writing to SD Card" on de10-nano/docs/Building-the-SD-Card-image.md https://github.com/zangman/de10-nano/blob/master/docs/Building-the-SD-Card-image.md#writing-to-sd-card )
ON PAGE:
Building-the-SD-Card-image.md
UNDER:
"Writing in Windows"
https://github.com/zangman/de10-nano/blob/master/docs/Building-the-SD-Card-image.md#writing-in-windows
ADD:

Rufus can also write from a zipped image file, without needing to unzip it.

@Andy2No Thanks for sharing - would you be okay to send a PR with these changes? I can review and accept it. You'll need to complete the steps in CONTRIBUTING.md file first though.

At first reading, I'm not happy with the terms of google's CLA.
I am happy for all or any part of any suggestion I make to be used by someone else though, whether or not it's with the same exact wording.  I'm happy for copyright to pass to whoever chooses to implement my suggestions, especially since they are just suggestions.

@zangman sorry it took so long. I became very busy with work. I hope that these changes complete this issue.
Let me know if you see something that needs to be corrected and I can go through it again.

@kirkster96 Amazing, thank you so much! Really appreciate your effort in cleaning up and making this wiki more user friendly.

@kirkster96 In trying to read through it all, I'm still getting a little lost because there isn't a consistent link at the bottom of each page to take me to a next one.
I think it would help if the whole thing was linked as a book, so that after reading one page you can go straight to the next one - or a next one.  At present, the last page of each sub section just has a back link, to the previous page in that sub section.
Also, at the bottom of the main readme page, it would be good if it had a link to either the table of contents, or the first linked page in that table - or maybe both, but people who've used github before are probably quite comfortable with clicking the Code tab to get back to the readme, so I'd suggest the "Next" link at the bottom of the readme should link to the "Getting Started" page.

@kirkster96 In trying to read through it all, I'm still getting a little lost because there isn't a consistent link at the bottom of each page to take me to a next one.
I think it would help if the whole thing was linked as a book, so that after reading one page you can go straight to the next one - or a next one. At present, the last page of each sub section just has a back link, to the previous page in that sub section.
Also, at the bottom of the main readme page, it would be good if it had a link to either the table of contents, or the first linked page in that table - or maybe both, but people who've used github before are probably quite comfortable with clicking the Code tab to get back to the readme, so I'd suggest the "Next" link at the bottom of the readme should link to the "Getting Started" page.

Thank you, I will create a new issue and work on creating the nagivation links. I will make a new PR for that when I am done. I think this issue can be closed now that the PR has been accepted.

@Andy2No I had some time today - so I updated the nav links to link to the next and previous section as you suggested. I don't think Next needs to go to the README as there is a link to the TOC on every page.
@kirkster96 FYI


Could not find "store" in the context of "Connect(WithStyles)".

Do you want to request a feature or report a bug?
bug
What is the current behavior?
I get an error:
Invariant Violation: Could not find "store" in the context of "Connect(WithStyles)". Either wrap the root component in a <Provider>, or pass a custom React context provider to <Provider> and the corresponding React context consumer to Connect(WithStyles) in connect options.
This error is:
{ Invariant Violation: Could not find "store" in the context of "Connect(WithStyles)". Either wrap the root component in a <Provider>, or pass a custom React context provider to <Provider> and the corresponding React context consumer to Connect(WithStyles) in connect options.     at invariant (/Users/evan.gatchell/code/resubmit-ui/node_modules/invariant/invariant.js:40:15)     at Connect.renderWrappedComponent (/Users/evan.gatchell/code/resubmit-ui/node_modules/react-redux/lib/components/connectAdvanced.js:162:32)     at a.render (/Users/evan.gatchell/code/resubmit-ui/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:42:437)     at a.read (/Users/evan.gatchell/code/resubmit-ui/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:39:436)     at renderToString (/Users/evan.gatchell/code/resubmit-ui/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:51:83)     at render (/Users/evan.gatchell/code/resubmit-ui/node_modules/next-server/dist/server/render.js:86:16)     at Object.renderPage (/Users/evan.gatchell/code/resubmit-ui/node_modules/next-server/dist/server/render.js:211:20)     at Function.module.exports.5w0S._Document.getInitialProps.ctx [as getInitialProps] (/Users/evan.gatchell/code/resubmit-ui/.next/server/static/XZU0ghjI86u3oGdCeahHH/pages/_document.js:411:20)     at Object.loadGetInitialProps (/Users/evan.gatchell/code/resubmit-ui/node_modules/next-server/dist/lib/utils.js:42:35)     at Object.renderToHTML (/Users/evan.gatchell/code/resubmit-ui/node_modules/next-server/dist/server/render.js:218:36) name: 'Invariant Violation', framesToPop: 1 }
What is the expected behavior?
Everything works in dev mode, but when I run build and start with Next.js, I get this error.
Which versions of Redux, and which browser and OS are affected by this issue? Did this work in previous versions of Redux?
Redux v4.0.1
React Redux v6.0.0
Misc
Here's the part where I use connect. If I remove the "withStyles" bit, I get the same error, but for the Index component.
export default connect(mapStateToProps, mapDispatchToProps)(withStyles(styles)(Index))

Something is stripping out your <Provider>. But it's not this library doing it, so you're going to have to look at your build tooling.


Unconferencing rooms

Add 6 Nethack themed unconference rooms, linked to a main unconference lobby with a note wall for topics. Hasn't been added to the map yet, since Em is working on a change to that atm.

Looks good to me! Will leave you to fix the merge conflicts (it won't let me do it via the web UI) and then I'm either happy to wire it up to the map myself or leave that for you to try.

It'll need the map updated (I don't have access to a Mac for the ASCII map software) but it should actually work now!!


Fatal error: Cannot use 'Object' as class name as it is reserved

Environment
CentOS 7 and PHP 7.2.23
Branch
master/cfef19ae05a6e28f3e34f351be1a33051f97241b
Issue
when exec ./yii walle/setup you get:
Fatal error: Cannot use 'Object' as class name as it is reserved in /home/aiji/github/walle-web/vendor/yiisoft/yii2/base/Object.php on line 77

Probably caused by this issue https://github.com/yiisoft/yii2/issues/14823：
Cannot use yii because of **Object** is a restricted word in PHP 7.2.

Similar issue already exists!


How do I install Burp Suite?

How do I install Burp Suite?

I think it's easy to do so. Just write the same question in the Google or YouTube search bar.
If your question was how to use the scripts, I have already updated the README with videos.


Copy and pasting text into a chat gives you html decoded strings

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

open newDot on the web.
Copy some text that contains whitespaces from the Mac's Note.app or any other application.
Paste that into the composer.

Expected Result:
Special Html entities should be shown correctly.
Actual Result:
You can see a lot of them in HTML form.
e.g. service.  ,   We’ve, haven’t .
Workaround:
None.
Platform:

Where is this issue occurring?

Web
iOS
Android
Desktop App
Mobile Web

Version Number: latest
Notes/Photos/Videos:
https://expensify.slack.com/archives/C01GTK53T8Q/p1629395261042000?thread_ts=1627358860.323100&cid=C01GTK53T8Q
View all open jobs on GitHub

Triggered auto assignment to @laurenreidexpensify (AutoAssignerTriage), see https://stackoverflow.com/c/expensify/questions/4749 for more details.

Explanation.

A lot of apps use HTML entities instead of ASCII characters which when pasted to composer are not decoded.

Proposal

There is a good library that does this for us https://www.npmjs.com/package/html-entities.
But this contains a lot of code which we don't really need.
So my idea is to extract the List of chars from https://github.com/mdevils/html-entities/blob/master/src/named-references.ts.
Then using the regexp match we can convert them easily without using the library.
This function will be an extended version of underscore the _.unscape https://github.com/jashkenas/underscore/blob/master/modules/_createEscaper.js

@parasharrajat already fixed some examples in this GH https://github.com/Expensify/App/issues/4169
Let's make sure we address any/all relevant ones so we don't end up with a ton of small issues/fixes (which eat away at valuable resources)

Triggered auto assignment to @kadiealexander (AutoAssignerTriage), see https://stackoverflow.com/c/expensify/questions/4749 for more details.

I'm OOO next week so reassigning this - looks like @mallenexpensify might have a steer on whether we need to move this one to engineering

Seems like a reasonable formatting change! Sending to eng for another set of eyes.

Triggered auto assignment to @Julesssss (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

Hi @parasharrajat. Love the proposal as it removes the library dependency. Please make a start when you're ready.

Triggered auto assignment to @arielgreen (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

Whoops, I assumed I was assigned via the Exported label. Please hold until we reach that step 😄

Posted job and invited @parasharrajat.

🚀 Deployed to staging by @Beamanator in version: 1.0.88-3 🚀

platform
result

🤖 android 🤖
cancelled 🔪

🖥 desktop 🖥
cancelled 🔪

🍎 iOS 🍎
cancelled 🔪

🕸 web 🕸
success ✅

🚀 Deployed to production by @roryabraham in version: 1.0.90-2 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅

Sorry, we missed the payment notification as this wasn't exported, my fault.

@arielgreen this can be paid today 🎉

Paid + bonus for reporting, thanks for your work here @parasharrajat!


Typing/iostream

remove setattr
add type annotations to spinner context manager

these changes alone knock out around 100 mypy errors.
note, this pull request is stacked on top of #298, so that should be merged first, and this pull request can be rebased

I've also flattened the spinner context manager. It was a context manager that yielded another context manager, so I've removed a level of nesting.

you might be wondering why i've add changes outside of iostream.py- this is because mypy type-checks recursively when given a single file. This extra changes were required to ensure that pdm run mypy pdm/iostream.py returns 0 errors


Total No. of meta_filtered_uids is 296k instead of 150k+50k+50k mentioned in the paper.

Hey Yash,
I really liked your paper. The general idea is interesting and I am trying out some personal experiments on objaverse.
I noticed that the data filtering code produces 298k uids instead of 250k.
Could you please tell why is that the case.

@yashkant apologies for 150k, as it's mentioned 150k+50k+50k. Extra 46k is for overlapping?

Thank you, Yash for clarification!


Add CertifyVEXStatement and IngestVEXStatement

@ivanvanderbyl I've implemented the VEX endpoints.
There are also all the tests written for the inmem backend (i.e. certifyVEXStatement_test.go): they're all working fine and the coverage is ~85%.

@ivanvanderbyl now everything should be changed as requested :wink:


Implement major color representations

Nice job with this library.
The only thing I think is that it does not implement all the colour representations. Here's a check-list to motivate you to add those.

[x] Web (Hex/Colour Names)
[x] RGB
[ ] CIELAB (LAB)
[ ] HCL / LCH
[x] HSL
[ ] HSV
[ ] XYZ
[ ] CYM
[ ] CYMK

Possibly the best part of this library is that it is independent (single-file, no dependencies). I request you to please keep it the same.

Hehe, thanks for your interest and support, and sharing your POV. There are no plans to get this more complex than it is now. And PR are welcome for all these other formats :), or even some links to page expliciting the conversion algorithms that could be used would surely help these to be released sooner.

You could use colorsys. That has RGB, HSV, HSL, YUI.

Also, here's a link to a page with the relevant color-conversion formulae. http://www.easyrgb.com/?X=MATH
Looks like colours should be stored as RGB (as decimal).

Bump. Any chance someone else would work on this?

Cool and nice library, but I'm looking for a library that supports LAB.

Closing since doesn't look like anyone's gonna be doing this. :)


Enforce type integer in DataLists byID and byIDs methods

Enforce type integer in DataLists byID and byIDs methods and throws exception if invalid type is given.
This is to prevent unexpected behaviour when filtering by ID without checking the type of params.
E.g. If using the get parameter 1;"}};alert("ALERT"); unfiltered as parameter in byID it would return the entry with ID 1.

PHP uses type juggling, so we treat (string)"1" as a perfectly valid database ID identifier. In addition, various DB connectors return different casting for integer values retrieved from sql queries. Sometimes they are type-safe (int), sometimes they are pure strings.
We could just use is_numeric to get the same value without the strictness.

Thanks @ichaber good job.


feat!: add raise_on_fail to send_transaction()

Opening this preemptively to track changes, git rebases, etc
#99

Closing in favor of https://github.com/ApeWorX/ape/pull/1125


[Plutus Playground] Can't run two actions from the same wallet

Area
Plutus Playground.
Tried https://playground.plutus.iohkdev.io/ and local version built against plutus-starter-devcontainer/v1.0.8 tag.
Summary
It appears you only have one action per wallet in the simulator.
Steps to reproduce

Start playground
Select Starter app
Compile
Open simulator
Create actions:

Publish 20 lovelace
wait for 1 block
publish another 30 lovelace
wait for 1 block

evaluate

Expected behavior
50 lovelace are published.
Screenshots and attachments

https://playground.plutus.iohkdev.io/ is out-dated. And plutus-starter-devcontainer/v1.0.8 is two weeks old already. I have tested against master and it works fine. Could you please try plutus-starter-devcontainer/v1.0.9? It was released yesterday.

@ak3n We have a similar issue in our test suite, where a redeem call is ignored.
It seems you can only run one action per wallet in the tests. I created a minimal repro case: https://github.com/hasufell/plutus-bug/blob/a3bb08a7f004910660c98b1e90e9bfe5bf6b5c10/test/UnitTests.hs#L51-L64


Video is playing and scroll listener's first and last complete visible positions are always -1

Hi,
I am trying to play videos but it is not working, i tried to figure out the issue and as far as i found is that scroll listener fist and last complete positions are always -1. and that is it not playing any video.
Can you help me to resolve the error.
Thanks

Yes, in this case, your whole Recyclerview item(not just AAH_VideoImage) must not be fitting in a single screen.
If you still require to play-video when whole Recyclerview item is not completely visible, you can replace firstCompletelyVisiblePosition  with firstVisiblePosition  and lastCompletelyVisiblePosition  with lastVisiblePosition  in AAH_CustomRecyclerView.java in library. Also comment/remove ((AAH_CustomViewHolder) holder).pauseVideo(); anywhere in that file.
Let me know if this doesn't work for you, I will send you a working sample.

Thanks for the reply. then how i will pause the video. because i want to play on video at a time.

I will really appreciate if  you can send me working sample because i tried to replace things as you suggested but its not working. Thanks

Hi, please check the below project. Hope that works for you.
https://drive.google.com/file/d/0B5i30XeKjRLNYXNuSmFyME01dVE/view?usp=sharing

Thanks @Krupen Sample is working fine
Last question is two videos are playing at a time. how to avoid playing two videos at a time.
Thanks

Instead of for loop in AAH_CustomRecyclerView.java which calls playVideo() , just call playVideo() on first item of for loop only.

Closing this issue, feel free to re-open if not solved yet.


Update Ballerina tool help text

Fixes #18744
Check List

[x] Read the Contributing Guide
[ ] Required Balo version update
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #18745 into master will increase coverage by 4.84%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #18745      +/-   ##
==========================================
+ Coverage   15.08%   19.92%   +4.84%
==========================================
  Files          47       52       +5
  Lines        1220     1355     +135
  Branches      184      208      +24
==========================================
+ Hits          184      270      +86
- Misses       1023     1072      +49
  Partials       13       13

Impacted Files
Coverage Δ

tool-plugins/vscode/src/overview/activator.ts
0% <0%> (ø)
:arrow_up:

tool-plugins/vscode/src/overview/renderer.ts
0% <0%> (ø)
:arrow_up:

composer/packages/tracing/src/ToolBar.tsx
71.05% <0%> (ø)

composer/packages/tracing/src/TraceList.tsx
94.44% <0%> (ø)

composer/packages/tracing/src/DetailView.tsx
79.31% <0%> (ø)

composer/packages/tracing/src/TraceLogs.tsx
70.37% <0%> (ø)

composer/packages/tracing/src/index.ts
0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11498b7...8a557b1. Read the comment docs.


Refine and document Diagnostics.cs

[x] Disable default behavior to create and log to a file
[x] Add one more Log Level (info) for the possibility to display a message when building a release that is not warning or error
[x] Try to reduce the extra information while displaying debug messages
[x] Create a wiki page about our log system

Wiki page created:
https://github.com/FUSEEProjectTeam/Fusee/wiki/Logging

ref: https://github.com/FUSEEProjectTeam/Fusee/pull/153


We probably need to set some http transport requirement in composer?

I'm never quite sure how to declare that you need some kind of transport (as opposed to that you need exactly guzzle 6 adapter or so...) but it's probably something along the lines of(?):
"guzzlehttp/guzzle": "@stable"

@patjnr you added this requirement (specifically Guzzle6, why?) but it does not at all seem to be used/required. Turns out the module itself just used a plain CURL method for requests... And Facebook doesn't seem to require any specific http transport (and if so seems to require Guzzle5)....


Missed Source Maps for a component with a simple script block

Describe the bug
I have the start project.
With one change: I added   build: { sourcemap: true } in the config.
After building it has missed source maps for App.vue file.
Here is:

While in dev it is showed:

App.vue file script block has just only an import:
<script setup>
import HelloWorld from './components/HelloWorld.vue'
</script>

If I will add, for example:
console.log(123);

It starts works as expected:

Reproduction
yarn create @vitejs/app my-vue-app --template vue
and use the follow config (in vite.config.js):
export default defineConfig({
  plugins: [vue()],
  build: {
    sourcemap: true
  }
})

System Info
Node: 15.9.0
@vitejs/plugin-vue: 1.2.1
vite: 2.1.5

Before submitting the issue, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the docs.
[x] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[x] Provide a description in this issue that describes the bug.
[x] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/vue-next instead.
[x] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.

I'm not the most familiar with sourcemaps, but I can see the same in the latest Vite Vue template too.

I don't think this is a bug. None of the mapping can be mapped to the output bundle. So it won't be included in the sourcemap.
source-map-visualization


[nrf fromtree] Bluetooth: Mesh: Health_cli: Fix async cmd rsp

Swaps response scheme for Fault_get & Fault_clear command so that it
aligns with description of the documentation.
Signed-off-by: Anders Storrø<EMAIL_ADDRESS>(cherry picked from 348d9771241bf3eaade1a4cd69ea6009ba334346)
Signed-off-by: Alperen Sener<EMAIL_ADDRESS>

reopen with two commits needed