[SUPPORT] Data loss due to incorrect selection of log file during compaction

Dear community:
I found an issue while using hudi, which is that there may be data loss when compacting incremental data.Through investigation, it is found that the problem is introduced by the following bugfix：[[HUDI-6758]] Detecting and skipping Spurious log blocks with MOR reads (#9545).After testing, it was found that choosing to speculate on the files generated by execution seems to have a probability of causing data loss.
Expected behavior
While retaining this optimization, it is also necessary to ensure that the selected files are always correct to ensure the accuracy of the data.
Environment Description

Hudi version : 0.14.0

Additional context
The main logic that led to this issue is as follows：

I temporarily avoided this issue by retaining all log files with the same number of blocks. I would like to know if there is a better way to solve this problem.

cc @nsivabalan for taking care of this issue~

More detailed information about this case：
There are 28872 records of commited log file.

The number of merged records is 20400

By the compaction plan file, it can be seen that this write operation creates 2 log files in this bucket

The number of merged records after deleting the speculated log file is correct

the speculative task was killed

thanks, will take a look.

hey @Ytimetravel : Do you have pipelines where spark speculative execution is disabled. Trying to gauge the burst radius of the impact. Do we see it even w/o speculative execution. Or only when speculative execution.

And do you happened to have the data intact. We can enable debug logging and see exactly whats going wrong. If you have the data, I can sync up w/ you directly https://join.slack.com/t/apache-hudi/shared_invite/zt-20r833rxh-627NWYDUyR8jRtMa2mZ~gg  my handle is "shivNarayan". Feel free to ping me directly. I would like to get to the bottom of this asap.
since its due to spark speculative execution, it might be tough to reproduce by ourselves.

One suspicion I have is, when there are two sequence of log blocks where both have similar number of log blocks, we had to choose one of them. but we can't deterministically say which sequence is good one, we choose a random one and chances that we chose a wrong sequence.
But curious how could that end up in data loss though.

So to summarize the issue,
for the file group of interest, if not for spark speculative execution, only one log file(file ending w/ 24780) should have been added.
but w/ spark speculative execution, we found two log files(file ending w/ 24780 and file ending w/ 25536) were added.
The compaction plan shows two log files.
From the reader side, hudi checks for log block sequences, and since both sequences have just 1 log file, hudi chooses one of them and there is no determinism here.
But I am curious how the 2nd log file(file ending w/ 25536) was fully formed w/o any corruptness.

Sorry about lot of follow up questions.
Can you tell us what storage scheme you are using. Partial write failures should not happen w/ S3 or other cloud stores. if it had been hdfs, we cold see partial write failures. We are trying to gauge if the 2nd log file was properly formed or was it corrupted due to partial write failure.
If you have a backup of the data, let us know. We can share some tool that can spit out info about the log files (valid log blocks, no of valid records, etc) and might help us in our triaging.

Hey, I wrote a tool that could help us spit out some meta info about our log blocks and records.
https://github.com/nsivabalan/hudi/tree/printAllVersionsOfRecordTool
here is the branch.
Can you help us run the tool and share us the output.
Its a spark submit command. Its going to log some info about the log files we are interested in.
sample command
./bin/spark-submit --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'  --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --class org.apache.hudi.utilities.PrintRecordsTool PATH_TO_BUNDLE/hudi-utilities-bundle_2.12-0.15.0-SNAPSHOT.jar --props /tmp/props.in --base-path /tmp/hudi_trips_mor/ --partition-path asia/india/chennai  --file-id c3ef010f-61ae-4aa3-a033-25b278da17c6-0  --base-instant-time<PHONE_NUMBER>2723362 --print-log-blocks-info

cat /tmp/props.in
hoodie.datasource.write.recordkey.field=uuid
hoodie.datasource.write.partitionpath.field=partitionpath
hoodie.datasource.write.precombine.field=ts

Ensure you set the right values for partition path, fileID and the base instant time.
This should help w/ our triaging

Thank you very much for following up on the issue and providing feedback. I will use the tool you provided to obtain some meta info about our log blocks and records, and will get back to you later~

@nsivabalan @Ytimetravel Another data loss case caused by the whole stage retry.
There are 4 cases that the task retry:

Task is slow, another speculation task is retried
The task failed and retry
The stage failed and retry
The executor failed and retry
For the third point, which stage retried. Task attempt number might be back to the original value.
Using attempt number to identify the block is not enough to handle this case. It might leads to wrong result to compare blocks size of each attempt no.
We might need to using stageAttemptNumber and AttemptNumber to identify it, or other solution. WDYT?

@Ytimetravel Did you got a chance to run that tool and collect meta info about our log blocks and records.

@ad1happy2go I'm sorry for just seeing your message, I didn't succeed in using that tool.

https://github.com/apache/hudi/pull/10922

can we close out the issue


Encoder not working as expected

Hello, love your libraries for micropython. I've been testing the encoder only mode for micro-gui but had a lot of issues with getting my encoders to work.
I also tried using your encoder library from another repo to test printing the deltas to see if I was getting the right values and it looked like that, so I added a print for the delta also in the ugui.py file on the encoder callback and saw that the callback was not getting called for every dent I was turning (encoder set as 1) but a lot less of that, while the other encoder class of yours has the division ratio at 1 and register every dent perfectly.
Here's my hardware setup:
from machine import Pin, SoftI2C, freq
import gc
from ssd1306 import SSD1306_I2C as SSD
# Create and export an SSD instance
scl_pin = Pin(4)
sda_pin = Pin(0)

# Create an I2C object using the specified pins
i2c = SoftI2C(scl=scl_pin, sda=sda_pin)
gc.collect()  # Precaution before instantiating framebuf
ssd = SSD(128, 64, i2c)

gc.collect()
from gui.core.ugui import Display, quiet
# quiet()
# Create and export a Display instance
# Define control buttons
enc_clk = Pin(32, Pin.IN, Pin.PULL_UP)
enc_dt = Pin(33, Pin.IN, Pin.PULL_UP)
enc_btn = Pin(27, Pin.IN, Pin.PULL_UP)

display = Display(ssd, nxt=enc_clk, sel=enc_btn, prev=enc_dt, incr=False, encoder=1)

And here the simple script that uses the encoder class just to check the delta
def cb(pos, delta):
    print(pos, delta)

def prova():
    print('Running encoder test. Press ctrl-c to teminate.')
    enc = Encoder(px, py, v=0, vmin=None, vmax=None, callback=cb)

    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print('Interrupted')
    finally:
        asyncio.new_event_loop()

prova()

I the encoder behaved randomly in both cases I would think it's my encoder issue but doesn't look like that.

I'm not sure what I can do here. Your setup is correct, if it doesn't work I would suspect some kind of hardware issue.


Setup demo CI flow for mor1kx

The flow should include:

Building of reference ELF files
Running of ELF file simulation with FuseSoC
Reporting of results

@wallento is also going to provide TAP reporting in FuseSoC. If it's done, we will need to write converter for xUnit plugin

I would recommend that we use the "official" or1k test cases https://github.com/openrisc/or1k-tests/tree/master/native/or1k for this.
They also all output 0x8000000d (good) or 0xbaaaaaad to signal test status. For these tests I think it would make more sense to convert that to xUnit.

@olofk makes sense.
So ideally all tests should be compilable via https://github.com/openrisc/or1k-tests/blob/master/native/Makefile if I setup the proper environment

Correct. Some of them fails in the current state, so I recommend running them with a timeout. Can't remember a good timeout value now unfortunately.
So the complete fusesoc command would be `fusesoc sim mor1kx-generic --elf-load=$file --timeout=$some_timeout_value
Might want to add --trace_enable also to get the CPU traces written to mor1kx-{general,trace}.log
I guess Jenkins can also handle timeouts, but if we do it with FuseSoC it will be independent of the speed of the build machine, since the timeout is in simulated clock cycles


fix doc links

Shall we keep these doc updates in the 5.0-release branch or in master?

The layer source code links in API reference are also broken.


[release-v1.16] Update golang.org/x/crypto to 0.31.0

Addresses https://github.com/advisories/GHSA-v778-237x-gjrc

/retest

/approve

/approve
/lgtm

/test 417-e2e-oncluster-test

/retest-required


Fix some lint warnings

I followed https://adamj.eu/tech/2019/09/12/how-i-import-pythons-datetime-module/ to fix the datetime warning.
Is there any interest in adding a linter and/or formatter to the CI?

Is there any interest in adding a linter and/or formatter to the CI?

Yes, it would be nice to add both to have a more uniform code and avoid some silly issues such as the one you fixed in git.py.


dbt: Start using dbt-cratedb2

About
Validate dbt-cratedb2, the dbt adapter for CrateDB.
Details
This effectively means fragments of the SQL macros provided per Using dbt with CrateDB are being slotted into dbt-cratedb2 now, so they can be removed here.

https://github.com/crate-workbench/dbt-cratedb2/pull/2

/cc @surister, @kneth

Hi. Because most recent amendments to this patch effectively removed all project-specific macro overrides, thus completely relying on dbt-cratedb2, I guess it will be good to merge without much ado.
I am keeping it open for a bit longer, to give @hlcianfagna the chance to share any objections on relevant inline comments I made above.


Signing API redesign

The current signing API has several problems that make it difficult to use correctly. It's probably the most complex of commonly-used APIs and there are multiple issues people have with it.
I propose a redesign that focuses on removing the complexity by reducing possible errors, simplifying input arguments, and providing convenience methods for common script templates.
Proposed API (roughly)
pub struct SighashCache<T: AsTransaction, Prevouts> {
    // internal abstraction helper, implements `AsExtendedTransaction` and statically proves prevouts.len() == inputs.len()
    tx: ExternallyExtendedTransaction<T, Prevouts>,
    cache: Cache,
}

impl<T: AsExtendedTransaction> SighashCache<T, ()> {
    pub fn new(tx: T) -> Self { /* ... */ }

    pub fn sign_single_taproot_key_spend(index: usize, prevout: &impl AsTxOut, sighash: NonSingleSighashType, signer: Sign</* ... */>) -> Result<(), InputIndexError>;
}

impl<Q: AsOutput, T: AsTransaction, P: AsRef<[Q]>> {
    // notice we require all - sighash single is handled separately
    pub fn with_all_prevouts(tx: T, prevouts: P) -> Result<Self, PrevoutsError> { /* ... */ }
}

impl<T, P> SighashCache where ExternallyExtendedTransaction<T, P>: AsExtendedTransaction {
    // might be possible to just use iterator, I'm not sure about lifetimes
    pub fn sign_all(&mut self, cb: FnMut(usize, Input<'_, T, P>));
}

/// Signing handle
pub struct Input<'a, T, P> { /* ... */ }

/// This is same as in #2347 with additional methods
impl<'a, T, P> Input<'a, T, P> where /* ... */ {
    // Signs *and updates* the transaction. We'll need some funny generics for PSBT to put the signatures into appropriate fields
    // look, there's no error!
    pub fn sign_non_single_taproot_key_spend(&mut self, sighash: NonSingleSighashType, signer: impl Sign<ExternallyExtendedTransaction<T, P>::Extra>) { /* ... */ }
    // analogous to above, have fns for tr script spend, segwit v0 and legacy
    // This is mainly to facilitate async signers
    pub fn taproot_non_single_script_spend_hash(&mut self, spend: TaprootScriptSpend<()>) -> TaprootScriptSpend<Message>;
}

// probably more generics to support non-p2tr
pub trait Signer<Extra> {
    fn sign(self, message: Message, extra: Extra) -> Signature;
}

// yep, this impl is allowed because FnOnce is special
impl<E, T: FnOnce(Message, E) -> Signature> Sign<E> { /* ... */ }

impl<E: AsRef<bip32::Path>> Sign for &'_ XPriv { /* impls by deriving the key and using it to sign */ }

// simplified; needs to support references
pub struct TaprootScriptSpend<S> {
    control_block: ControlBlock,
    annex: Option<Annex>,
    leaf_script: ScriptBuf,
    sighash: TaprootSighash,
    signing_data: S
}

// needs to somehow inorporate sighash type, didn't figure out how; also could have a builder
impl TaprootScriptSpend<()> {
    pub fn new(sighash_type: NonSingleSighashType), cb: ControlBlock, annex: Option<Annex>, script: Script<Tapscript> /* notice tagged type */) -> Self { /* ... */ }
}

impl TaprootScriptSpend<Message> {
    // people who have an isolated signer that communicates using async can first call this, then set_signature
    // everyone else would use a method on `Input` directly and not bother with this type
    pub fn message(&self) -> Message { /* ... */ }
    pub fn set_signature(&mut self, signature: schnorr::Signature) -> TaprootScriptSpend<schnorr::Signature> { /* ... */ }
}

impl TaprootScriptSpend<schnorr::Signature> {
    // efficiently constructs witness
    // primarily used internally but users can also call this
    pub fn to_witness(&self) -> Witness { /* ... */ }
}

// statically proves SIGHASH_SINGLE is not used.
enum NonSingleSighashType {
    Default,
    All,
    None
    AllPlusAnyoneCanPay,
    NonePlusAnyoneCanPay,
}

Usage example
The simplest option: sign all inputs, which are taproot with default sighash using xpub and extended transaction.
SighashCache::new(&mut extended_tx).sign_all(|input| {
    input.sign_non_single_taproot_key(Default::default(), &xpriv);
})

One of the most complex use cases: Taproot script spend with async signer:
// hypothetical iterator; we can also use `while let` and do it without iterator
for input in SighashCache::new(&mut extended_tx).inputs() {
    let spend = TaprootScriptSpend::new(Default::default(), control_block, None, script);
    let spend = input.non_single_taproot_script_spend_hash(script_spend);
    let signature = call_external_signer(spend.message()).await?;
    let spend = spend.set_signature(signature);
    *input.witness_mut() = spend.to_witness();
})

I also have some super-crazy ideas about compile-time processed miniscript that would create appropriate structs for you so that you don't have to worry about control block etc but that's for another day. :)

It's hard for me to tell what's going on without seeing how we handle non-Taproot inputs. What do the sign_all or inputs iterators yield in this case?
Also your second impl block is missing a type so we can't tell what Self is.

It's hard for me to tell what's going on without seeing how we handle non-Taproot inputs.

Similarly, the Input handle will have methods for signing them. Which reminds me I forgot about them in #2347

Also your second impl block is missing a type so we can't tell what Self is.

Fixed.

So, the Input handle has methods for producing both Taproot and non-Taproot signatures? How does the user ensure that they're using the correct method?

How does the user ensure that they're using the correct method?

Just like now, they have to know it out of band. We might add even more machinery to put that into extra but maybe it's too much. I can try to think about it though.

Ok. So, concept ACK adding these iterator methods which eliminate annoying IndexOutOfRange errors (and having the associated Input type to facilitate this). I kinda think Input should be an enum over all the different input types so that users can match/destructure and then be forced to use the correct signing methods and signature types.
I'm uncertain about having some signing methods update the transaction ... you can do this with p2pk(h), p2wpkh and p2tr keyspends, but not for any other type of spends, so I think the resulting API is going to feel inconsistent.

This might be as good a time as ever to introduce testing against bitcoind in CI (regtest), I rekon we should have tests that create various chains of transactions, and verify that we did it all correctly. This would serve as good documentation as well as testing.
For example

Take some inputs and create a 2-of-3 multisig output
Sign the multisig output and send it to some other set of outputs (spend, change etc.)
Do key path spend and script path spend of outputs created in (2)

Try to cove various output types segwit v0 and taproot.

This might be as good a time as ever to introduce testing against bitcoind in CI (regtest), I rekon we should have tests that create various chains of transactions, and verify that we did it all correctly. This would serve as good documentation as well as testing.

This could be quite difficult because it would create a circularity between the bitcoind crate and rust-bitcoin. (Though we could skip this and hack together a "spawn a bitcoind" script in CI.)

Good point, we could put the testing in the new psbt crate? It is going to be pretty tightly coupled with rust-bitcoin anyways and I was considering maintaining a constant dev branch to track rust-bitcoin master and test stuff as we do it.

In BitcoinJS we had a hard time designing the signing API in regards to async/await.
Many users had use cases for signing on hardware wallets without blocking the event loop (JS).
It seems like the async use case is pretty strong for signing in particular, and I wonder how wallets coded in Rust would handle that with this API? Spawning threads seems to be the only option for now.
Would something using the async_trait crate be acceptable here? Or would we want to wait until we bump MSRV above 1.75 or some other stable version that adds enough functionality to async traits (dynamic dispatch is a big one I think it's missing rn)
Thoughts?

it would create a circularity between the bitcoind crate and rust-bitcoin

Not after we split primitives out - bitcoincore-rpc will only depend on primitives, and sighash will dev-depend on primitives (I'm thinking that bitcoin-sighash crate could be appropriate.)
Further even if we wanted to do funny things like testing serialization against bitcoind, it's a dev-dependency which might be allowed in integration tests and we can depend on an older version anyway, which is allowed. Maybe we'd have to bump the version to 1.x.$(last + 1)-dev right after release but that may be a good thing.

I think that the signing logic should be moved to HWW entirely

In the ideal situation, that would be best. But I can tell from personal experience that some of these HSM vendors really make it hard to run custom binaries on their machines, and some companies require certain HSMs for regulation compliance etc.
Obviously, if you're a Trezor or a Ledger and you're building your own boards from the ground up then I fully agree.
Maybe async is something for a potential 2.0 many years down the line once stable Rust figures out async traits a bit more and that becomes an easily MSRV-able version... but I was just bringing it up as something to possibly consider when designing the non-async API. (ie. making the API easier to expand into having async counterparts later on etc.)

and some companies require certain HSMs for regulation compliance

Not sure if I should laugh at them or curse them... :laughing: but sure, I can have no problem if people are bypassing that crap and I'll be happy to help them.

Maybe async is something for a potential 2.0 many years down the line once stable Rust figures out async traits

Not really, this is perfectly achievable in 1.0 and something we actually aim to provide.

Not after we split primitives out - bitcoincore-rpc will only depend on primitives, and sighash will dev-depend on primitives (I'm thinking that bitcoin-sighash crate could be appropriate.)

We'd depend on bitcoind whose maintainer doesn't go AWOL for months at a time, not bitcoincore-rpc which has stalled updates to the last 5 or 6 rust-bitcoin releases for everyone who depends on it :). But yes, your point is the same.
I think we can evade a literal dependency circle by using a separate-crate integration test, as you hint at. This would work even without crate smashing. But we still have a "conceptual" circularity .. where if we update rust-bitcoin, our integration tests will be broken until bitcoind updates, which it can't do until we release rust-bitcoin.
Maybe crate smashing will get us out of this. I hope so. I think we're still uncertain whether bitcoin-primitives can actually work as an independent crate or whether we run into too many coherency issues.
@junderw

In the ideal situation, that would be best. But I can tell from personal experience that some of these HSM vendors really make it hard to run custom binaries on their machines, and some companies require certain HSMs for regulation compliance etc.

Lol!! If HSMs are signing blind hashes to comply with some sort of regulation that requires HSMs but allows that, they can write their own APIs for it.
But we should move this discussion to the async discussion #2407 that you opened (or Kix's isssue #2409 if there are specific API pain points we need to address).

We'd depend on bitcoind whose maintainer doesn't go AWOL for months at a time, not bitcoincore-rpc

bitcoind depends on bitcoincore-rpc, not directly on bitcoin.

"conceptual" circularity .. where if we update rust-bitcoin, our integration tests will be broken until bitcoind updates, which it can't do until we release rust-bitcoin.

This is only a problem if we break the API which we will eventually stop doing.

not bitcoincore-rpc which has stalled updates to the last 5 or 6 rust-bitcoin releases for everyone who depends on it

I got added as maintainer to the rust-bitcoincore-rpc repo as well as publish perms on crates.io - so we should be right during releases from now on. (Assuming I don't go AWOL too ;)

Oh, excellent :) So maybe this fear is not an issue.

@Kixunil, Still parsing through the post, but this is a bit heavy for me after months of absence :) . One point I think we should investigate if we really need the efficiency gains from SigHashCache.
I am in part responsible for suggesting/implementing here, but I think we should revisit now if the efficiency gains are worth the complexity/API mess. We base this primarily from bitcoin core verification logic.

Signing is already an expensive crypto EC operation. Signing schnorr takes 40-50 micros, hashes take 50ns; we are not saving a lot by caching things here. It makes sense for bitcoin core, because they use caching for verification for all inputs in the same transaction and verification is a frequent operation that needs to be fast. Signing on the other hand can be slow, infrequent, often is clubbed up with other expensive steps like bip32 derivations and waiting for network calls to external signer.
We still use SIghashCache internally for APIs that sign multiple inputs at once. However, the APIs that want fine grain control over APIs they pay the extra cost of hashing the same thing again.

I think one could still conceivably use it for verification (e.g. in Firefish we do verify signatures but we don't need it to be crazy fast).  Also I don't think it's cache that's causing most API issues. It's the inputs to the functions being dynamically checked, so there's a bunch of possible errors and then whatever happens after signing - putting the data into witness etc.
We've improved some things but there's more stuff to do.

@Kixunil
Most people I have seen in workshops (maybe @tcharding can chime in here) struggle with this additional magical parameter SigHashCache that pops up in between and unintuitive at first.

The optimization will still work for sign_all APIs that sign/verify multiple inputs in the single function call. I feel we should be able to achieve a simple and efficient API for most wallets that mostly want sign_all method. The advanced users can deal with cache APIs.

Most people I have seen in workshops (maybe @tcharding can chime in here) struggle with this additional magical parameter SigHashCache that pops up in between and unintuitive at first.

Interesting! Maybe if we just had a method on transaction .signing_builder() it would work? It does act a bit like a builder once you have it. :) I suspect people mostly search for sign method on Transaction.

The optimization will still work for sign_all APIs that sign/verify multiple inputs in the single function call. I feel we should be able to achieve a simple and efficient API for most wallets that mostly want sign_all method.

Oh, good point! That's pretty much what SighashCache::sign_all does and we could simply have:
impl Transaction {
    fn sign_all(&mut self, prevouts: &[impl AsRef<TxOut>], f: impl FnMut(SigningInput)) {
        SighashCache::with_prevouts(self, prevouts).sign_all(f)
    }
}

Which makes a lot of sense.

I like the idea of using "builder" terminology rather than "cache" terminology, and hanging it off of Transaction rather than making the user construct one and passing in a transaction reference. It's super unintuitive that signing would need a "cache" and that you should start the signing process with such a thing.
We should, of course, make sure the advanced process is still possible and allow creating the cache with a generic T: Borrow etc. But it shouldn't be foregrounded.

make sure the advanced process is still possible and allow creating the cache with a generic T: Borrow

Of course! And we can obviously have a method on ExtendedTransaction too which will use the other method so it uses its own prevouts.
One issue though is that inherent method interferes with stabilization of Transaction before SighashCache (do we actually want to rename it to SigningBuilder?) I think we could just wait with adding it until SighashCache stabilizes at which point we can just invert the dependencies. This is not a problem since we use a trait anyway.


6.2 - Stackblitz examples don't link

Bug, feature request, or proposal:
On angular material website, the stackblitz examples are inaccesible.  This must have just happened, a few hours ago it showed 6.1 and now is at 6.2.
What is the expected behavior?
Stackblitz example is shown.
What is the current behavior?
This JSON returns:  {"error":"UNSATISFIED_RANGE","data":{"name":"@angular/cdk","range":"6.2.0"}}

I'm having the same issue.

There's another similar bug at https://github.com/stackblitz/core/issues/324, but it was last updated April 16th.  There was no solution there either.  Just a couple of "it works for me."  Which isn't helpful.  I get the same error as @kristofvdj88:
`
{"error":"UNSATISFIED_RANGE","data":{"name":"@angular/cdk","range":"6.2.0"}}
`
I'm trying to visit https://run.stackblitz.com/api/angular/v1?file=app%2Fdialog-overview-example.ts from https://material.angular.io/components/dialog/examples.
I agree that it might be the version number.  I've tried in chrome, ff, and edge with the same error message.  Is there a way to put the range/version number in the url?  That might be a good workaround until it is fixed.

Same here
{"error":"UNSATISFIED_RANGE","data":{"name":"@angular/cdk","range":"6.2.0"}}

What you can do for a temp fix is switch to 5.2.5 on material and then view. Not that big of  a change at all from 5.2.5 to 6.2.0

@EricSimons FYI

Yeah, this is npm throttling our access to the registry. Seems to be happening a lot more recently. We've got a fix we're working on to solve it permanently that should be out sometime next week, till then we're manually trigger resyncs when these issues arise.
I just resync @angular/cdk - can y'all confirm this is working again on your end? Sorry about the confusion! 😓

@EricSimons works for me now, thanks!

works fine now thanks.


After #save, object may not fully represent the row

Make sure after #save, if we reused the object, the fields 100% represent the row in the DB. I will confirm this. Thanks

I think I can confirm this. I will have a deeper look into the actual model code to double check it.

Just fixed this.


catalyst.api.order_percent(self, asset, percent, limit_price=None, stop_price=None, style=None)

Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
Environment

Operating System: (macOS High Sierra)
Python Version: $ Python 3.6.4
Python Bitness: $ [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-9<IP_ADDRESS>)] on darwin
How did you install Catalyst: (pip)
Python packages: ``

Now that you know a little about me, let me tell you about the issue I am
having:
Description of Issue

What did you expect to happen?
I was expecting that 100% of my portfolio will be used if I specify percent = 1

What happened instead?
Each time it takes around 91-97%

Here is how you can reproduce this issue on your machine:
Reproduction Steps

call the function

...
What steps have you taken to resolve this already?
...
Anything else?
...
Sincerely,
$ whoami

Hi @AndrejLischishin ,
What mode are you referring to?

Hi @lenak25,
backtest


Recommended way to access semantic variables in other less files

Hi,
Is there a recommended way to access the semantic variables (color names, dimensions, etc.) in our own less files? It's really not obvious as to what needs to be imported first.
I realise that I can add CSS to the site override file, but I don't wish to do that. My project is divided into packages and I wish to keep the CSS for each package localized to that package. At the moment I'm having to do this at the top of each file:
@semanticFolder: "../../../client/semantic-ui/";
@import<EMAIL_ADDRESS>@import<EMAIL_ADDRESS>@import<EMAIL_ADDRESS>
@import<EMAIL_ADDRESS>...+ more for other variables that I need access to...

Is there a better way?

:+1:

I'm also having the same doubt and the documentation is not quite clear to me as to how achieve this.


Menu icon only displaying on analytics page

Hi @tomdyson,
One tiny thing, the fontawesome icon for analytics in the menu is only displaying on the actual analytics dashboard.
All of the other menu pages show the title without the icon.

Yes, I spotted this belatedly - this was the motivation for https://github.com/torchbox/wagtail/pull/2276. Once that's in Wagtail 1.4 I'll switch the hook so it shows properly for all pages.

I also noticed this. Oddly enough, the icon does appear when I'm on a custom admin page that I defined myself. I wonder if that page counts as an editor page?

Fixed via alexgleason/wagtailfontawesome#1
Make sure you're using wagtailfontawesome version >= 1.0.2 and Wagtail >= 1.4 in order for it to work.

Hmmm, perhaps this is just my own ignorance at work, but when I updated my dev machine's copy of wagtailfontawesome to 1.0.2, pip did quite a bit more than just updating that one package:
$ pip install --upgrade wagtailfontawesome
Collecting wagtailfontawesome
  Downloading wagtailfontawesome-1.0.2-py2.py3-none-any.whl (516kB)
    100% |████████████████████████████████| 522kB 588kB/s
Collecting wagtail>=1.4.0 (from wagtailfontawesome)
  Downloading wagtail-1.4.1-py2.py3-none-any.whl (6.5MB)
    100% |████████████████████████████████| 6.5MB 196kB/s
Collecting Django>=1.7.1 (from wagtailfontawesome)
  Downloading Django-1.9.4-py2.py3-none-any.whl (6.6MB)
    100% |████████████████████████████████| 6.6MB 194kB/s
Collecting djangorestframework>=3.1.3 (from wagtail>=1.4.0->wagtailfontawesome)
  Downloading djangorestframework-3.3.3-py2.py3-none-any.whl (662kB)
    100% |████████████████████████████████| 665kB 1.9MB/s
Requirement already up-to-date: django-taggit>=0.17.5 in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from wagtail>=1.4.0->wagtailfontawesome)
Requirement already up-to-date: Unidecode>=0.04.14 in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from wagtail>=1.4.0->wagtailfontawesome)
Requirement already up-to-date: beautifulsoup4>=4.3.2 in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from wagtail>=1.4.0->wagtailfontawesome)
Collecting django-treebeard<5.0,>=3.0 (from wagtail>=1.4.0->wagtailfontawesome)
  Downloading django-treebeard-4.0.tar.gz (92kB)
    100% |████████████████████████████████| 102kB 8.4MB/s
Requirement already up-to-date: html5lib<1,>=0.999 in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from wagtail>=1.4.0->wagtailfontawesome)
Collecting Willow<0.4,>=0.3b4 (from wagtail>=1.4.0->wagtailfontawesome)
  Downloading Willow-0.3-py2.py3-none-any.whl (110kB)
    100% |████████████████████████████████| 112kB 9.7MB/s
Requirement already up-to-date: Pillow>=2.6.1 in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from wagtail>=1.4.0->wagtailfontawesome)
Requirement already up-to-date: django-modelcluster<1.2,>=1.1 in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from wagtail>=1.4.0->wagtailfontawesome)
Requirement already up-to-date: six in /Users/rrollins/.pyenv/versions/wagtail-multitenant/lib/python2.7/site-packages (from html5lib<1,>=0.999->wagtail>=1.4.0->wagtailfontawesome)
Collecting pytz>=2015.2 (from django-modelcluster<1.2,>=1.1->wagtail>=1.4.0->wagtailfontawesome)
  Downloading pytz-2016.1-py2.py3-none-any.whl (476kB)
    100% |████████████████████████████████| 481kB 3.0MB/s
Building wheels for collected packages: django-treebeard
  Running setup.py bdist_wheel for django-treebeard ... done
  Stored in directory: /Users/rrollins/Library/Caches/pip/wheels/4d/86/56/0eca65a0a432037604df3f4a03d81ae97b767b5a0e3bb6832f
Successfully built django-treebeard
Installing collected packages: djangorestframework, Django, django-treebeard, Willow, wagtail, wagtailfontawesome, pytz
  Found existing installation: djangorestframework 3.3.2
    Uninstalling djangorestframework-3.3.2:
      Successfully uninstalled djangorestframework-3.3.2
  Found existing installation: Django 1.9.3
    Uninstalling Django-1.9.3:
      Successfully uninstalled Django-1.9.3
  Found existing installation: django-treebeard 3.0
    Uninstalling django-treebeard-3.0:
      Successfully uninstalled django-treebeard-3.0
  Found existing installation: Willow 0.2.2
    Uninstalling Willow-0.2.2:
      Successfully uninstalled Willow-0.2.2
  Found existing installation: wagtail 1.4a0
    Uninstalling wagtail-1.4a0:
      Successfully uninstalled wagtail-1.4a0
  Found existing installation: wagtailfontawesome 1.0.1
    Uninstalling wagtailfontawesome-1.0.1:
      Successfully uninstalled wagtailfontawesome-1.0.1
  Found existing installation: pytz 2015.7
    Uninstalling pytz-2015.7:
      Successfully uninstalled pytz-2015.7
Successfully installed Django-1.9.4 Willow-0.3 django-treebeard-4.0 djangorestframework-3.3.3 pytz-2016.1 wagtail-1.4a0 wagtailfontawesome-1.0.2

Is this my fault for using pip wrong, or is wagtailfontawesome's dependency setup at fault? I'm especially surprised by that django update, as I don't think 1.9.4 specifically would actually be required, so why didn't pip just leave 1.9.3 in place?

@coredumperror I think that's a pip thing. I've done the same, with other packages, followed by a "no no no no no!!" It's pretty annoying. I don't know how to actually do it properly.
But if you edit the requirements.txt for your project and set wagtailfontawesome==1.0.2, you can just pip install -r requirements.txt and it handles it correctly.
I think the --upgrade command doesn't do what we want it to do, here.

Ah, I hadn't thought of explicitly specifying the version number for wagtailfontawesome. Thanks!

Yeah, pip install --upgrade will upgrade all the installed packages, rather than only upgrading any named packages. You can pip install wagtailfontawesome==1.0.2 wagtail==1.4.1, which will only install the named versions, and upgrade any dependencies that are no longer satisfied.

Thanks for the clarification. That is so bad, IMO

Agreed, especially when you consider that pip itself tells you to use pip install --upgrade <package_name> for this purpose.

Oh wait no, I was slightly wrong. Doing a pip install --upgrade package will fetch and install the latest version of package. It will not upgrade all of your installed packages. However, if package depends on wagtail>=1.3, even if you already satisfy that dependency by having wagtail==1.3 installed, it will look for an updated wagtail package to install in order to better satisfy the dependency from package. Still an annoying way to work, but not quite as horrible as I initially said.
The best way to fix this is to do as @alexgleason suggested, and update your requirements for your project, and then install by running pip install -r requirements.txt.

I suggest using class='icon icon-site', which is in the initial setup. Then icon is inserted. I think that the formal '' icon-site "within the meaning fits.

Closing this as fixed in wagtailfontawesome version >= 1.0.2 and Wagtail >= 1.4


Indicate on Readme.md which CCO-using Ontologies won't have Public URLs

The Readme.md has an attractive list of domain-specific CCO-using ontologies. However, all but the Emotion ontology do not offer a URL. Further, a Google search for the ontologies by name and visits to ontology repositories yield disappointing misses.
If these cited CCO-using ontologies are never going to be public (because of the sensitivity of their concepts?) then can they be moved to a sub-heading of the Readme.md like "Known Usages but Private to their Creators"?

Yes, will do.

Addressed with d4ea051.
@lonniev Is this satisfactory?


bundle name is unknown

Hey, I really like where this project is going. Thanks for doing it.
I want to ask if there's any way to deal with unknown bundle file names because I'm using webpack and my bundle files are hashes.
I also have another constraint: I currently generate 2 JS files - vendor and app specific.
Also, my test script is custom and has other commands how would I configure it to work with bundlesize?
Thanks

Yes, bundlesize uses glob inside, so you can use
path: "dist/*.js"

or something more specific, like "dist/home-*.js"

Although I don't know how well will it play with github status, worth a try though. Let me know if it doesn't work as expected.

Also, my test script is custom and has other commands how would I configure it to work with bundlesize?

You just need to run bundlesize as a shell command. If you can show me your code, I'll be helpful

Maybe I could run it as a circleci custom step?

Maybe I could run it as a circleci custom step?

Yep, that totally works.

If you want to run it programmatically through your code, you can probably just do this:
require('bundlesize')

This should work as long as you run it from in the root directory.

Oh I guess I got it now..
It's basically a tool that checks file size (after gzip) against a predefined threshold. If it exceeds the threshold it exits with 1.
I think that in my case I'd have to do it within my Dockerfile because that's the only place I actually build the front-end application.
Thanks!

Yep


Unknown node Options

The <svelte:options> does not have handling and causes prettier-plugin-svelte to throw an error.
The options (and all other <svelte:*> elements should be added (or check if they already exist) to the printer.
I'll try to submit a PR for this this afternoon. If someone wants to tackle it instead, go for it.

Going to tackle this now


Point data from csv file not visible in data preview (following steps from Mapbox Studio Source Quickstart)

Hello,
I am trying to create a webmap of point data on U.S. universities. I followed the tutorial at https://www.mapbox.com/mapbox-studio/source-quickstart/ with no trouble using the linked Earthquake data. However when I try to follow the same steps using my own csv (which has point data with x/y coordinates in columns headed "latitude" and "longitude"-- these are the third and fourth columns-- do they need to be the first and second?), the points do not appear in the data preview window, nor do they appear when I upload the source, associate it with a style, and attempt to style the points in a similar way as done in the tutorial. The column names do appear in the field descriptions area for the layer.
The csv is one that I've used in QGIS and ArcGIS with no problem- it shows about 1000 points of university locations in the US.
The MapID for the source layer is stevemcf.d8b362ba
Some screenshots are below.
Thanks in advance for any suggestions!
Steve

@stevemcf - What happens when you click the 'Zoom to extent' icon?

@amyleew - it's same as in the pictures above-- a black rectangle.

Add this data's Configure panel screenshot @stevemcf

Here it is @amyleew

@stevemcf Is it possible to share your csv file that we can have a look?
If you want to share privately send to<EMAIL_ADDRESS>

@stevemcf another thing:
Are you using the latest version of Mapbox Studio?
The path to your csv looks a bit strange with forward / and backward \ slashes.
This shouldn't happen with the latest 0.2.7.
Try removing C: and replacing \ with /.
Does this help?

@BergWerkGIS thanks! I'll send the csv along to you just now. I assume I have the latest version as I just downloaded it two days ago. I just tried taking out the c: and switching the slash, but it didn't seem to change anything. Might it work better if the file wasn't parked in a google docs folder?

@stevemcf thanks, got it.
Looks the same (black in black) on my machine.
Will investigate.

@stevemcf
If you remove the colum the_geom it works.
Mapbox Studio gets confused because of latitude/longitude and the_geom.
When Mapbox Studio loads the csv it also looks for a column that has geom in its name and prefers that one over lat/lng fields.
It expects to find a WKT string in the geom column, which isn't the case with your file, and cannot display any data.

Aha! Splendid! Thanks for sussing that one out Wilhelm. @BergWerkGIS
Best,
Steve


Resource parameters cannot be deleted

Description:
Resource parametes defined in API definition cannot be deleted. When the delete button is clicked, nothing happens.
Affected Product Version:
2.1.0-update2
Steps to reproduce:

Add a resource parameter
Click delete button of the paramater

Related Issues:

Fixed with https://github.com/wso2/carbon-apimgt/pull/4839


Pass function to list: possible PR?

I'm wondering if this has been considered and rejected already? I've searched thru the Issues and Pull Requests, but couldn't find anything that clearly addressed this. So, I will ask myself explicitly:
Would you accept a PR that allowed the list config option to have a function as a value?
In essence, I'm conceiving of a very simple alteration, where Awesomplete does little else than alter the set list(list) method (https://github.com/LeaVerou/awesomplete/blob/gh-pages/awesomplete.js#L117) to add one more control flow path:
else if (typeof list === "function") {
	this._list = list();
}

This would allow users to deal with AJAX dependent datasets in a simpler, more consistent way. Simply wrap your AJAX request in a thunk and pass that to list on initialization.

There is a discussion #16806 to make Bliss a dependency. Bliss relies on Promise. When (and if) we add Bliss as dependency we can just start using promises in Awesomplete too.

Promises are awesome, but do we need promises in Awesomplete? What for? I don't remember having anything particularly async in there (except events, for which promises are a bad idea).

I was thinking about Ajax example and this.list handling it.

Ah ok!

So the goal is to make handling async lists cleaner. Promises are certainly one way to do that. In that case, list could be an array, a comma-separated string, a selector string, an HTML node, or a Promise object.
The other general way to go about this would be to allow list to be a callback function, similar to how jQuery UI Autocomplete's source param works. In that case, a user could use Promises internally in their callback, or they could not. The callback option seems to me to be the most flexible—as it puts full control in the user's hands—but maybe that isn't what we want.

How would someone use Promises in their callback if Awesomplete expects the callback to return a list, synchronously?

I was thinking poorly when I wrote that. My current approach is to introduce a new configuration param (source) that can be a string (an API route to get back suggestions), an object (to be passed to Bliss.fetch), or a function.
The basic idea is that source is a mechanism for generating the filtered suggestions directly, rather than setting a list and a filter method. The current implementation has source taking precedence over the list. The best thing about this approach, however, is that it is totally backwards compatible with the 1.0 API.
I'm making these changes in my fork because my job requires the suggestion list to be generated server side.


Discovery endpoints should be available without login

Today I tried to include this backend in the openEO Hub, but the crawling process failed because all endpoints (except /) require login. I think that the discovery endpoints (/collections, /processes, /output_formats, /service_types) should be publicly available. Could you fix that so we can add this backend to the Hub? :)

Solved with https://github.com/Open-EO/openeo-openshift-driver/pull/9. Will close the issue when it is pushed to production


Suggestion

Could you add a version of "Flat Dimensions with Caves" with the bedrock ceiling in the nether please?

With the ceiling? You mean the nether ceiling?

With the ceiling? You mean the nether ceiling?

Yes please

One more question, would you want there to be a layer of like 3 blocks below the bedrock ceiling (like it is a reflection of the floor)

This is what I have right now. I capped it at 104, so it still looks cave-y (and retains base worldgen, which is the point of the flat dimension versions) and has flat tendencies. Do you think it would be better to just make it solid up top?

With the ceiling? You mean the nether ceiling?

Yes please

This is what I have right now. I capped it at 104, so it still looks cave-y (and retains base worldgen, which is the point of the flat dimension versions) and has flat tendencies. Do you think it would be better to just make it fully solid and flat up top (minus the carver caves, or should I limit those vertically too??)

I think you should take out the carver caves and just have the bedrock.

I think you should take out the carver caves and just have the bedrock.

If this is good, I'll add it to a separate branch on here.

That is great! Is there no way to take out the basalt dripping down?

That is great! Is there no way to take out the basalt dripping down?

Yeah it's possible to remove, you just didn't specify that you wanted that feature removed.
I'll be able to do that and then publish it tomorrow, I'm headed to bed right now in my time zone.

That is great! Is there no way to take out the basalt dripping down?

Yeah it's possible to remove, you just didn't specify that you wanted that feature removed.
I'll be able to do that and then publish it tomorrow, I'm headed to bed right now in my time zone.

 Thank you! Love this datapack

That is great! Is there no way to take out the basalt dripping down?

Yeah it's possible to remove, you just didn't specify that you wanted that feature removed.
I'll be able to do that and then publish it tomorrow, I'm headed to bed right now in my time zone.

 Thank you! Love this datapack

Alrighty, done.
https://github.com/Quidvio/Flattened-Dimensions/tree/FD-With-Caves-Roof-Edit


chore(release): 0.9.10

Do we decide to hide some values?

If the transaction is being processed, then the token will appear in the list of transactions of tokens to which it does not belong. or this is ok?

p.s. will continue my tests tomorrow

if I click on the 6 WTT token and view the details, I will see the following (the same on dev version):

Names tab:

Set name pointer error:

waiting for name to register:

Web wallet:
Create a new wallet - pass
Verify Seed phrase-pass
Change tokens/currency - pass
Send ae - pass
Receive ae - pass
Send FT - pass
Receive FT - pass
Balances view- wrong balance for some tokens
Send via scan QR - pass
AE/FT is displayed in Tx History (All/Sent/Received) - pass
Send Tip - pass
Claim tip -pass
Tx history - pass
Tx details - pass
Invite link: create - pass
top up -pass
claim back - pass
Register a name - pass
Set name pointer - pass but failed in some cases
Set as default name - pass
Extend - pass
Settings - pass
Networks - pass
The new name - make default button is absent. - the same on dev version.
Not all names have a Make default button, in many, it is missing. - the same on dev version.
Web wallet is not connected with superhero.com. - the same on dev version.
Buy AE - haven’t tested
During testing, I noticed that the balance of tokens is displayed incorrectly after sending tokens to another address, for example:
I had 142.7 ae - sent 1.1 = should be 141.6 ae — I have 141.59 ae in fact
I had 700 Davidyuk - sent 1.1 = should be  698,9 Dav —  I have 699 IN FACT
The same issue on dev version #1505

tip icon shifted on twitter:

Permission settings - not working on superhero.com
While the transaction of sending tokens was being processed, this pop-up constantly popped up:

FF/Chrome Extensions:
Create a new wallet - pass
Verify Seed phrase-pass
Change tokens/currency - pass
Send ae - pass
Receive ae - pass
Send FT - pass
Receive FT - pass
Balances view- wrong  balance for some tokens
Send via scan QR - pass
AE/FT is displayed in Tx History (All/Sent/Received) - pass
Send Tip - pass
Send Tip via Twitter tip button - pass
Claim tip - pass
Tx history -pass
Tx details - pass
Invite link: create - pass
top up -pass
claim back - pass
Register a name - pass
Set name pointer - pass
Set as default name - pass
Extend - pass
Settings - pass
Permission Settings - not working with superhero.com
Networks - pass

While the transaction of sending tokens was being processed, this pop-up constantly popped up:

fixed

I found the following problems:

modals and popups are displayed below the top nav bar.

When placing bid for .chain name it took more than 5 mins. to see my bid in the Bid History.

Set name pointer, when the name registration transaction is still pending:

if I click on the 6 WTT token and view the details, I will see the following (the same on dev version):

Names tab:

During testing, I noticed that the balance of tokens is displayed incorrectly after sending tokens to another address, for example:
I had 690.70 Davidyuk - sent 1.1 = should be 689,6 Dav — I have 689,5 IN FACT
The same issue on dev version https://github.com/aeternity/superhero-wallet/issues/1505
pop up in names/invite link is cropped:

Web wallet:
Create a new wallet - pass
Verify Seed phrase-pass
Change tokens/currency - pass
Send ae - pass
Receive ae - pass
Send FT - pass
Receive FT - pass
Balances view- wrong balance for some tokens
Send via scan QR - pass
AE/FT is displayed in Tx History (All/Sent/Received) - pass
Send Tip - pass
Claim tip -pass
Tx history - pass
Tx details - pass
Invite link: create - pass
top up -pass
claim back - pass
Register a name - pass
Set name pointer - pass but failed in some cases
Set as default name - pass
Extend - pass
Settings - pass
Networks - pass
The new name - make default button is absent. - the same on dev version.
Not all names have a Make default button, in many, it is missing. - the same on dev version.
Web wallet is not connected with superhero.com. - the same on dev version.
Buy AE - haven’t tested

tip icon shifted on twitter:

Permission settings - not working on superhero.com
FF/Chrome Extensions:
Create a new wallet - pass
Verify Seed phrase-pass
Change tokens/currency - pass
Send ae - pass
Receive ae - pass
Send FT - pass
Receive FT - pass
Balances view- wrong balance for some tokens
Send via scan QR - pass
AE/FT is displayed in Tx History (All/Sent/Received) - pass
Send Tip - pass
Send Tip via Twitter tip button - pass
Claim tip - pass
Tx history -pass
Tx details - pass
Invite link: create - pass
top up -pass
claim back - pass
Register a name - pass
Set name pointer - pass
Set as default name - pass
Extend - pass
Settings - pass
Permission Settings - not working with superhero.com
Networks - pass
after the name pointer is set, the make default button does not appear in some names, and they cannot be applied, they have only 2 buttons set pointer and autoextend.

approx icon:

Web wallet:

Create a new wallet - pass

Verify Seed phrase-pass

Change tokens/currency - pass

Send ae - pass

Receive ae - pass

Send FT - pass

Receive FT - pass

Balances view- pass

Send via scan QR - pass

AE/FT is displayed in Tx History (All/Sent/Received) - pass

Send Tip - pass

Claim tip -pass

Tx history - pass

Tx details - pass
Invite link: create - pass

top up -pass

claim back - pass

Register a name - pass

Set name pointer - pass

Set as default name - pass

Extend - pass

Settings - pass

Networks - pass
The new name - make default button is absent. - the same on dev version.
Not all names have a Make default button, in many, it is missing. - the same on dev version.
Web wallet is not connected with superhero.com. - the same on dev version.
Buy AE - haven’t tested
pending transactions view:

FF/Chrome Extensions:
Create a new wallet - pass
Verify Seed phrase-pass
Change tokens/currency - pass
Send ae - pass
Receive ae - pass
Send FT - pass
Receive FT - pass
Balances view- pass
Send via scan QR - pass
AE/FT is displayed in Tx History (All/Sent/Received) - pass
Send Tip - pass
Send Tip via Twitter tip button - pass
Claim tip - pass
Tx history -pass
Tx details - pass
Invite link: create - pass
top up -pass
claim back - pass
Register a name - pass
Set name pointer - pass
Set as default name - pass
Extend - pass
Settings - pass
Permission Settings - pass
Networks - pass
The balance of some tokens (Davidyuk) is updated with a long delay.
The names after registration appear in the wallet with a long delay


Status of LND / RPC - whether running/fully syncd or not

Background
In the lifecycle of iOS apps they can be running in the foreground, running in the background, or dismissed and terminated.  When in foreground or backgrounded and still running (not suspended), it would be nice to know if LND and RPC are still in good health and running.
I have set up timers running every 30 seconds to getinfo and to connect peers to notify if RPC and LND respectively are working well. Obviously it is expensive to run timers and to run them at frequent intervals makes the app a resource hog etc.
Ideally we would like to run one timer to learn of the LND status - whether syncd, running , not started yet etc.. Thus a rest endpoint would be a great add
Your environment

version of lnd - 0.5beta
which operating system (uname -a on *Nix) - iOS
version of btcd, bitcoind, or other backend - btcd

Expected behaviour
A way to know if LND has fully syncd and started up
Actual behaviour
No such rest endpoint

If you can hit the RPC server, then you know if lnd is active or not. The best call to hit is getinfo.

Hi @Roasbeef , even though getinfo works and returns results even indicating whether or not syncd with the blockchain, from my experience, it doesn't seem to be enough to know if LND is actually listening on 9735. When using neutrino there is a delay between when RPC is ready and when LND is listening on 9735. Many trials show the same result, that RPC is ahead of LND which appears to be catching up with neutrino filters and best block estimation. To know if LND is ready i do use Connect 2 peer. It would be nice if Start LND could push status available to end user apps via a rest endpoint.

Typically the "main" RPC is the last to start up. Things like ConnectToPeer won't actually work before the server has started up. While syncing we start up RPCs that allow you to get things like getinfo and other status reports. You won't be able to do things like connect to peers or make channels during this phase however. Once the "synced_to_chain" bool flips to true, then you know that lnd is ready for action.


Attribute to pack archived types

Per rodarmor:

Have you considered not bothering with padding and alignment, and making all Archive types alignment 1, so that they can be read at any alignment? I started out including padding and alignment, but I read the Flatbuffers 2.0 thread, and they mentioned that it doesn't seem like unaligned loads are penalized on modern hardware, so they would probably leave out alignment and padding if they could.

This can probably be implemented just by changing types to #[repr(packed)]. Adding an argument to the derive attribute would do the trick.

Can cause undefined behavior, pushing back until it's a feature that can feasibly be supported.

Since this is probably going to cause undefined behavior forever, let's close this until someone really needs it.


Additional changes to the hammerdb addition

Had to fix a few bits and pieces, should only affect hammerdb

@mkarg75 can you please squash the commits into one and rebase against the master so that its easier to see the differences that this pr actually introduces?

/rerun all

Results for SNAFU CI Test

Test
Result
Runtime

cluster_loader
FAIL
00:00:00

fio_wrapper
PASS
00:06:22

fs_drift_wrapper
PASS
00:06:56

hammerdb
FAIL
00:12:46

iperf
PASS
00:05:28

pgbench-wrapper
PASS
00:06:53

smallfile_wrapper
PASS
00:08:13

sysbench
PASS
00:05:09

uperf-wrapper
PASS
00:06:33

ycsb-wrapper
PASS
00:05:16


Error's not reported to web interface

I've just found strider and was very excited, however after an hour or so trying to get it running locally (with docker using https://github.com/macropin/docker-strider) I'm pretty frustrated.
The primary problem I've found is that errors occurring in strider are not displayed in any way to the user in the web interface. Not even "something's gone wrong", surely as an admin I should get a stracktrace?
For example, I've been trying to build the public project https://github.com/samuelcolvin/arq as an experiment. I've tried with both the python plugin and custom. With both plugins I've got errors:
16-09-06T14:05:54.693Z - info: [runner:simple-runner] Queued new job. Project: samuelcolvin/arq Job ID: 57cecd42ca256c530055b4e4
2016-09-06T14:05:54.762Z - info: [runner:simple-runner] Job started. Project: samuelcolvin/arq Job ID: 57cecd42ca256c530055b4e4
2016-09-06T14:05:54.763Z - info: [runner:simple-runner] Found a disabled plugin: python Project: samuelcolvin/arq Job ID: 57cecd42ca256c530055b4e4
/data/node_modules/step/lib/step.js:39
        throw arguments[0];
        ^

TypeError: Cannot set property 'errored' of undefined
    at /data/node_modules/strider-simple-runner/lib/index.js:375:29
    at Object.Job.done (/data/node_modules/strider-runner-core/lib/job.js:145:5)
    at Object.Job.phaseDone (/data/node_modules/strider-runner-core/lib/job.js:391:21)
    at /data/node_modules/strider-runner-core/node_modules/async/lib/async.js:232:13
    at /data/node_modules/strider-runner-core/node_modules/async/lib/async.js:136:21
    at /data/node_modules/strider-runner-core/node_modules/async/lib/async.js:229:17
    at /data/node_modules/strider-runner-core/node_modules/async/lib/async.js:556:34
    at updateCache (/data/node_modules/strider-git/worker.js:159:14)
    at /data/node_modules/strider-git/lib/index.js:126:5
    at Function.<anonymous> (/data/node_modules/gitane/index.js:173:7)
    at next (/data/node_modules/step/lib/step.js:51:23)
    at ChildProcess.<anonymous> (/data/node_modules/gitane/index.js:159:9)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:852:16)
    at Socket.<anonymous> (internal/child_process.js:323:11)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at Pipe._handle.close [as _onclose] (net.js:492:12)
83 died 1 null
114 forked
2016-09-06T14:05:57.171Z - info: RUNNER INIT
2016-09-06T14:06:05.539Z - info: Fetching projects...
2016-09-06T14:06:05.540Z - info: [ null ]
2016-09-06T14:06:05.605Z - info: Saved repo cache
2016-09-06T14:06:29.953Z - info: GET [ecosystem]/plugins.yml
2016-09-06T14:06:40.123Z - info: [runner:simple-runner] Queued new job. Project: samuelcolvin/arq Job ID: 57cecd6fa290b27200889374
2016-09-06T14:06:40.169Z - info: [runner:simple-runner] Job started. Project: samuelcolvin/arq Job ID: 57cecd6fa290b27200889374
/opt/strider/lib/utils/index.js:139
function defaultVal(val) {
                   ^

RangeError: Maximum call stack size exceeded
    at defaultVal (/opt/strider/lib/utils/index.js:139:20)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
    at defaultSchema (/opt/strider/lib/utils/index.js:157:17)
    at defaultVal (/opt/strider/lib/utils/index.js:150:39)
114 died 1 null
126 forked
2016-09-06T14:07:53.986Z - info: RUNNER INIT

(and numerous more)
None of these were displayed in the UI the build screen just hung.
I'm also not clear on how I would isolate services (eg. dbs) wrt separate builds and the configure > plugins page seems to have a UI bug where I can't remove a plugin once I've added it.
I'm really keen to get self hosted CI working, however with fundamental problems like error reporting not working I'm not particularly keen to invest more time in strider.

Some errors are so critical that it is not possible to communicate them to the UI. That being said, this could be improved.
Some of the errors you're encountering are probably fixed in the latest version, which is pending release. Please pull the latest sources from master and try again. Set DEBUG to strider* to capture all relevant output.
Please also take note of https://github.com/Strider-CD/strider/wiki/Advanced-Configuration


Strange field cache behaviour using MongoDB + .find()

Describe the bug
A strange cache is going on when I'm using an EntityRepository<T>'s .find() providing custom fields.
It seems the fields are cached for the EntityRepository the first time .find() is called, and never changed after.
To Reproduce
I've recorded a video demonstrating the issue here: https://drive.google.com/file/d/1ytQP3UuPgvfrhFVgwUhpYpSQzbUqtMKK/view?usp=sharing
In the video you can see:

A URL query string is used to determine which fields are going to be used
I open the URL using query string "league"
The query string is changed to "ranked", having no effect on the fields argument
I restart the server (thus clearing the "cache") and refresh the page still using "ranked" query string, and it displays correctly (changing back to "league" has no effect over the fields)

(I'm not using any kind of caching on my side)
Expected behavior
The fields should have changed
Additional context
I've used a global User Repository for the video, and later tried to retrieve a Repository for each HTTP request via DI.orm.em.getRepository(User) but the problem persisted
Versions

Dependency
Version

node
14.3.0

typescript
4.0.5

mikro-orm
4.3.2

your-driver
mongodb

It seems the fields are cached for the EntityRepository the first time .find() is called, and never changed after.

Not just fields, the whole entity is stored in identity map. If you select it for the first time with limited fields, you will have partial entity there. You can use refresh: true in FindOptions to force reload it. This is one of the reasons why each request needs to have its own identity map (aka request context).
https://mikro-orm.io/docs/identity-map

Thanks for the quick reply @B4nan, I've totally missed this. It's working perfectly now! 🥳


vite/spa: npm run start using http-server does not support SPA routes

Reproduction
Run this script (PowerShell):
npx create-remix@latest remix-spa-test -y --template remix-run/remix/templates/spa
cd .\remix-spa-test
echo "export default function Hello() { return <>Hello</>; }" | Out-File ./app/routes/hello.tsx -Encoding utf8
echo "import { Link } from `"@remix-run/react`";`n`nexport default function Index() { return <Link to=`"/hello`">Navigate</Link>; }" | Out-File ./app/routes/_index.tsx  -Encoding utf8
npm run build
npm run start

Open browser directly to http://<IP_ADDRESS>:8080/hello (change port if needed). Also try reloading the browser (F5) after navigating to the /hello route from the root (/) path.
System Info
System:
    OS: Windows 10 10.0.19045
    CPU: (24) x64 12th Gen Intel(R) Core(TM) i7-12850HX
    Memory: 31.75 GB / 63.67 GB
  Binaries:
    Node: 18.17.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.19 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 9.6.7 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Chromium (121.0.2277.83)
    Internet Explorer: 11.0.19041.3636
  npmPackages:
    @remix-run/dev: ^2.5.1 => 2.5.1
    @remix-run/node: ^2.5.1 => 2.5.1
    @remix-run/react: ^2.5.1 => 2.5.1
    vite: ^5.0.12 => 5.0.12

Used Package Manager
npm
Expected Behavior
When previewing a build, the server should support SPA fallback mode when using the SPA template. Use vite preview or sirv-cli --single as alternatives. (vite preview is powered by sirv).
No simple solutions that I found are especially production ready as they don't set the cache-control header for ./assets files separately from other files.
Actual Behavior
Navigating directly to a sub-route such as http://<IP_ADDRESS>:8080/hello fails with an HTTP ERROR 404.
Navigating to the page when starting at http://<IP_ADDRESS>:8080/ works correctly, but reloading once navigated to http://<IP_ADDRESS>:8080/hello also fails with an HTTP ERROR 404.
http-server does not support SPA mode. (it has been much discussed, but left unresolved, see https://github.com/http-party/http-server/pull/772).

I'm a bit torn here 🤔
Yes - Remix SPA mode is built to allow you to load/hydrate any path on initial load.  So in theory you're correct that when testing the prod build it should also allow that.
However, I don't love encouraging testing the prod build via something that does not match what you will actually use in prod (since vite preview is explicitly not production ready).  I've not used vite preview enough to know if there's a surface area between vite preview and serving SPA mode in production some other way (i.e., via Express app.use('*', (req, res) => res.sendFile('index.html'))).
For example, would it be a potential footgun for folks to run vite preview to test their "prod" build and conclude they can load/hydrate non-/ routes - just to deploy to a CDN or github pages and find that it's broken?
It almost seems potentially safer to use something that aligns with the lowest common denominator of deploy modes (i.e., something like http-server which doesn't support this "SPA fallback" concept) and then let folks opt-into that via vite preview if that is their intended deployment strategy?
I dunno - maybe this is all a non-issue - if vite's decided that previewmode differences from production deploys isn't a concern, so can we?

@brophdawg11 Thanks for looking into this. I agree that this command isn't production ready, but neither is any other simple option without going back to a remix-serve type solution. From my issue:

No simple solutions that I found are especially production ready as they don't set the cache-control header for ./assets files separately from other files.

This is why my proposed solution is to drop the npm run start script and use npm run preview like Vite suggests to alleviate the possible confusion.

ok yeah I think I can get on board with removing any opinions about how to deploy to production and just linking to Vite's docs there and main that apparent in the docs/template.  I'll take a look over the PR when I have a moment 👍


Any plans for a stable release?

Any plans for a stable release?

One of our dependency libraries is currently tagged as preview. Once it becomes GA, we can go GA.
That preview package is used for Logic Apps Standard, which has already become GA. I need to check the status.
/cc: @spboyer @fabiocav

Thanks for the update!

Just FYI – this library is a successor of the Aliencube one. Although currently it's labelled as "preview", it has richer feature sets than the other one.

We know, that's why we'd love to see this one go stable so we can use this one instead 😁

Just FYI - we're planning its GA with .NET 6 GA.


Enable NewCops up to a specific version of rubocop

Is your feature request related to a problem? Please describe.
I don't prefer to use NewCops: enable because I like finding out what cops are new and configuring them to my liking. That said, I end up with a ton of configuration that is redundant with the default config just to enable all the cops.
Describe the solution you'd like
I would like to be able to do something like NewCops: 1.50.0 where it will have the default behavior of enable/disable up until that version and then show me warnings for just what's new. That way I have a smaller/simpler config but I still opt-in as new versions are released.
Describe alternatives you've considered
The only other alternative I'm aware of is manually maintaining this like I'm currently doing.
Additional context
I'm not sure how this should interact with plugins. It might need multiple version entries or some way to configure separately.
I thought there was an issue around this from a long time ago but I can't find it so sorry if this ends up being a duplicate.

There's https://github.com/rubocop/rubocop/pull/8565 that got stuck for various reasons. Feel free to revive it.


Export artifact that lists provided service interfaces

What does this Pull Request accomplish?
Generates the file server_impl_info.json adjacent to the server executable. It is included in the exported artifacts.

Currently the file contains only the provided service interfaces. This can be easily extended in the future.
Chose to not put this in the same file as the server configuration, because it's intended for a totally different use.

Why should this Pull Request be merged?
Task 2206702: Export information with gRPC server about what proto files are supported
What testing has been done?
Built locally. It contains these lines:
{
    "provided_service_interfaces": [
        "nidaqmx_grpc.NiDAQmx",
        "nidcpower_grpc.NiDCPower",
        "nidigitalpattern_grpc.NiDigital",
        "nidmm_grpc.NiDmm",
        "nifgen_grpc.NiFgen",
        "nimxlcterminaladaptor_restricted_grpc.NimxlcTerminalAdaptorRestricted",
        "nirfmxbluetooth_grpc.NiRFmxBluetooth",
        "nirfmxinstr_grpc.NiRFmxInstr",
        "nirfmxinstr_restricted_grpc.NiRFmxInstrRestricted",
        "nirfmxlte_grpc.NiRFmxLTE",
        "nirfmxnr_grpc.NiRFmxNR",
        "nirfmxspecan_grpc.NiRFmxSpecAn",
        "nirfmxspecan_restricted_grpc.NiRFmxSpecAnRestricted",
        "nirfmxwlan_grpc.NiRFmxWLAN",
        "nirfsa_grpc.NiRFSA",
        "nirfsg_grpc.NiRFSG",
        "niscope_grpc.NiScope",
        "niswitch_grpc.NiSwitch",
        "nisync_grpc.NiSync",
        "nitclk_grpc.NiTClk",
        "nixnet_grpc.NiXnet",
        "nixnetsocket_grpc.NiXnetSocket"
    ]
}

Although this file contains all interfaces, in theory on RT for example we could limit it to only the APIs that are supported. But I had to start somewhere. Would like feedback on it, though. @reckenro @astarche

Although this file contains all interfaces, in theory on RT for example we could limit it to only the APIs that are supported. But I had to start somewhere. Would like feedback on it, though. @reckenro @astarche

FYI, printing a list of interfaces to stdout would be more dynamic, but I don't think we need that. Generating a static file and installing it is adequate for our purposes and allows us to use this list at build time (though I understand your concerns about doing so).

[bikeshed] Re: server_impl_info.json
Is this really "implementation info"? It describes the interface to the server (the package/service names used in its .proto files), not the implementation.
Consider something like services.json or server_capabilities.json.

Consider something like services.json or server_capabilities.json.

Since this is intended to potentially grow over time, 'services' is too limiting. But I do like the other name, server_capabilities.json so I'll go with that.


IAO Assignment Function Update - IAO Re-alignment

As an IAO user
I want to be able to assign any request, general or personal, to any other team/individual that is an IAO user
so that I can manage requests that are assigned across teams

Assumptions and Scope:
With the upcoming IAO reorganization, we will require greater flexibility in assigning requests. During the transition period, there will be files being passed across teams, or remain with analysts where the mappings currently do not site.
To address this, and any future changes, we need to make it such that any request can be assigned to any IAO user, regardless of the team they belong to.
There is also a new team, called the Coordinated Response Unit - which may assist other teams with requests when they need support. As such, there is a need for flexibility compared to how we currently manage assignments.
What is IN scope?
Updating who requests can be assigned to
What is NOT in scope?
Sort order of names should remain the same, with the exception of the position of the Ministry that is mapped to the team
Restricted requests will continue to have the same restrictions applied to them
Intake in Progress requests will continue to be managed by the Intake team
How assignments/watchers are added will not be changed
Acceptance Criteria
Scenario 1: General FOI Request in Open State - Primary Assignment

GIVEN an IAO user is on a general request in any state from the Open state onwards
WHEN they activate the assignment dropdown
THEN they will be able to see all IAO teams and users in the list
AND the team that is "mapped" to the Ministry code will appear at the top of the list

Scenario 2: Personal FOI Request in Open State - Primary Assignment

GIVEN an IAO user is on a peneral request in any state from the Open state onwards
WHEN they activate the assignment dropdown
THEN they will be able to see all IAO teams and users in the list
AND the team that is "mapped" to the Ministry code will appear at the top of the list

Scenario 3: General FOI Request in Open State - Watcher

GIVEN an IAO user is on a general request in any state from the Open state onwards
WHEN they activate the watcher dropdown
THEN they will be able to see all IAO teams and users in the list
AND the team that is "mapped" to the Ministry code will appear at the top of the list

Scenario 4: Personal FOI Request in Open State - Watcher

GIVEN an IAO user is on a peneral request in any state from the Open state onwards
WHEN they activate the watcher dropdown
THEN they will be able to see all IAO teams and users in the list
AND the team that is "mapped" to the Ministry code will appear at the top of the list

Dependencies? What is the impact of this dependency? (If so, link dependency in the ticket, make it visible in a team´s backlog)
#3499
Validation Rules? (If yes, list here)
Design
@xxx - please link the Design here
Definition of Ready

[ ] Is there a well articulated User Story?
[ ] Is there Acceptance Criteria that covers all scenarios (happy/sad paths)?
[ ] If there is a user interface, is there a design?
[ ] Does the user story need user research/validation?
[ ] Does this User Story needs stakeholder approval?
[ ] Design / Solution accepted by Product Owner
[ ] Is this user story small enough to be completed in a Sprint? Should it be split?
[ ] Are the dependencies known/ understood? (technical, business, regulatory/policy)
[ ] Has the story been estimated?

Definition of Done

[ ] Passes developer unit tests
[ ] Passes peer code review
[ ] If there's a user interface, passes UX assurance
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Confirm Test cases built and succeeding
[ ] No regression test failures
[ ] Test coverage acceptable by Product Owner
[ ] Ticket ready to be merged to master or story branch
[ ] Developer to list Config changes/ Update documents and designs
[ ] Can be demoed in Sprint Review
[ ] Tagged as part of a Release
[ ] Feature flagged if required
[ ] Change Management activities done?

Note to devs/team: if doing both 3499 and 3500 is going to be a lot of work, we could just do this one as a priority and I could modify the AC to remove the sort feature until after we do the remapping

Also discussed - @richard-aot will join Nick on this task after his bug ticket completes

All working as it should. closing ticket!


A few small fixes

remove PR # from GitHub status line to save space
correct typo LLMV in main.cpp
make DocExampleTest installation of onnx on x86 less verbose

@gongsu832 You need to sign off the commit properly.

@caoimhinuibrian All my patches are signed off after we turned DCO on. The DCO failure was due to one single patch that was not signed off before we turned on DCO. I've tried all kinds of ways of trying to get it fixed with no luck. So I gave up.


Fix MATTER_TRACING_ENABLED checks.

We were checking MATTTER_TRACING_ENABLED without including the config header that defines it.
Fixes https://github.com/project-chip/connectedhomeip/issues/29214

One complain would be that the changed files does not actually use MATTER_TRACING_ENABLED macros at all, so it is correct from a "include only what you need" point of view.
The issue is that underlying users of the macros may not know what exactly to include, but maybe they should be forced to include the right thing...

One complain would be that the changed files does not actually use MATTER_TRACING_ENABLED macros at all, so it is correct from a "include only what you need" point of view.
The issue is that underlying users of the macros may not know what exactly to include, but maybe they should be forced to include the right thing...

Never mind ... I had not expanded enough. the fix is correct.


Improve UI for Alert Bar

The current alert bar covers the full width of the screen but looks unattractive and unappealing. This issue aims to enhance its UI by:
Adjusting the design to make it visually appealing.
Adding padding and margin for better spacing.
Improving color contrast and font styles for better readability.
Could you please assign this issue to me and add the hacktoberfest and other lables

@dipexplorer the project is not under hactoberfest , as of now it is for GSSoC-extd

Could you please assign this issue to me I like to contribute to it.


BackupPolicy should provide a way to use fsGroup securityContext

When using BackupPolicy with ephemeral volumes, backups can fail with a permission denied error trying to write dump to the volume. This is because moco-backup run as uid 10000 and persistent volume is created with a root folder owned by root.
BackupPolicy CR should provide a way to set an fsGroup or an fsGroup should be hardcoded in Cronjob definition to prevent this problem.

For those who are looking for a way to workaround this problem, here is our solution using a Kyverno policy to mutate CronJob's Pods on the fly:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: mutate-add-fsgroup-moco-backup-jobs
spec:
  rules:
  - name: add-fsgroup-moco-backup-jobs
    match:
      resources:
        kinds:
        - Pod
        selector:
          matchLabels:
            app.kubernetes.io/created-by: moco
            app.kubernetes.io/name: mysql-backup
    mutate:
      patchStrategicMerge:
        spec:
          securityContext:
            +(fsGroup): 10000

Thank you for the issue!
As you suggested, we need fsGroup for Backup CronJob and also Resotre Job.
We will add the securityContext field to the JobConfig.
https://github.com/cybozu-go/moco/blob/v0.12.1/api/v1beta2/job_types.go#L10

Alternatively, it might be better to add securityContext.fsGroup automatically.
Since the GID 10,000 is a kind of internal value.
Doing so would also reduce the troubles of moco users.

Thank you both for your answer! :-)
I agree with @ymmt2005, and I don't seen any drawback to set an fsGroup even if permissions are already properly set.

@ymmt2005
Thank you. I understood.
runAsGroup: 10000 is automattically set for each containers so that looks better to set fsGroup too.
https://github.com/cybozu-go/moco/blob/808863e82358b581d39c28d9c6d761875fe4d8bd/controllers/mysql_container.go#L342-L344
My understanding is that fsGrpuo: 10000 will be set to all containers like runAsGroup.
So the following document will no longer be necessary, isn't it?
https://github.com/cybozu-go/moco/blob/v0.12.1/docs/troubles.md

@masa213f
Not exactly. fsGroup is a Pod-level security policy, so we don't need to add them to each container.

Also, I'd like to include fsGroupChangePolicy: "OnRootMismatch" as it is available on all supported k8s versions.

@ymmt2005
Thanks! Certainly, fsGroup is only exists in PodSecurityContext...
Should I set the fsGroup (and fsGroupChangePolicy) to the mysql pod too?
If I don't need, I just set the fsGroup to the restore and backup jobs.

@masa213f Cc: @d-kuro

Should I set the fsGroup (and fsGroupChangePolicy) to the mysql pod too?

We should.

Released.
https://github.com/cybozu-go/moco/releases/tag/v0.13.0


Consider converting to Sinatra

I would like to do some refactoring in the gitdocs UI, and I wonder if we should  the web framework first.
@nesquena or @joshbuddy do you consider Renee to still be supported? I see that it has been in pre-release since 2012, and according to rubytoolbox it is only being used by gitdocs.
What would you think of switching to Sintara? (Padrino would be an option, but my first instinct is that it probably more heavy than necessary for this gitdocs.)

Resolve by PR #98 which has been merged into mater.
Closing this issue.


Fix sample definition

The repackage execution is inherited from the parent so there is no
need to duplicate it here.
@ZhijunZhao this related to our discussion in #159. If you generate an empty app on start.spring.io, you'll get the best practices that we are currently recommending.

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@snicoll, thanks! You're making a valid point. I must have missed something back then.


xDescriptors not functioning as documented

Bug Report
What did you do?

mkdir memcached-operator && cd memcached-operator
operator-sdk init --domain example.com --repo=github.com/example/memcached-operator --project-version 3-alpha --license apache2 --plugins go.kubebuilder.io/v2
operator-sdk create api --group=cache --version=v1alpha1 --kind=Memcached
Add markers as in Step 2 in API Markers example
make bundle

fill in information

What did you expect to see?
ClusterServiceVersion should container
specDescriptors:
  - description: Size is the size of the memcached deployment.
    displayName: Number of pods
    path: size
    x-descriptors:
      - urn:alm:descriptor:com.tectonic.ui:podCount
      - urn:alm:descriptor:io.kubernetes:custom

What did you see instead? Under which circumstances?
ClusterServiceVersion contains container
specDescriptors:
  - description: Size is the size of the memcached deployment.
    displayName: Number of pods
    path: size
    x-descriptors:
      - urn:alm:descriptor:com.tectonic.ui:podCount,urn:alm:descriptor:io.kubernetes:custom

Environment

operator-sdk version:

operator-sdk version: "v1.0.0", commit: "d7d5e0cd6cf5468bb66e0849f08fda5bf557f4fa", kubernetes version: "v1.18.2", go version: "go1.13.11 linux/amd64", GOOS: "linux", GOARCH: "amd64"

go version:

go version go1.14.6 linux/amd64

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3", GitCommit:"002a51f", GitTreeState:"clean", BuildDate:"2020-08-06T07:16:38Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:  OKD

Are you writing your operator in ansible, helm, or go?
go

Possible Solution
Update documentation to separate each descriptor by line
    // Size is the size of the memcached deployment.
    // +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Number of pods"
    // +operator-sdk:csv:customresourcedefinitions:type=spec,xDescriptors="urn:alm:descriptor:com.tectonic.ui:podCount"
    // +operator-sdk:csv:customresourcedefinitions:type=spec,xDescriptors="urn:alm:descriptor:io.kubernetes:custom"
    Size int32 `json:"size"`

/kind bug
/assign

On investigation, this isn't actually a bug but rather a change in how string lists are parsed: they must be in the form {x,x} or x;x where x is an individually quoted element.
I'll update the docs.
/remove-kind bug
/kind documentation


How to use bootstrap modal?

@rubanshah to go through the implementation you can have a look at the official bootstrap for vue docs for reference.
The implementation is straight forward as mentioned there and for any props clearance you can have a look at the code : https://github.com/cdmoro/bootstrap-vue-3/blob/main/packages/bootstrap-vue-3/src/components/BModal.vue

https://github.com/cdmoro/bootstrap-vue-3/issues/491#issuecomment-1191446667

The docs are incomplete. You can review the actual code if you want things to be a little bit more in depth, but perhaps more confusing to someone that has never reviewed the code... I'd suggest reviewing the props and emits since the internals are not all to useful. In addition, this example https://github.com/cdmoro/bootstrap-vue-3/issues/491#issuecomment-1191446667 may help with getting started.

I'm going to close this as completed since there has been no additional reply, this is added, just not well documented.


[WIP] Add AOT capability

Currently fails with this exception:
[FATAL:flutter/runtime/dart_vm.cc(409)] Error while initializing the Dart VM: Snapshot not compatible with the current VM configuration: the snapshot requires 'product no-dwarf_stack_traces use_bare_instructions no-"asserts" causal_async_stacks x64-sysv' but the VM has 'release no-dwarf_stack_traces use_bare_instructions no-"asserts" causal_async_stacks no-code-comments x64-sysv'

Also see https://github.com/go-flutter-desktop/hover/pull/77

The error happens because of this: https://github.com/flutter/flutter/issues/44577#issuecomment-553149711 and it may take while or two (who knows) for the flutter team to fix it (desktop is not their top priority).

https://github.com/flutter/flutter/issues/48018


Patch 1

+1 for using Bower. The standard practice is to include external libs via Bower at deployment time. We just need to make sure that there are no version discrepancies among some shared components. For example, if one widget requires some version of module X and another widget requires a different version, then that could cause issues.
I think most of these can be solved by some of the social process, though - if we all keep an eye on each other's PRs and tests, that'll go a long way.


Why there's not function Sum?

?

use db.Model(&User{}).Select("sum(age)").Scan()

With Scan(), can I directly read the SUM data into a int/float variable?
It is asking me to use a struct as destination.

use db.Model(&User{}).Select("sum(age)").Scan()

result := make(map[string]interface{})
db.Select(selectString).Scan(&result)
我想使用灵活的map[string]interface{}，这样不用在每个Sum都重新构造结构体

With Scan(), can I directly read the SUM data into a int/float variable? It is asking me to use a struct as destination.

Supported in V2

我想使用灵活的map[string]interface{}，这样不用在每个Sum都重新构造结构体

Supported in V2

谢谢


cmake doesn't check for presence of yaml-cpp

@tarunik want to work on this one when you're finished with the static analysis stuff?

This is being fixed by pulling in the FindYaml.cmake from https://github.com/lbaehren/CMakeModules/blob/master/FindYaml.cmake


Datapath in debug mode spams "Successfully mapped addr=<IP_ADDRESS> to identity=0" messages

Version: Cilium v1.8.2
To reproduce

Follow the GKE deployment guide: https://docs.cilium.io/en/v1.8/gettingstarted/k8s-install-gke/
Deploy connectivity check and hubble relay via instructions from the above documentation.
Edit the ConfigMap to add datapath debug messages:   debug: "true"
   debug-verbose: datapath

Problem
When I go into one of the nodes and run cilium monitor -v, this message floods the output:
CPU 02: MARK 0x0 FROM 2462 DEBUG: Successfully mapped addr=<IP_ADDRESS> to identity=0

Other messages will show up, but they're drowned out by this message.
In this environment, endpoint 2462 is the host endpoint and the destination is another node's IP address.

$ cilium status
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.14+ (v1.14.10-gke.42) [linux/amd64]
Kubernetes APIs:        ["CustomResourceDefinition", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Endpoint", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Probe   [eth0 (DR)]   [NodePort (SNAT, 30000-32767, XDP: DISABLED), HostPort, ExternalIPs, HostReachableServices (TCP, UDP), SessionAffinity]
Cilium:                 Ok      OK
NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok
IPAM:                   IPv4: 11/255 allocated from <IP_ADDRESS>/24,
Masquerading:           BPF   [eth0]   <IP_ADDRESS>/14
Controller Status:      62/62 healthy
Proxy Status:           OK, ip <IP_ADDRESS>, 3 redirects active on ports 10000-20000
Hubble:                 Ok              Current/Max Flows: 4096/4096 (100.00%), Flows/s: 29.31   Metrics: Ok
Cluster health:         2/2 reachable   (2020-08-06T17:50:29Z)

Perhaps also relevant:
$  cilium ip list
IP                 IDENTITY   SOURCE
<IP_ADDRESS>/0          2
<IP_ADDRESS>/32      1
<IP_ADDRESS>/32      4
<IP_ADDRESS>/32      28679      k8s
<IP_ADDRESS>/32     5759       k8s
<IP_ADDRESS>/32     29178      k8s
<IP_ADDRESS>/32     8085       k8s
<IP_ADDRESS>/32     55632      k8s
<IP_ADDRESS>/32     51604      k8s
<IP_ADDRESS>/32     7179       k8s
<IP_ADDRESS>/32     2643       k8s
<IP_ADDRESS>/32     3660       k8s
<IP_ADDRESS>/32      50394      k8s
<IP_ADDRESS>/32      37076      k8s
<IP_ADDRESS>/32      6
<IP_ADDRESS>/32      25444      k8s
<IP_ADDRESS>/32      27935      k8s
<IP_ADDRESS>/32      61293      k8s
<IP_ADDRESS>/32     47544      k8s
<IP_ADDRESS>/32     42115      k8s
<IP_ADDRESS>/32     30857      k8s
<IP_ADDRESS>/32     49622      k8s
<IP_ADDRESS>/32     27261      k8s
<IP_ADDRESS>/32     5759       k8s
<IP_ADDRESS>/32     4
<IP_ADDRESS>/32     38218      k8s
<IP_ADDRESS>/32     42029      k8s
<IP_ADDRESS>/32     6
<IP_ADDRESS>/32     1

It may be that this is the only debug message on the path incoming into the node towards the stack from remote nodes (ie node-to-node), and we don't log any other debug or trace messages. I would expect incoming trace messages in addition to these debug messages, but at least my initial code inspection doesn't reveal where such a trace message would be triggered when hostfw is disabled.

There's another aspect to this which is a bit unusual, specifically the identity is determined to be 0 despite the identity being in the ipcache. I'm looking sideways at:
https://github.com/cilium/cilium/blob/3996e5a1a36e60674585173a2cadaa973ab93b9f/bpf/bpf_host.c#L393
@pchaigno do you recall why the from_host && case was added here? My understanding of the other portion is that it's explicitly intended for traffic from remote nodes. Related, the original commit that introduced the check: 5b0689b1204aa

Seems like later on we'll actually look up the ipcache anyway in ipv4_host_policy_ingress() and disregard this lookup in the "traffic ingressing from outside the node" case. The logic is also subtly different. This could probably do with a bit more close attention.

I was able to reproduce with v1.7 and 1.7.0. The message is a bit different in v1.7.0; Failed to map instead of Successfully mapped. It is also not required to deploy the connectivity check to reproduce. As far as I understand, the messages corresponded to the SSH packets in my case, thus the high rate.

@pchaigno do you recall why the from_host && case was added here? My understanding of the other portion is that it's explicitly intended for traffic from remote nodes. Related, the original commit that introduced the check: 5b0689b

If was added because of the #ifndef ENABLE_EXTRA_HOST_DEV one line guard above. From the commit that introduced that from_host (a695f532d0607):

ENABLE_EXTRA_HOST_DEV was only defined in init.sh for the from_host path.
It is now defined from the Go side for all paths, and we check whether
from_host is true in the C code instead.

Seems like later on we'll actually look up the ipcache anyway in ipv4_host_policy_ingress() and disregard this lookup in the "traffic ingressing from outside the node" case.

Yep. I still have an item on my TODO to rework this and avoid the unnecessary lookup (though it predates the host endpoint IIRC), but like you said, it requires close attention.
Given we report a 0 identity, this rework should allow us to also avoid the excessive debug messages.

Unpinning and closing, I'm gonna assume that this is no longer a problem for datapath developers, otherwise someone would have done something about it by now.


Chain of thought is not a good fit here

[Enter feedback here]
I don't see how chain of thought is relevant to this page. Even the example is a typical CoT example totally irrelevant with the vision capabilities of GPT4V.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: cf90aaaf-f9bc-4ce3-c4fd-04b30dece452
Version Independent ID: 0559d74a-b828-7878-ec7e-04027c5db208
Content: Introduction to prompt engineering with GPT-4 Turbo with Vision - Azure OpenAI
Content Source: articles/ai-services/openai/concepts/gpt-4-v-prompt-engineering.md
Service: azure-ai-openai
GitHub Login: @PatrickFarley
Microsoft Alias: pafarley

@melkor54248
Thanks for your feedback! We will investigate and update as appropriate.

@melkor54248 , This page contains prompt engineering tips for both text and image prompts. GPT-4 Turbo with Vision has the same text capabilities as other GPT-4 models

@melkor54248
if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.

@melkor54248 , This page contains prompt engineering tips for both text and image prompts. GPT-4 Turbo with Vision has the same text capabilities as other GPT-4 models

True but isn't this page about specific prompt engineering techniques used with images?

Yes, I'm going to clarify the theme of the doc and remove the non-image example from it

#please-close