Improve the "validation failed" error message wording

The current validation message seems to imply to users either that the write failed, or that Etcher damaged the card. We need to make clearer what it is that we are trying to convey. Perhaps a better message would be something like:
"The write has been completed successfully but Etcher detected potential corruption issues in reading the image back from the drive. It's possible that the drive is corrupted, please consider writing the image to a different one."

It would also be helpful to provide more detail on the verification error. For example, what failed -- was it a comparison between expected and actual data that failed, or an inconsistent result over multiple reads? And where was it on the drive? It would be nice if the raw sector information were reported at least, but if you were able to load the drive image further to get a partition or even specific file that would be great.
I've had verification issues after flashing drives on multiple occasions, but without extra information there's not much that can be done.

It would also be helpful to provide more detail on the verification error. For example, what failed -- was it a comparison between expected and actual data that failed, or an inconsistent result over multiple reads? And where was it on the drive? It would be nice if the raw sector information were reported at least, but if you were able to load the drive image further to get a partition or even specific file that would be great.

Sadly, validation is currently implemented by just comparing checksums, so more detailed information about what exactly failed is not possible at the moment, but definitely something to consider.
I'd love to have sector specific information, and also a percentage of the data that failed, since 50% failure is very different from 0.01% failure.

@jviotti, I guess if we really got hardcore about it we could save more
checksums for smaller parts of the image and compare piece by piece, giving
us more detail on exactly what failed, right? And I guess if the chunk is
small enough and we have the image available (or does this conflict with
archives?) we could compare byte by byte and find the precise point. But
yeah, this is for later.
Or maybe @petrosagg has some crazy idea about an efficient encoding that
would automatically tell us where the failure was :P
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Fri, Sep 30, 2016 at 8:14 AM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:

It would also be helpful to provide more detail on the verification error.
For example, what failed -- was it a comparison between expected and actual
data that failed, or an inconsistent result over multiple reads? And where
was it on the drive? It would be nice if the raw sector information were
reported at least, but if you were able to load the drive image further to
get a partition or even specific file that would be great.
Sadly, validation is currently implemented by just comparing checksums, so
more detailed information about what exactly failed is not possible at the
moment, but definitely something to consider.
I'd love to have sector specific information, and also a percentage of the
data that failed, since 50% failure is very different from 0.01% failure.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/735#issuecomment-250771098,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCDgCSsK_4RYZCwDKwxDb_3CnS3uqks5qvSdjgaJpZM4KLI71
.

Sadly, validation is currently implemented by just comparing checksums, so more detailed information about what exactly failed is not possible at the moment, but definitely something to consider.

Ah, I was scared of that. If you were able to do it on a sector-by-sector basis, it would also let you show a "failed" message much earlier in the process because you wouldn't necessarily need to wait for the whole drive to be scanned. Obviously you would still want to verify the whole thing to provide more information, but you could at least tell the user what happened and let them abort more quickly.

I'd love to have sector specific information, and also a percentage of the data that failed, since 50% failure is very different from 0.01% failure.

Yeah, that would be great. In general I have just continued using a card even if validation failed for it, because if the device doesn't operate quite right I just re-flash it :laughing: I see people complain about Etcher's validation errors fairly often, and honestly I haven't had the time or patience to look into whether it's an issue with Etcher or an issue with the image/card/reader.

Many smaller checksums could work. Alternatively, by the way checksum calculation over streaming works, we could retrieve the checksum for all the bytes what were consumed so far and save various "checkpoints" we could use for comparison purposes.

The checkpoints approach sounds much more implementation-friendly and
should accomplish the same result, given that validation should always be a
linear process as far as I can think.
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Sun, Oct 2, 2016 at 5:50 PM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:

Many smaller checksums could work. Alternatively, by the way checksum
calculation over streaming works, we could retrieve the checksum for all
the bytes what were consumed so far and save various "checkpoints" we could
use for comparison purposes.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/735#issuecomment-251007933,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCAtDWK4qYsFfTCX5khhvtQNRQOgUks5qwFFugaJpZM4KLI71
.

Hmm, so if you're storing multiple checksum-checkpoints-along-the-way, does that mean it'd be better to change it from:

Write the whole image
Read the whole image, verifying the "rolling checksum" at various checkpoints

to:

Write a chunk of the image
Read that chunk of the image, calculating the rolling checksum
Write the next chunk of the image
Read that chunk of the image, calculating the rolling checksum

That way if the user had a 'bad' card they'd get notified earlier, rather than having to wait for the whole image to be written first (and I guess you could give them the options of 'Ignore verification error' or 'Abort writing'). Only possible downside is that flipping back and forth between writing and reading may be slower than just writing all in one go. Would need some experimentation...

I think @jviotti wanted to keep the processes in one piece for performance
reasons, though I could imagine that there could exist natural "stopping
points" where the switching penalty is small or non-existant.
I also had a quick chat with @petrosagg who wasn't quite convinced the
process couldn't be multiplexed.
In terms of UX, I can envision a pretty awesome three-coloured progress
bar, with colour 1 meaning "unwritten", colour 2 meaning "written, not
validated", and colour 3 meaning "written and validated". In fact this
would be an improvement over our current approach whether we multiplex or
not.
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Sun, Oct 2, 2016 at 6:26 PM, Andrew Scheller<EMAIL_ADDRESS>wrote:

Hmm, so if you're storing multiple checksum-checkpoints-along-the-way,
does that mean it'd be better to change it from:

Write the whole image
Read the whole image, verifying the "rolling checksum" at various
checkpoints

to:

Write a chunk of the image
Read that chunk of the image, calculating the rolling checksum
Write the next chunk of the image
Read that chunk of the image, calculating the rolling checksum

That way if the user had a 'bad' card they'd get notified earlier, rather
than having to wait for the whole image to be written first (and I guess
you could give them the options of 'Ignore verification error' or 'Abort
writing'). Only possible downside is that flipping back and forth between
writing and reading may be slower than just writing all in one go.
Would need some experimentation...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/735#issuecomment-251010247,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCA6tYTgk18U3nlY3qoJJIyQp1B6pks5qwFnBgaJpZM4KLI71
.

I'd need to measure this to be able to provide a sane answer. The possibilities are:

Retrieve checksums of the current data after X amount of blocks. This may mean that the first data that passed through the stream must be taken in consideration each time we want to calculate a checkpoint, which may slow things down, unless the checksum calculator stream provides good memoization support.

Create a custom transform stream that calculates a checksum for every block, as suggested by @lurch, but I worry that calculating per chunk will impose a noticeable overhead. Currently, calculating the checksum is one of the must CPU intensive tasks in the whole backend, but maybe we can calculate for an X amount of chunks instead, or find a way to this asynchronously in web workers or something along those lines.

@jviotti Are you sure about checksums having an overhead? On my system I get these numbers on a single core:

CRC: 280MB/s
sha256sum: 341MB/s
sha512sum: 507MB/s
md5sum: 520MB/s
sha1sum: 660MB/s

You can't possibly be handling data at a faster rate than this. Are you using node's crypto API that calls out to openssl?

Yeah, I'd always thought that calculating file checksums tended to be IO-bound rather than CPU-bound?
One problem I realised with switching between writing and reading after every 'chunk', is that you'd need to insert a 'sync' between writing and reading, to ensure that you read the data back from the disk rather than from the in-memory buffer, and multiple 'sync's could indeed slow down the entire process.
Probably still worth actually measuring though? ;)

And WRT calculating the 'rolling checksum' at various points - there's no need for memoization, you can just keep adding data to the checksum function, and it'll return the checksum of all the data seen so far. See e.g. https://docs.python.org/2/library/hashlib.html where you can keep calling update with additional data, and then simply get the digest at various points. Like this:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> m = hashlib.md5()
>>> m.update("Nobody inspects")
>>> m.hexdigest()
'3ef729ccf0cc56079ca546d58083dc12'
>>> m.update(" the spammish repetition")
>>> m.hexdigest()
'bb649c83dd1ea5c9d9dec9a18df0ffe9'
>>> m2 = hashlib.md5()
>>> m2.update("Nobody inspects the spammish repetition")
>>> m2.hexdigest()
'bb649c83dd1ea5c9d9dec9a18df0ffe9'

Unfortunately it seems like https://www.npmjs.com/package/md5 doesn't offer the same API.

Why would you need to calculate the "rolling" version to include previous data? You already have checked the stuff up until the current "block", so checking it again is unnecessary. Even if you needed to, you could just prepend the previous hash string to the new data before calculating the checksum.

I guess one advantage of a "rolling" checksum (not sure I've explained it very well) is that you can choose arbitrary-size chunks to verify, and you always have a valid checksum at each point; whereas if you only checksummed individual chunks, you'd be forced to verify each chunk individually.
e.g. if you take a 'rolling checksum' every 1 MB, if you wanted to you could instead only verify the checksums every 10 MB (or for just the whole file), and they'd still be valid. Kinda hard to explain in words...
So, using a code example calculating a 'rolling checksum' with a chunk-size of 4 characters:
>>> m = hashlib.md5()
>>> m.update('1111')
>>> m.hexdigest()
'b59c67bf196a4758191e42f76670ceba'
>>> m.update('2222')
>>> m.hexdigest()
'821f3157e1a3456bfe1a000a1adf0862'
>>> m.update('3333')
>>> m.hexdigest()
'95ebebaa68041c9faae8673241eaae1b'
>>> m.update('4444')
>>> m.hexdigest()
'7b8802b5aa06e55f70c9d8711213364b'
>>> m.update('5555')
>>> m.hexdigest()
'fa06fbd962ca90ba3a574486c17b8d00'
>>> m.update('6666')
>>> m.hexdigest()
'7525aecccfaf9f40d53950df929256e6'

Now when we want to verify the data, we could obviously do it in 1-chunk steps as above, or we could do it in 2-chunk steps:
>>> m2 = hashlib.md5()
>>> m2.update('11112222')
>>> m2.hexdigest()
'821f3157e1a3456bfe1a000a1adf0862'
>>> m2.update('33334444')
>>> m2.hexdigest()
'7b8802b5aa06e55f70c9d8711213364b'
>>> m2.update('55556666')
>>> m2.hexdigest()
'7525aecccfaf9f40d53950df929256e6'

...or 3-chunk steps:
>>> m3 = hashlib.md5()
>>> m3.update('111122223333')
>>> m3.hexdigest()
'95ebebaa68041c9faae8673241eaae1b'
>>> m3.update('444455556666')
>>> m3.hexdigest()
'7525aecccfaf9f40d53950df929256e6'

...or even in a single 6-chunk step:
>>> m6 = hashlib.md5()
>>> m6.update('111122223333444455556666')
>>> m6.hexdigest()
'7525aecccfaf9f40d53950df929256e6'

...and at each point we always get the correct checksums :-)

the point makes sense to me, but is the option to decide later useful? I
would assume we know what we want to do at the time we're sampling the
checksums. no?
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Mon, Oct 3, 2016 at 6:24 AM, Andrew Scheller<EMAIL_ADDRESS>wrote:

I guess one advantage of a "rolling" checksum (not sure I've explained it
very well) is that you can choose arbitrary-size chunks to verify, and you
always have a valid checksum at each point; whereas if you only checksummed
individual chunks, you'd be forced to verify each chunk individually.
e.g. if you take a 'rolling checksum' every 1 MB, if you wanted to you
could instead only verify the checksums every 10 MB (or for just the whole
file), and they'd still be valid. Kinda hard to explain in words...
So, using a code example calculating a 'rolling checksum' with a
chunk-size of 4 characters:

m = hashlib.md5()>>> m.update('1111')>>> m.hexdigest()'b59c67bf196a4758191e42f76670ceba'>>> m.update('2222')>>> m.hexdigest()'821f3157e1a3456bfe1a000a1adf0862'>>> m.update('3333')>>> m.hexdigest()'95ebebaa68041c9faae8673241eaae1b'>>> m.update('4444')>>> m.hexdigest()'7b8802b5aa06e55f70c9d8711213364b'>>> m.update('5555')>>> m.hexdigest()'fa06fbd962ca90ba3a574486c17b8d00'>>> m.update('6666')>>> m.hexdigest()'7525aecccfaf9f40d53950df929256e6'

Now when we want to verify the data, we could obviously do it in 1-chunk
steps as above, or we could do it in 2-chunk steps:

m2 = hashlib.md5()>>> m2.update('11112222')>>> m2.hexdigest()'821f3157e1a3456bfe1a000a1adf0862'>>> m2.update('33334444')>>> m2.hexdigest()'7b8802b5aa06e55f70c9d8711213364b'>>> m2.update('55556666')>>> m2.hexdigest()'7525aecccfaf9f40d53950df929256e6'

...or 3-chunk steps:

m3 = hashlib.md5()>>> m3.update('111122223333')>>> m3.hexdigest()'95ebebaa68041c9faae8673241eaae1b'>>> m3.update('444455556666')>>> m3.hexdigest()'7525aecccfaf9f40d53950df929256e6'

...or even in a single 6-chunk step:

m6 = hashlib.md5()>>> m6.update('111122223333444455556666')>>> m6.hexdigest()'7525aecccfaf9f40d53950df929256e6'

...and at each point we always get the correct checksums :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/735#issuecomment-251104550,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCMaaC_0VKsK4y2z10IVe2PwJFS5gks5qwQIVgaJpZM4KLI71
.

Yeah... I still don't understand :laughing: Wouldn't you know the granularity with which you want to verify the drive before you start?

I didn't mean to suggest you should change the granularity with which you verify the drive (unless you wanted to automatically vary it depending on the read-speed of the drive, to keep the GUI responsive), I was just giving an example of a 'possible' advantage of 'rolling checksums' as opposed to individually checksumming every chunk. AFAIK there'd be no difference in 'cost' of either rolling or separate checksums, but a (very small) practical advantage of a rolling checksum is that the value of the final rolling checksum will also match the checksum of the whole image, as generated by a standalone tool such as md5sum.

Hmmm, looking back at the earlier messages in the thread, I guess the disadvantage of a rolling checksum is that once it's failed, you can't tell if any subsequent chunks are okay. Whereas with individual checksums, you can tell which chunks are 'good' and which are 'bad'. Apologies for dragging this thread OffTopic ;-)

As far as I've seen, CPU usage goes up a lot when computing the CRC32 checksum, which reaches almost ~90% in my Macbook Pro when doing multi writes (like 8 at the same time). We could mitigate this in part by only calculating the checksum once during the writing phase (instead of re-calculating in each child process), but we can't get away with it during the validation phase.
Anyway, it may be the CRC32 implementation we're using (its a third party module), but I believe MD5 is much less intensive (and we can use NodeJS crypto module directly), and we're actually looking forward to replace CRC32 with it (https://github.com/resin-io/etcher/issues/643).

We could mitigate this in part by only calculating the checksum once during the writing phase (instead of re-calculating in each child process)

Doh, I keep forgetting that for disk images that aren't extended archives, Etcher has to calculate the hashes itself, rather than simply reading the list of already-computed checksums from the metadata :blush:

@jviotti why not using sha1sum? It seems to be the fastest. The reason CRC is slow currently is because it's implemented in JS[1] whereas sha1sum is implemented in C.
Regarding multiplexing of read/writes, we can try using the O_DIRECT and O_SYNC flags when opening the disk. But also, even assuming that there is a cost of switching from reading to writing you can amortise the overhead with the expense of RAM. So basically you can keep the last 50MB in memory and instead of doing write 1 block, sync, read 1 block you could do write 50MB, sync, read 50MB. The more memory you sacrifice, the less syncs you'll do. That said, if we assume the source slowness of sync calls is due to the speed of the medium, then it doesn't matter that it's slow because you'll have to write X amount of data anyway. Either you'll do a lot of small and relatively fast syncs or a big and slow at the end.
@lurch what you're proposing regarding "rolling hashes" (that I'll call length extension here to not confuse them with rolling hashes which are a bit different) doesn't have a big advantage regarding memory usage but I think it's the right approach. All the common hash functions (MD5, SHA1, SHA2) are based on the Merkle–Damgård construction. This construction allows length extension by design[2].
So say we have an image with N blocks, memoisation of single block hashes would require storing the following in memory:
hashes = [
    hash(blocks[0..1]),
    hash(blocks[1..2]),
    ...
    hash(blocks[N-1..N])
]

Calculating an overall hash while keeping checkpoints would require storing the following in memory:
hashes = [
    hash(blocks[0..1]),
    hash(blocks[0..2]),
    ...
    hash(blocks[0..N])
]

So both cases need N * sizeof(hash) bytes of memory and both cases need the same CPU because hash(blocks[0..M]) can be rewritten as length_extend(hash(blocks[0..M-1]), blocks[M-1..M]), assuming it's one of MD5, or SHA families. This means that each step always processes a constant sizeof(block) amount of data.
Practically I think the second approach will also be faster since there won't be a setup cost for each block (no mallocs etc) and it will also end up with an overall hash which could be useful for other purposes.
[1] https://github.com/brianloveswords/buffer-crc32
[2] https://en.wikipedia.org/wiki/Merkle–Damgård_construction#Security_characteristics

Practically I think the second approach will also be faster since there won't be a setup cost for each block (no mallocs etc) and it will also end up with an overall hash which could be useful for other purposes.

I don't follow. Why do you need to store all the hashes in memory? Why not just keep them one at a time? And how does the "length extension" method make a difference?
If you want Etcher to be able to report the specific sectors/blocks which didn't match, you can't do the "rolling hashes" thing: if you do that, all comparisons after one corrupt one will also appear corrupt, because the ones from the live disk would include the hashed data from previous checks.

I don't follow. Why do you need to store all the hashes in memory? Why not just keep them one at a time?

If you do 2 passes, one that writes the whole image and a second that validates the image, then the second needs to have access to all the checkpoints so that it can accurate pinpoint where the error occurred. This is true for both methods.
I also realised that if we don't do 2 full passes but instead we do either write 1 block,sync,read 1 block or write 50MB,sync,read 50MB there is no point messing with hashes at all. We can just keep a window of data in memory and do a byte-by-byte comparison.

And how does the "length extension" method make a difference?

As mentioned, there is no difference in RAM or CPU usage between the two methods but I expect length extension to be faster because you won't have to initialise/allocate a new Hash object every time.

If you want Etcher to be able to report the specific sectors/blocks which didn't match,

I think the requirement can be relaxed and have ether report the first corrupt sector and then bail out. If what we're after is full reporting even after a corrupt sector then yes, the length extension method doesn't work.

I'm still not sure I understand, but I could easily just be too tired.

I think the requirement can be relaxed and have etcher report the first corrupt sector and then bail out. If what we're after is full reporting even after a corrupt sector then yes, the length extension method doesn't work.

For me, I think it makes a big difference whether it's only one corrupt sector or whether the whole thing is defective. If it's just a small section, I would probably (and have in the past) ignore it anyway if I don't have a new card handy, because the system will probably run fine for the most part, which is all I may need for testing things. If the whole thing is broken, there isn't any point in continuing without a new card.
My main motivation here is to get more information on these failures. I often have people complaining of Etcher failing at the validation stage, and I would like to know whether it seems more like a validation bug or real corruption (I suspect the former, but I really have no way of knowing). This use case isn't a main target, though. I don't actually care how it is implemented internally as long as it either helps me solve that problem from my end or helps users solve it themselves.

why not using sha1sum? It seems to be the fastest.

I wonder if that's only true on certain CPU arch-es? I always thought that MD5 was faster than SHA1, but I think some CPUs added special hardware-cryptography instructions?
(did some searching and http://stackoverflow.com/questions/2722943/is-calculating-an-md5-hash-less-cpu-intensive-than-sha-family-functions and http://stackoverflow.com/questions/20692386/are-there-in-x86-any-instructions-to-accelerate-sha-sha1-2-256-512-encoding )

That said, if we assume the source slowness of sync calls is due to the speed of the medium, then it doesn't matter that it's slow because you'll have to write X amount of data anyway. Either you'll do a lot of small and relatively fast syncs or a big and slow at the end.

True. I wonder if it depends what kind of buffering / write-strategy the internal disk controller uses though (the FTL on SD cards, etc.) ? Again, only way to be sure is via experimentation.

I also realised that if we don't do 2 full passes but instead we do either write 1 block,sync,read 1 block or write 50MB,sync,read 50MB there is no point messing with hashes at all. We can just keep a window of data in memory and do a byte-by-byte comparison.

Yeah, I guess on modern PCs there's no harm in allocating two 50 MB memory buffers, but in the "olden days" you'd have done "write 50MB in 1MB chunks calculating the cumulative hash as you go, sync, read 50MB in 1MB chunks calculating the cumulative hash as you go" needing only a single 1MB buffer.
I agree that having Etcher being able to detect which blocks failed will probably be useful, and ties in nicely with @alexandrosm 's three-coloured progress bar, which we could extend to 4 colours: unwritten, written but not validated, written and validated good, written and validated bad. And if we do do that, I still think the user should be given the option to "Abort" or "Ignore and Continue" when the first error is detected.

I expect length extension to be faster because you won't have to initialise/allocate a new Hash object every time.

I suspect that any differences in hash-speed (assuming you're using a C implementation rather than pure JS) will probably still be dwarfed by the slowness of actually writing or reading data to/from disk?

@jviotti why not using sha1sum?

The problem is that users still want to see a checksum on completion, and md5 is by far the most popular one in the image writing world. This also means that even if we calculate checksums per block, we still have to calculate a checksum for the whole thing as well to present to the user. This can be separated though, so we calculate an MD5 for the whole thing, and use something like sha1sum in a per-block fashion.

I think the requirement can be relaxed and have etcher report the first corrupt sector and then bail out. If what we're after is full reporting even after a corrupt sector then yes, the length extension method doesn't work.

That's no different to the current situation, where we can tell if writing failed or not, but can't point out specific blocks.
If we decide to calculate a sha1sum per block, this means we need to store ~40 bytes per block in an object to be able to do the comparison later on. This is not a big deal, but if we mix the validation and writing as a single step, we can get away with only storing the checksums we immediately need, although in this case, its probably easier to just to byte by byte comparison.
Is there any benefit on having writing and validation as two separate steps? If we can't think of any, switching to a single pass seems like the way to go, since it simplifies a lot of things.

Is there any benefit on having writing and validation as two separate steps? If we can't think of any, switching to a single pass seems like the way to go, since it simplifies a lot of things.

I can't think of much. Maybe if someone doesn't care about validation, they wouldn't want the write operation being slowed down by the validation, but that can be solved with a "perform validation" checkbox in the settings menu.

Yeah, the only issue is some presumed drop in performance, but I also think
we should take the hit and do the right thing.
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Tue, Oct 4, 2016 at 6:31 AM, Wasabi Fan<EMAIL_ADDRESS>wrote:

Is there any benefit on having writing and validation as two separate
steps? If we can't think of any, switching to a single pass seems like the
way to go, since it simplifies a lot of things.
I can't think of much. Maybe if someone doesn't care about validation,
they wouldn't want the write operation being slowed down by the validation,
but that can be solved with a "perform validation" checkbox in the settings
menu.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/735#issuecomment-251388256,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCMo-7k66D3VpiOvbA9GojjK7kkbLks5qwlVIgaJpZM4KLI71
.

The problem is that users still want to see a checksum on completion, and md5 is by far the most popular one in the image writing world.

As an anecdote, raspberry pi downloads use SHA1[1].
Full image checksums aren't very useful for the question "Is the image correctly written to disk?". You can do a byte-by-byte or any of the previous techniques mentioned in this thread. The full image checksum is useful to verify that the original file downloaded was not corrupted. So it doesn't have to be tied to the writing process, you can spawn a thread in the background to do the checksum.

Is there any benefit on having writing and validation as two separate steps? If we can't think of any, switching to a single pass seems like the way to go, since it simplifies a lot of things.

To me two separate steps sound simpler implementation-wise. And the more I think about it the more sense the byte-by-byte approach makes. Why doesn't etcher just re-read the file from disk and compare the sectors byte-by-byte?
[1] https://www.raspberrypi.org/downloads/raspbian/

For one, we're keeping the implementation compatible with a "download
straight to the card" future. Which means streaming type reasoning for most
things.
Dare I say we have gone completely off topic? Maybe a new issue is in order?
On 4 Oct 2016 1:39 p.m., "Petros Angelatos"<EMAIL_ADDRESS>wrote:
The problem is that users still want to see a checksum on completion, and
md5 is by far the most popular one in the image writing world.
As an anecdote, raspberry pi downloads use SHA1[1].
Full image checksums aren't very useful for the question "Is the image
correctly written to disk?". You can do a byte-by-byte or any of the
previous techniques mentioned in this thread. The full image checksum is
useful to verify that the original file downloaded was not corrupted. So it
doesn't have to be tied to the writing process, you can spawn a thread in
the background to do the checksum.
Is there any benefit on having writing and validation as two separate
steps? If we can't think of any, switching to a single pass seems like the
way to go, since it simplifies a lot of things.
To me two separate steps sound simpler implementation-wise. And the more I
think about it the more sense the byte-by-byte approach makes. Why doesn't
etcher just re-read the file from disk and compare the sectors byte-by-byte?
[1] https://www.raspberrypi.org/downloads/raspbian/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/735#issuecomment-251506376, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABLUCGrGuHPLGTH7_QsAfY7mmK1zSv-1ks5qwrmKgaJpZM4KLI71
.

For one, we're keeping the implementation compatible with a "download
straight to the card" future. Which means streaming type reasoning for most
things.

You can always write a temp file with the download (even gzipping it) on disk that will get cleaned up at the end.

Dare I say we have gone completely off topic? Maybe a new issue is in order?

We're definitely not off topic since the specific implementation of validation is tied to the behaviour of "validation failed" message. Maybe we can change the title of this issue to better describe the thread, but the messages in the thread have been consistent on the subject.

Well, what I mean is that my original suggestion was to improve the wording.
I love the conversation and insights shared here, but it comes down to a much deeper reconsideration of the whole writing/validating process, and I want to make sure we don't hold off improving the wording until we've retooled the entire algorithm.

It would be nice to avoid needing X GB of temporary space if we can, no?

Sparing some storage for the few minutes of writing (and only in the case of a network stream) could be justified, especially if it is compressed.
There is also an important advantage. If etcher is writing X number of cards in parallel, during validation it will be handling X * max_read_speed amounts of data. Storing the temp file allows for virtually zero overhead checking (versus calculating X checksums in parallel). Thus making the IO subsystem the only bottleneck and etcher a very good CPU citizen.

You can still do the amortising technique though, but one order of magnitude bigger. Write 1GB, sync, validate 1GB.

Replying to a few comments at once...

The problem is that users still want to see a checksum on completion, and md5 is by far the most popular one in the image writing world.

Unless the user is using an extended archive, we have no way of knowing whether they want the checksum displayed as MD5 or SHA1. So if (as I suspect) the hashing time is minor compared to the actual disk reading/writing time, we could probably calculate both at the same time, and display them both to the user at the end? (or maybe have it selectable in the options, defaulting to MD5)

The full image checksum is useful to verify that the original file downloaded was not corrupted.

Yeah, there's obviously a difference between the download (.zip) checksum, and the image (.img) checksum.
Hmmm, perhaps if Etcher is given a 'normal' zipfile (i.e. not an extended archive) containing e.g. some-disk-image.img, maybe it could also look for each variant of some-disk-image.img.md5, some-disk-image.img.md5sum, some-disk-image.md5, some-disk-image.md5sum, some-disk-image.img.sha1, some-disk-image.img.sha1sum, some-disk-image.sha1, some-disk-image.sha1sum also being in the zipfile, and using the contents of that as the image checksum?

To me two separate steps sound simpler implementation-wise.

Unless (as I pointed out earlier) we do step-by-step checksumming, and allow the user to bail out on the first checksum error, instead of having to wait until the entire image is written.

Sparing some storage for the few minutes of writing (and only in the case of a network stream) could be justified, especially if it is compressed.

IMHO using a temporary cache file should be optional, for people with low diskspace. OTOH, downloading the entire image to a local cache, allows you to checksum the downloaded image (in case of any download errors) before you start writing it. Swings & roundabouts ;-)

So... coming back to this old issue again. I've just done some benchmarking on my laptop (i5-5200U CPU @ 2.20GHz) and (assuming the JS hashing algorithms aren't written incredibly inefficiently), it seems I was right in my assumption that the time to do the actual hash-calculations will be vastly swamped by anything else:
$ dd if=/dev/urandom of=random_1GB_file.img bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 101.898 s, 10.5 MB/s
$ time crc32 random_1GB_file.img
e5866893

real	0m3.742s
user	0m1.348s
sys	0m0.380s
$ time crc32 random_1GB_file.img
e5866893

real	0m3.213s
user	0m1.368s
sys	0m0.360s
$ time md5sum random_1GB_file.img
17f552abc2960e656478add76ef50fc5  random_1GB_file.img

real	0m3.572s
user	0m2.176s
sys	0m0.352s
$ time md5sum random_1GB_file.img
17f552abc2960e656478add76ef50fc5  random_1GB_file.img

real	0m3.043s
user	0m2.204s
sys	0m0.376s
$ time sha1sum random_1GB_file.img
8bcf866254b168a606c0b33cec322e402f567a5e  random_1GB_file.img

real	0m6.808s
user	0m3.636s
sys	0m0.244s
$ time sha1sum random_1GB_file.img
8bcf866254b168a606c0b33cec322e402f567a5e  random_1GB_file.img

real	0m6.135s
user	0m3.416s
sys	0m0.384s
$ time sha256sum random_1GB_file.img
a6673312b856d36906424d903e849e14ef7c736e3c9e8874e74c4a28e893ad67  random_1GB_file.img

real	0m7.336s
user	0m6.388s
sys	0m0.360s
$ time sha256sum random_1GB_file.img
a6673312b856d36906424d903e849e14ef7c736e3c9e8874e74c4a28e893ad67  random_1GB_file.img

real	0m7.068s
user	0m6.372s
sys	0m0.332s

I've sent a PR fixing the original concern of this issue (the validation error message). There were some cool ideas mentioned here, however I believe that taking the effort to say to the user how much of the flash failed is not critical and probably not worth it, since as a user, even if it fails 1%, I wouldn't trust the drive enough to use it.

since as a user, even if it fails 1%, I wouldn't trust the drive enough to use it.

Oh? Is it possible that it might be only a 'temporary' failure, and attempting to write the image a second time might work? Or do you think any errors means the whole card is unusable?

Or do you think any errors means the whole card is unusable?

It might be fine in most cases, or the error is too subtle to be noticed (or the file-system is able to cope with it at runtime), however my point is that telling the user exactly what blocks failed, etc is not very useful from its point of view.

I'd argue that the feature isn't necessarily useful until you are attempting to figure out what does work. Knowing that a particular block failed isn't particularly useful in itself, but if you determine that different images fail on the same block for a given drive, that tells you a lot -- and gives you a logical next step for finding a configuration that does work. Likewise, if it fails at the same point regardless of the drive used, you'd start to suspect the image file and/or Etcher itself.

I wasn't suggesting that if e.g. only 0.5% of a drive failed validation that it'd be worth trying booting the drive anyway. (I was only referring to percentages with my previous comment, not necessarily knowing which blocks failed - which is more of an advanced-user feature)
I was more suggesting that if 0.5% of a drive failed to flash that it might indicate only a temporary writing failure, and so maybe attempting to rewrite the same image to the same drive may result in 100% validation on a second attempt. Whereas if 50% of the drive failed verification, it's more likely that the whole drive should be thrown in the bin.
But of course that's very much a judgement call, and the safest option is to simply recommend trying a different drive, as we do now.

Let's bring @jhermsmeier to the loop (though we should probably open a new issue for this). He's re-architecting our drive flashing engine, so it would be nice for him to keep this in mind, and see if its feasible to get a percentage of failed blocks somehow (then we can figure out what to do with the data).

@jviotti @lurch @WasabiFan I made a new issue to track this: https://github.com/resin-io/etcher/issues/1074

I continue to get errors on any SD card running Etcher on a Mac.  From what I can tell, the search feature in OSX is writing information on the card as well, and I think the CRC checks are detecting them, and determining that the write failed.
Is there any workaround for this?

@cdeerinck Please create a new issue, describing your problem in detail, and someone from the Etcher team will look into it :)


Could we configure this to choose to open NA lodestone links rather than EU lodestone links?

Would be nice to either read from what region we're connected to, or maybe just make it a dropdown in the settings. It's not a super huge deal but timezones are hard. Let's get rid of timezones.

You can already see the exact time in the plugin itself, it is auto converted to your timezone that the pc is running on :)

Uh huh, it sure does. I guess that's good enough then!


New PR to test SignCLA at 2020-05-18 11:09:13.173274086 -0700 PDT m=+2.979760158

Body of PR

corp-pass-fork-staging: test succeeded


SPARKC-682 Support mixed case Cassandra column names when performing DF join

SPARKC-682 Support mixed case Cassandra column names when performing a DF join operation
Description
When performing a DF join operation, Cassandra column names with mixed case is not handled properly leading to a column not found exception.
How did the Spark Cassandra Connector Work or Not Work Before this Patch
See description above. Without this patch, column names with mixed case will not be found during a DF join operation.
General Design of the patch
How the fix is accomplished, were new parameters or classes added? Why did you
pursue this particular fix?
Per @jtgrabowski's suggestion, implement .asCql(true) to the join operation.
Fixes: SPARKC-682
How Has This Been Tested?
Almost all changes and especially bug fixes will require a test to be added to either the integration or Unit Tests. Any tests added will be automatically run on travis when the pull request is pushed to github. Be sure to run suites locally as well.
Checklist:

[x] I have a ticket in the OSS JIRA
[x] I have performed a self-review of my own code
[x] Locally all tests pass (make sure tests fail without your patch)

Hi Jaroslaw, as you have merged this to master, are you planning to have this fix in a release (3.2.1) any time soon? is there any workaround for this issue without this release? I mean is there any way to perform the join to enforce the camel case in 3.2.0 version?


Fix get_latest_version returning version < 0.

Description
Fix an edge case in get_latest_version.
Related Issue(s)

closes #312

Documentation

Thanks @zijie0 , looks like you are getting very proficient with Rust already ;)

@houqp Thanks, I'm feeling good to learn a new language through contributing to a real project. And we are using delta-rs in our data platform heavily to support high performance data consumption. Great job!


gdrive: delete connector if error creating it

Description

Risk

Deploy Plan
https://dust.tt/poke/655a9f073b
DELETE FROM connectors WHERE id=1134;

https://dust.tt/poke/38b1c53fb4
DELETE FROM connectors WHERE id=1128;

Deploy connectors

Also
dust-connectors=> DELETE FROM google_drive_configs WHERE "connectorId"=1134;
DELETE 1
dust-connectors=> DELETE FROM google_drive_configs WHERE "connectorId"=1128;
DELETE 1

cc @flvndvd

Email shipped


Non visual properties

Added meta property to Cesium3DTileStyle to evaluate non-visual properties.
ex.
"style" : {
    "meta" : {
        "description" : "Hello, ${FeatureName}"
    }
}

var str = style.meta.description.evaluate(feature);

If meta is not supplied, or something other than the meta properties are supplied, (such as "meta" : 1), meta defaults to {}.
Added tests and updated sandcastle example.

If meta is not supplied, or something other than the meta properties are supplied, (such as "meta" : 1), meta defaults to {}.

In general, don't "fix" bad input like this.
undefined can default to {}
1 is invalid.

Can you also open a PR to update the 3D Tiles spec?

1 is invalid.

So throw a developer error?

Until we have full validation worked out, if it is defined, just assume it is an object like we do elsewhere.

OK

OK, I made the updates and also added the examples I used in the spec to the tests.

Looks good, open a new PR for any new edge cases we find in https://github.com/AnalyticalGraphicsInc/3d-tiles/pull/85.


Contradicting FAQ and README

Description
The FAQ states that "Slack Developer Kit for Python does not provide access to the RTM" while this repo's README states that it "supports ... Real Time Messaging (RTM) API".
Should the FAQ be updated or is the RTM support implementation not finished yet?
What type of issue is this? (place an x in one of the [ ])

[ ] bug
[ ] enhancement (feature request)
[ ] question
[x] documentation related
[ ] testing related
[ ] discussion

Requirements (place an x in each of the [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've searched for any related issues and avoided creating a duplicate issue.

@revolter good catch! I'll open a PR for this, unless you'd like to? 😎

Is the FAQs page open source?

@revolter yep! https://github.com/slackapi/python-slackclient/blame/master/docs-src/faq.rst#L11


DevTools becomes useless as soon as I refresh the page

Version
6.0.8
Browser and OS info
Chrome: Version 98.0.4758.80/ Mac: Version 12.1
Steps to reproduce
Whenever I refresh the page the Vue devtools becomes completely unusable, i.e don't see any of my components/ vuex store etc.
What is expected?
DevTools should work fine even after page reload
What is actually happening?
DevTools stop working on page-refresh

Getting the following error on console:
backend.js:2205 Uncaught TypeError: Cannot read properties of null (reading 'instanceMap')
at Bridge. (backend.js:2205:55)
at Bridge.emit (backend.js:9922:5)
at Bridge._emit (backend.js:6557:12)
at backend.js:6483:42
at Array.forEach ()
at backend.js:6483:18
at listener (backend.js:13476:13)

Duplicate of #1711


Edit pass, updates to formatting, and style changes in Learn index

Hi Randall, what happened here?

The branch was a bit of a mess so I archived the updated file and will make a new branch with this change and the additional updates we discussed


Use first_published_to_production instead of publish_date for sorting new courses

Pre-Flight checklist

[ ] Screenshots and design review for any changes that affect layout or styling
[ ] Testing

[ ] Code is tested
[ ] Changes have been manually tested

What are the relevant tickets?
Closes #1188
What's this PR do?
Switches the sorting of the api query for new courses from publish_date to first_published_to_production
How should this be manually tested?
In an incognito browser window (not logged in to studio), go to http://localhost8043/api/websites and verify that the list of courses is in descending order by the first_published_to_production field.
Set OCW_STUDIO_BASE_URL=http://localhost:8043 in your .env file for ocw-hugo-themes, run npm run start:www, and check that the list of new courses is the same as in the response above.
Where should the reviewer start?
(Optional)
Any background context you want to provide?
(Optional)
Screenshots (if appropriate)
(Optional)
What GIF best describes this PR or how it makes you feel?
(Optional)

Codecov Report

Merging #1191 (314dfdf) into master (0c0f3e6) will increase coverage by 4.46%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1191      +/-   ##
==========================================
+ Coverage   90.92%   95.38%   +4.46%
==========================================
  Files         210       98     -112
  Lines        8398     2689    -5709
  Branches     1441      554     -887
==========================================
- Hits         7636     2565    -5071
+ Misses        661      124     -537
+ Partials      101        0     -101

Impacted Files
Coverage Δ

websites/factories.py

websites/serializers.py

websites/views.py

content_sync/utils.py

videos/constants.py

main/settings.py

gdrive_sync/conftest.py

main/apps.py

users/utils.py

news/urls.py

... and 102 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c0f3e6...314dfdf. Read the comment docs.


404 on links on bottom of intro page

The links on the bottom of the introduction page (https://mockbee.netlify.app/docs/introduction/) return 404, I hope the changes I provided go to the right link, else please correct any mistakes I have made, thanks,
Onkar Deshpande

The solution was found and implemented in another pull req , cz I didnt know how to update this one.
Apologies for spam.


Bug when reloading a page

When using fullyEnterViewport and partiallyExitViewport together, they both trigger one after another if the user is scrolled into the watched element and then refreshes/reloads the page. Refreshing the page causes the browser to scroll automatically to the same scroll position.
Please watch the following video
https://www.youtube.com/watch?v=Yah362YmCvk&feature=youtu.be
Note: In the codepen you'll have to scroll the output area so that the element with the red background covers the entire view port and then right click the frame and select 'reload frame' (in chrome)
The javascript which initializes scrollMonitor is at the bottom.
The codepen used in the video is at the following link.
http://codepen.io/anon/pen/epYmxW

I see, the partially left event should not fire. I'll see if I can look this weekend. Thanks for a great bug report.

+1
Still happening.
Has anyone found a sensible solution for this?

+1


Fix skipped deployment task after ci_make/ci_scripts switch

This nit was introduced with the merge of [1]. Instead of skipping the task when the make target is not defined, we should just run make with no target (i.e. default target) to keep previous behaviour.
[1] https://github.com/openstack-k8s-operators/ci-framework/pull/653/files#diff-0af62e35d536a23bbb025c5e1ba37906f6a187ad7e44ef4d6af6b4674e61b5e3
As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running
[x] Appropriate documentation exists and/or is up-to-date:

/lgtm

/approve


"Download" file not working.

Describe the bug
When you open Remote File Manager, right clicking a file and choosing Download won't work.
System

Server OS: Windows 10 Home x86-64
Client OS: Windows 10 Pro x86-64
Quasar Version: latest stable as of 30/08/2021 (v1.4.0)

Need more information. Any error message(s) and if so post a screenshot. On same network or internet? Both Client and Server have proper permissions and or ports allowed? Client running as guest or admin? Have you tested using loopback, AV and Firewall temporarily disabled? Using a VM or normal system?

yeah, I have been having the same issue lol

Where is the download file path though?

Your browser or Anti Virus is blocking it. I just download the zip but needed to go to my downloads tab (I use Brave browser) to "Keep Dangerous File". It's not dangerous do a search using our search function to learn more.

@BurntDog I think @DannyBrock123 is talking about the file download in remote file manager. What is shown in the transfers tab? There is also a button to open the download directory on your PC.

@BurntDog I think @DannyBrock123 is talking about the file download in remote file manager. What is shown in the transfers tab? There is also a button to open the download directory on your PC.

Lol, oops sorry about that.

Lol, oops sorry about that.

We are too used to the "download contains virus"-question, every week. :smile:

Ok, thanks guys. I will see if I can find it


Think about to fix the issues found by adapter checker

I am an automatic service that looks for possible errors in ioBroker and creates an issue for it. The link below leads directly to the test:
https://adapter-check.iobroker.in/?q=https://raw.githubusercontent.com/inpex/iobroker.testprojekt

[ ] [E003] Repository must have name ioBroker.adaptername, but now io"b"roker is in lowercase
[ ] [E004] No adapter name found in URL: https://raw.githubusercontent.com/inpex/iobroker.testprojekt/main
[ ] [E020] Name of adapter in package.json must be lowercase and be equal to "iobroker.". Now is "undefined"
[ ] [E019] Invalid repository URL: https://github.com/inpex/ioBroker.testprojekt. Expected: git+https://github.com/inpex/iobroker.testprojekt.git
[ ] [E103] common.name in io-package.json must be equal to "". Now is testprojekt
[ ] [E145] No news found for actual version 0.0.2
[ ] [E200] Not found on npm. Please publish
[ ] [E801] No repository about text found. Please go to "https://github.com/inpex/iobroker.testprojekt", press the settings button beside the about title and add the description.
[ ] [E802] No topics found in the repository. Please go to "https://github.com/inpex/iobroker.testprojekt", press the settings button beside the about title and add some topics.

I have also found warnings that may be fixed if possible.

[ ] [W400] Cannot find "" in latest repository

Thanks,
your automatic adapter checker.
P.S.: There is a community in Github, which supports the maintenance and further development of adapters. There you will find many experienced developers who are always ready to assist anyone. New developers are always welcome there. For more informations visit: https://github.com/iobroker-community-adapters/info

Thanks, that all bugs have been fixed.


alfred --new assists the developer to create a new command

alfred --new trigger a wizard to create a new command. The wizard guides the developper in the process.
alfred --new

What command do you want to create ?
$ ci

What does the `ci` command do [optional] ?
$

In which module do you want to create the `ci` command?
$ ci

Do you want to create the following `ci` command in `alfred/ci.py` [y, n]?

>>> @alfred.command('ci', help="")
>>> def ci():
>>> 	alfred.run("echo 'ci command not implemented yet'")

implement in #66


Adding ability to create html snippets - instead of just entire document

I've added the top-level "snippet()" function that enables creating blocks of html/dom elements that aren't an entire html page. This will allow people to create templates and partial sections of html documents using this library. This could be really useful for Singe Page Applications that want to use this library.
I also put together some basic overview information In the readme.md file. I hope you like it!

Hi @orangy, I have updated the implementation based on your comments. One thought I had: instead of using the term "snippet" we could also use "partial" So the function would become "partial()" and the class would become HtmlPartial. But I'm fine with the naming as it is now too.


When icon-size is lower than the default of 48, icon sizes start shrinking with a lot of free space still available

Describe the bug
When -i is lower than the default (e.g. 24), icons start reducing sizes too early, even though a lot of space is still available
To Reproduce
Steps to reproduce the behavior:

Set icon size to something lower than 48 px
Spawn a bunch of different windows
See that icon sizes start reducing size with the same number of active windows as it would with 48 px, even though there's still a lot of available space in the dock
See error

Expected behavior
Icons should only start shrinking when there actually is no space left on the dock
Screenshots
With icon size of 48 (default), icons start shrinking only when there is no space left on the dock. Works as expected:

With icon size of 24, icons shrink even though a lot of space is still available:

Desktop (please complete the following information):

Linux distribution: Arch
go version: 2:1.23.2-1
gtk-layer-shell version: 0.9.0-1

Additional context
Add any other context about the problem here.

It doesn't happen to me, unless I use the -f argument.

yes, I'm using -f. sorry I should have mentioned it 😅

So the dock is forced to take the full screen height, regardless of the icon size.

I just tested without the -f. Icons still shrink earlier than they should (in my opinion), as there is still space for the dock

The algorithm that scales icons is quite simple and arbitrary, but for now I have no better idea on how to change it to fit different initial icon sizes.
	divider := 1
	if len(allItems) > 0 {
		divider = len(allItems)
	}

	// scale icons down when their number increases
	if *imgSize*6/(divider) < *imgSize {
		overflow := (len(allItems) - 6) / 3
		imgSizeScaled = *imgSize * 6 / (6 + overflow)
	} else {
		imgSizeScaled = *imgSize
	}

I'm sure it would represent way more work, but in my opinion the best way to deal with too many apps in the dock would be scrolling rather than the icons shrinking.
For now, maybe add an option to disable this functionality entirely, if it's not too much work? It's distracting that the icons keep resizing, and as you mentioned the icon size also has an effect on the bar/dock's thickness which is even more distracting 🙁.


Add CloudWatch alarm for lambda function failure during web-scraping

Added this functionality, tested and confirmed to work.


fix(user): correctly configure the issuer_account claim when users ar…

…e signed by a signing key

Closing in favour of #53


Add $_ and $. to list of variables you can use with --output

We should probably note that $_ is unchomped, or change the behavior so that $_ is passed to --output as a chomped string.

Ultimately, I think we should whitelist the $ vars that we allow to be used in --output.  I see no reason to allow $] or similar in output.

I'm not opposed to that, but how do we go about doing that?

I think that we want to not do eval on --output, and instead do the substitutions ourselves.

That really limits what you can do with --output, though; for issue #345 I devised a hack that did something like --output="@{chomp; [length() > 250 ? 'too long' : $_]", and I think being able to run some code in there is really handy.

Shoot, I'd forgotten about those.  Mark Fowler has a bunch of --output magic here, too: http://www.perladvent.org/2014/2014-12-21.html

But is that capability overdoing ack's capabilities?  That could just have easily been done like so:
ack 'https?://\S+' --output='$&' | perl -MLWP::Simple -e'......'

That's a good point; maybe we shouldn't make ack super powerful and rely on other shell scripts to deliver that power.  My biggest concern is backwards compatability.

I'm OK with breaking backward computability in the name of locking down the overly-wide-open door of eval.


Use docopt for argument parsing

This commit switches bindgen over to using the docopt crate for argument parsing
instead of manual argument parsing. This required two notable changes in the
arguments and flags style:

All flags of the form -foo are now of the form --foo.

We can no longer pass unknown flags straight through to clang. Instead, the
user appends -- after the bindgen flags and input header, after which point
any more flags and arguments get collected and passed to clang. This required
changes to the test runner and the // bindgen-flags comments.

r? @emilio

Wohoo, thanks for doing this! :-)
I'll merge it, though now I'll have to rebase stuff and change scripts :P
@bors-servo: r+

:pushpin: Commit 1826827 has been approved by emilio

:zap: Test exempted - status

This needs a corresponding PR to the servo side (regen.py), btw

Was the first thing I did a few hours ago: https://github.com/servo/servo/pull/13079
Feel free to review it :P

Thanks for merging :)
Will need to update servo/rust-mozjs/etc/bindings.sh too -- I can do that after the smup lands.


(MAINT) Remove one httparty gem reference from the Gemfile

The prior commit added an extra reference to the httparty gem in the
puppet-server Gemfile which is unnecessary.  This commit removes that
reference.

:gem:


Cancel payment - "Auth CancelReimbursement" error shows up when cancelling payment with Expensify

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Found when executing PR: https://github.com/Expensify/App/pull/32674
Version Number: v1.4.16-4
Reproducible in staging?: Y
Reproducible in production?: N
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause-Internal Team
Slack conversation: @
Action Performed:
Precondition:

User A (Admin)- have Fidelity verified business bank account set up on Old Dot.
User B (Workspace member) - have Chase deposit bank account set up on Old Dot.

[Member] Request money from the workspace chat.
[Admin] Open the expense report and click Pay with Expensify.
[Admin] Click 3-dot menu > Cancel payment > Cancel payment.

Expected Result:
Payment cancellation is successful.
Actual Result:
Cancellation for payment with Expensify fails as "Auth CancelReimbursement returned an error" shows up.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/115492554/8e943858-de55-4eb9-99ed-4d07d432a60c

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01d0f4ed22cb60b3ca
Upwork Job ID:<PHONE_NUMBER>348860416
Last Price Increase: 2023-12-23

Reverting the original PR

PR reverted, Assigning @Gonals so you can track this issue and fix it in next iteration

Setting it as monthly, as this is no longer happening (but I want it to keep track)

@izarutskaya, were these real accounts? The test bank accounts don't work for reimbursement cancellation

@izarutskaya bump on https://github.com/Expensify/App/issues/33540#issuecomment-1873902501.
Switching this back to monthly based on @Gonals comment here

Closing this, as it got reverted

@Gonals Our QA team is still able to reproduce when validating this PR #34719. So reopening the issue.
https://github.com/Expensify/App/assets/43996225/7f0cbb8a-7e4d-465d-9b65-6fb04648d568

@Gonals Our QA team is still able to reproduce when validating this PR #34719. So reopening the issue.
Record_2024-02-16-15-47-53.mp4

Can you confirm you are using real bank accounts and NOT the test ones? This fails with test bank accounts 😅

@Gonals Our QA team is still able to reproduce when validating this PR #34719. So reopening the issue.
Record_2024-02-16-15-47-53.mp4

Can you confirm you are using real bank accounts and NOT the test ones? This fails with test bank accounts 😅

Logs:
855e27c8bc8471e5-JHB | db1.sjc | 2024-02-15 14:14:07 640 |<EMAIL_ADDRESS>| Cannot reimburse from Plaid test bank account
-- | -- | -- | -- | --

Basically, we don't really pay anyone with a test bank account, so we can't really cancel the payment
855e27c8bc8471e5-JHB	db1.sjc	2024-02-15 14:14:07 640<EMAIL_ADDRESS>Cannot reimburse from Plaid test bank account

Sounds like we're good to re-close this @Gonals?

I think so


bug and no solution to fix it

Describe the bug
Unhandled Runtime Error
Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.
Check the render method of NavItem.
Call Stack
createFiberFromTypeAndProps
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (27799:16)
createFiberFromTypeAndProps
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (27828:14)
createFiberFromElement
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9106:27)
createChild
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9440:24)
reconcileChildrenArray
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9858:15)
reconcileChildFibersImpl
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9912:26)
mountChildFibers
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (15627:27)
reconcileChildren
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (16589:2)
updateHostComponent$1
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (18427:13)
beginWork$1
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (26791:13)
beginWork
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (25637:11)
performUnitOfWork
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (25353:4)
workLoopSync
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (25308:6)
renderRootSync
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (24525:19)
recoverFromConcurrentError
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (24470:25)
callback
node_modules/next/dist/compiled/scheduler/cjs/scheduler.development.js (256:33)
workLoop
node_modules/next/dist/compiled/scheduler/cjs/scheduler.development.js (225:13)
flushWork
node_modules/next/dist/compiled/scheduler/cjs/scheduler.development.js (534:20)
Link to the Bolt URL that caused the error
https://bolt.new/~/bolt-nextjs-shadcn-axupac
Steps to reproduce
i don't know
Expected behavior
Unhandled Runtime Error
Error: Element type is invalid: expected a string (for built-in components) or a class/function (for composite components) but got: undefined. You likely forgot to export your component from the file it's defined in, or you might have mixed up default and named imports.
Check the render method of NavItem.
Call Stack
createFiberFromTypeAndProps
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (27799:16)
createFiberFromTypeAndProps
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (27828:14)
createFiberFromElement
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9106:27)
createChild
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9440:24)
reconcileChildrenArray
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9858:15)
reconcileChildFibersImpl
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (9912:26)
mountChildFibers
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (15627:27)
reconcileChildren
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (16589:2)
updateHostComponent$1
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (18427:13)
beginWork$1
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (26791:13)
beginWork
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (25637:11)
performUnitOfWork
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (25353:4)
workLoopSync
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (25308:6)
renderRootSync
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (24525:19)
recoverFromConcurrentError
node_modules/next/dist/compiled/react-dom/cjs/react-dom.development.js (24470:25)
callback
node_modules/next/dist/compiled/scheduler/cjs/scheduler.development.js (256:33)
workLoop
node_modules/next/dist/compiled/scheduler/cjs/scheduler.development.js (225:13)
flushWork
node_modules/next/dist/compiled/scheduler/cjs/scheduler.development.js (534:20)
Screen Recording / Screenshot
No response
Platform
Browser name = Chrome
Full version = <IP_ADDRESS>
Major version = 128
navigator.appName = Netscape
navigator.userAgent = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 OPR/<IP_ADDRESS>
performance.memory = {
  "totalJSHeapSize": 185353439,
  "usedJSHeapSize": 147329607,
  "jsHeapSizeLimit":<PHONE_NUMBER>
}
Username = parisboy92
Chat ID = dba5dfd52e3f

Additional context
No response

I'd have to see the code itself, but this error can occur when the wrong case in a variable is used in reference to the one defined. They need to match exactly. Take a look at the code and see if you can spot the typo (likely your code is the issue and not the node_modules).


@TMcFarland-Heritage to store the files & associated spreadsheet for later processing

Images missing from the standard HRIA filesystem need to be added as a post-MVP task. The list of missing files sent to @TMcFarland-Heritage is to be stored in a safe place and then manually added after MVP.

@TMcFarland-Heritage confirmed in an email on 2023-04-14 that the files have been stored for processing post-MVP.

Z:\Heritage_Branch@ops\HPI\REGISTRAR\ARCHES\BC Register\Images info


Will IE, Firefox and safari support LCP in the future?

I also have this need, can you support it？

LCP is not a metric that can be easily polyfilled, so it is not something this library will support unless those browsers implement the native API.
If you would like to see support in those browsers, I'd recommend reaching out to them or filing bugs on their issue trackers expression your support for this feature.


preact.config.js snippets

As it is now possible to change the configuration via preact.config.js, some useful snippets can be found in several issues: https://github.com/developit/preact-cli/search?q=preact.config.js&type=Issues&utf8=✓
May it be worth to collect (some of) them here in the Github Wiki or in a README? It may be hard to maintain, what do you think about it?

Agreed. I think the quickest thing that can be done is to add a short paragraph in the README to tell the user that WebpackConfigHelpers is very useful for changing the webpack config. Here's a suggestion of what can be added to the README:

Use the helpers argument to easily modify the config object. For example, you can disable the sourceMaps option of UgflifyJsPlugin like this:
helpers.getPluginsByName(config, "UglifyJsPlugin")[0].plugin.options.sourceMap = false;

Github already has a Wiki section, what do you think @developit ?

That would be great !! That makes the README clean too !!

Just dropping a note here - I think plugins would make this event better.  I'm fine with the wiki, since these are likely too specific and lengthy for the readme itself.

Wiki space for the above:
https://github.com/developit/preact-cli/wiki/Config-Recipes

Filled with recipes


Enhancing the capability for reporting node resource allocation information

What would you like to be added?
Periodically acquire the current resource allocation status on the nodes and report it to the KCNR CRD. Additionally, demonstrate this process by incorporating a practical use case, such as the spark operator.
Why is this needed?
Katalyst divides different CPU pools for containers of different QoS classes on a node, thus achieving resource isolation. However, presently, this information regarding CPU resource pools has not been exposed through CRD, causing certain inconveniences for observation.

/assign


Can you share the BID database?

The BID dataset is difficult to find online,Can you share the BID database?

You can find the link to BID on the readme of our previous work https://github.com/zwx8981/UNIQUE

You can find the link to BID on the readme of our previous work https://github.com/zwx8981/UNIQUE

thank you


Added some useful features

In use, I found some useful features that were not included, and this PR added some of them.

The features are not enabled in the class that is derived from MonoBehaviour. So check the base class name that current line belongs to, instead of checking if current line is in a MonoBehaviour. I've added settings to control these features will be enabled in which classes.
This is still not the perfect solution, but it's better than before.

CodeLens was enabled for the local functions. This will be also fixed when use the base class check.

Some people prefer to add the access modifier, and there should also be a setting to configure it.

Sorry about my poor English. The "derived from MonoBehaviour" in the first point actually refers to the "derived from other class that is derived from MonoBehaviour".

This pull request is too big to be merged at once.

The features are not enabled in the class that is derived from MonoBehaviour. So check the base class name that current line belongs to, instead of checking if current line is in a MonoBehaviour. I've added settings to control these features will be enabled in which classes.
This is still not the perfect solution, but it's better than before.

I've seen this issue too. It seems there's no simple solution. I'm thinking about removing the MonoBehaviour check in a future release.

CodeLens was enabled for the local functions. This will be also fixed when use the base class check.

I'll look into that later. You could make an another, smaller pr, if you wish.

Some people prefer to add the access modifier, and there should also be a setting to configure it.

I've added this feature on the V3 branch, thanks for the suggestion.
Thanks again!!


Fixed Importing campaign error handling

This pull request fixes the error handling for importing a campaign. It will  now display error message dialogs instead of raising an error for invalid xml files, incorrect version, or no campaign data to import.
To test attempt to import a file by going to file import > Campaign XML

[x] Error messages make sense and are accurate
[x] Error message box displays in attempting to import a non xml file
[x] Error message box displays when xml root is not king_phisher
[x] Error message box displays when xml version is less then 1.3
[x] Error message box displays when there is no campaign data to import

Looks good, made some minor changes but I'll land this now. Thanks @wolfthefallen.


Feature/token transfer

Description:
Added a new API "TokenTransfer" to "Token Service"

@fakepaulbugeja you can preview the build here https://614a5b228321fb000792f3c1--hedera-fee-tool-master.netlify.app/


Update Manta/Strelka + Add Manta Single mode

Use new containers for Strelka and for Manta cf CAW-container/#28

Manta evolves from 1.0.3 to 1.1.1
Strelka evolves from 1.0.15 to 2.8.2

Manta now analyses not only pairs but single samples too:

closes #455
add new RunSingleManta process

get diploidSV for Normal only sample
get tumorSV for Tumor only sample

add new normal single sample manta tsv file
update vcf selection for reports/annotation
update tests

@marcelm We will now be able to run Manta for Normal only samples

I am merging this, as have to try out Strelka for real data.


Django 1.8, key error for 'use_ipv6' when running ./manage.py runserver

The fix is to change commands/runserver.py:206 from
                 run(self.addr, int(self.port), app, mixin, ipv6=options['use_ipv6'])

to either
                run(self.addr, int(self.port), app, mixin, ipv6=options.get('use_ipv6', False))

or
                run(self.addr, int(self.port), app, mixin, ipv6=self.use_ipv6)

Not at all familiar with the code from devserver, so I didn't want to make a pull request.
  File "./manage.py", line 18, in <module>
    execute_from_command_line()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/devserver/management/commands/runserver.py", line 91, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 49, in execute
    super(Command, self).execute(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 441, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/devserver/management/commands/runserver.py", line 102, in handle
    return super(Command, self).handle(addrport=addrport, *args, **options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 88, in handle
    self.run(**options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 99, in run
    self.inner_run(None, **options)
  File "/usr/local/lib/python2.7/dist-packages/devserver/management/commands/runserver.py", line 206, in inner_run
    run(self.addr, int(self.port), app, mixin, ipv6=options['use_ipv6'])
    KeyError: 'use_ipv6'

I'm getting the same error on Django 1.8 after setup.

Fixed via #114


SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '0-<IP_ADDRESS>' for key 'forum_online_user_id_ident_idx'

A very strange bug, as it slips in the request:
INSERT INTO ::online (user_id, ident, logged, o_position, o_name)
SELECT 0, ?s:ip, ?i:logged, ?s:pos, ?s:name
FROM ::groups
WHERE NOT EXISTS (
    SELECT 1
    FROM ::online
    WHERE user_id=0 AND ident=?s:ip
 )
 LIMIT 1

Again there was a single error for the modified query:
INSERT INTO ::online (user_id, ident, logged, o_position, o_name)
SELECT tmp.*
FROM (SELECT 0 AS f1, ?s:ip AS f2, ?i:logged AS f3, ?s:pos AS f4, ?s:name AS f5) AS tmp
WHERE NOT EXISTS (
    SELECT 1
    FROM ::online
    WHERE user_id=0 AND ident=?s:ip
)

Using IGNORY solved this problem.


Fix behavior of filter-abund and filter-abund-single with respect to Ns.

filter-abund.py and filter-abund-single.py no longer discard sequences with Ns in them, and instead treat Ns as As.  Fixes #985.
Also unified the rest of the codebase to treat Ns as As.

[ ] Is it mergeable?
[ ] Did it pass the tests?
[ ] If it introduces new functionality in scripts/ is it tested?
Check for code coverage with make clean diff-cover
[ ] Is it well formatted? Look at make pep8, make diff_pylint_report,
make cppcheck, and make doc output. Use make format and manual
fixing as needed.
[ ] Did it change the command-line interface? Only additions are allowed
without a major version increment. Changing file formats also requires a
major version number increment.
[ ] Is it documented in the ChangeLog?
http://en.wikipedia.org/wiki/Changelog#Format
[ ] Was a spellchecker run on the source code and documentation after
changes were made?
[ ] Is the Copyright year up to date?

Note to @mr-c: this changes the output behavior, and may require a version increase to 2.0 upon merge.

@mr-c @luizirber @camillescott ready for review.
@drtamermansour your comments on the approach solicited!

Two comments:

You keep the N's in the output only if the reads are less than the
"normalize_to" cutoff. I think you mean it and I can see why.
Should we change the behavior of normalize-by-median to keep the 'N's in
the output?

On Fri, May 8, 2015 at 10:35 AM, C. Titus Brown<EMAIL_ADDRESS>wrote:

@mr-c https://github.com/mr-c @luizirber https://github.com/luizirber
@camillescott https://github.com/camillescott ready for review.
@drtamermansour https://github.com/drtamermansour your comments on the
approach solicited!
—
Reply to this email directly or view it on GitHub
https://github.com/ged-lab/khmer/pull/987#issuecomment-100274552.

I think we always keep the Ns, as long as they're not trimmed off.
I believe normalize-by-median already keeps Ns in the output.

You assume that trimming will always remove the newly changed 'A's which
should not be always true, right?
You are right. I am sorry. You use the transformed read for counting kmers
but you output the original record
On Fri, May 8, 2015 at 11:30 AM, C. Titus Brown<EMAIL_ADDRESS>wrote:

I think we always keep the Ns, as long as they're not trimmed off.
I believe normalize-by-median already keeps Ns in the output.
—
Reply to this email directly or view it on GitHub
https://github.com/ged-lab/khmer/pull/987#issuecomment-100287735.

On Fri, May 08, 2015 at 09:57:41AM -0700, Tamer Mansour wrote:

You assume that trimming will always remove the newly changed 'A's which
should not be always true, right?

I didn't intend to assume that... if the trimming doesn't remove the
new "A" base, an N is correctly output for the filtered sequence.

Oooh ya, I see it now. This is very clever
On Fri, May 8, 2015 at 12:04 PM, C. Titus Brown<EMAIL_ADDRESS>wrote:

On Fri, May 08, 2015 at 09:57:41AM -0700, Tamer Mansour wrote:

You assume that trimming will always remove the newly changed 'A's which
should not be always true, right?

I didn't intend to assume that... if the trimming doesn't remove the
new "A" base, an N is correctly output for the filtered sequence.
—
Reply to this email directly or view it on GitHub
https://github.com/ged-lab/khmer/pull/987#issuecomment-100299743.

On Fri, May 08, 2015 at 10:09:47AM -0700, Tamer Mansour wrote:

Oooh ya, I see it now. This is very clever

:)

@luizirber @camillescott ready for review.  Note that this should not be merged yet, as it will bump the version number significantly @mr-c.

Can not we have a beta version for Khmer 2 were we merge such updates?
Tamer
On Tue, May 12, 2015 at 6:31 AM, C. Titus Brown<EMAIL_ADDRESS>wrote:

@luizirber https://github.com/luizirber @camillescott
https://github.com/camillescott ready for review. Note that this should
not be merged yet, as it will bump the version number significantly @mr-c
https://github.com/mr-c.
—
Reply to this email directly or view it on GitHub
https://github.com/ged-lab/khmer/pull/987#issuecomment-101245949.

We'll figure something out; as @mr-c and I discussed yesterday, the plan is to
release more quickly over the upcoming months.

@bocajnotnef @luizirber @camillescott CR please? Now that it's after 1.4 release, we could merge.

Only one nitpick and it isn't mergable. otherwise LGTM.

Ready for review and merge @luizirber @camillescott

LGTM


Gwen doesnt support for sql statements using DSL (apart from select)

 <is|will be> defined in the  database by sql ""
this dsl supports select, have tested it  but we dont have any dsl for update or create (any other ddl or dml)

We haven't provided this capability because it is a data management concern. But you can call out to external scripts (like sqlplus scripts for example) using the system process DSL as follows to achieve what you need:
I execute system process "insert call to script here"

@anshu781126, did you give the above a try?

Yes this works but for this I need to maintain a piece of code which invokes some sql client. It would be more convenient to directly use dsl; like other automation tool supports. Data Management can be part of testcases so in that case user will be at ease if he gets to do it by DSL rather than maintaining additional piece of code.

My concern is that this could open up a whole world of data management problems and issues that could result in high maintenance and instability for Gwen. Also, since you are using a native SQL client, it makes sense call out to it through the system process DSL. That's the precise reason we provide that DSL; so native capabilities can be utilised in a way that they are not coupled to Gwen.

@anshu781126, we've given it some thought and propose to provide some basic support for executing SQL insert, update, or delete statements with the following DSL:

I update the <dbName> database by sql "<updateSmt>"

<dbName> = the name of database to execute the query on. This name is used to configure the database as follows:

The GWEN_CLASSPATH environment variable must be set to include a path to the JDBC driver JAR
A gwen.db.<dbName>.driver setting must be set to the name of the JDBC driver implementation class
A gwen.db.<dbName>.url setting must be set to the JDBC URL of the database

<updateStmt> = the SQL insert, update, or delete statement (can be a DocString)

Example using mysql:

Set GWEN_CLASSPATH variable in the environment to include the JDBC driver JAR

Unix: export GWEN_CLASSPATH=/path-to-driver/mysql-connector-java-5.1.40-bin.jar
Windows: set GWEN_CLASSPATH c:/path-to-driver/mysql-connector-java-5.1.40-bin.jar

Configure the database in your Gwen settings. In this example, <dbName>=feedback

gwen.db.feedback.driver = com.mysql.jdbc.Driver
gwen.db.feedback.url = jdbc:mysql://localhost/feedback?user=myUser&password=myPassword&serverTimezone=UTC&useSSL=false

Execute an update statement

Example using DocString syntax: When I update the feedback database by sql
     """
     update comments set<EMAIL_ADDRESS>where<EMAIL_ADDRESS>     """

Or standard one line syntaxWhen I update the feedback database by sql "update comments set<EMAIL_ADDRESS>where<EMAIL_ADDRESS>

Check the number of rows affected:

Then feedback rows affected should be "1"

We've implemented this in the snapshot build below. Please download it and try it out (see example usage above):

https://oss.sonatype.org/content/repositories/snapshots/org/gweninterpreter/gwen-web/2.11.1-8-g4343bf5-SNAPSHOT/gwen-web-2.11.1-8-g4343bf5-SNAPSHOT.zip

Let us know how it goes.

Working fine for update.

Released in gwen-web v2.12.0


black screen on root view controller and no sidemenu

#fixed, segue was to be set from SideMenuController to Navigation controller, not from nav controller to sidemenucontroller


Allow adding a default source with a base URI of None

Minimally fixes #75.  This can be used to match complete URIs, particularly URN-ish URIs using schemes that do not readily break down into a base and a relative reference.
The details of the change (using a string prefix after verifying the base URI when it is not None) is invisible to the public API, and the fundamental semantics are preserved.
This is sufficient for the use case that was blocking me:  how to load a URI like tag:modern-json-schema.com,2023-03:something, which cannot be split.  Its only components are the scheme tag and the path modern-json-schema.con,2023-03:something, and that path cannot be used as the start of a relative reference because it contains a :.
Potential workarounds
An alternative would be to register a source for each expected URI using the whole URI as a base URI, but that wouldn't work for URIs not known in advance.
Discarded approaches
I attempted the more flexible uri_prefix idea proposed in #75, but it became complicated and required awkward contortions to maintain backwards compatibility in the public API.  I decided that the extra flexibility was not worthwhile as a source registered in this way can do whatever re-mapping it wants with the relative_path, such as converting : characters to something else so that the URI can be used on Mac OS X filesystems.

I'm happy with this approach. It makes sense for base_uri=None to imply a default source; this just needs to be clarified in the docstring. Comments coming up...


Platantly stealing someone's work and changing the names and then having the audacity to create a pull request on the original repo


Which data file correspond to table 4 fasttext?

Hi, this is really a great work! May I know which data file in the repo corresponds to table 4 fasttext method as there are many files under exp folder. And the positive/negative samples to train the fasttext. Thank you so much for any help.

Same comment here. Thanks for the great work. Are you going to release the fastText OH-2.5 +ELI5 text classifier?

the classifier is available here: https://huggingface.co/mlfoundations/fasttext-oh-eli5

great, thanks a lot!

Thanks a lot! Btw, would we be able to access those files under s3 in your exp_data (e.g. filtered data), it seems currently the access is denied. Thank you.

If you can tell me which files you want, I can move them to huggingface, we unfortunately cannot make the bucket public.

@Vaishaal Is it possible to share the script for training this classifier?

If you can tell me which files you want, I can move them to huggingface, we unfortunately cannot make the bucket public.

Could you please release all classifiers in table 5 for further study?


Add libpango1.0-dev for trusty 14.04

I tried using the script on my my clean 14.04 trusty server, and I had to install libpango1.0-dev again, otherwise the Watermark tests etc would fail. See https://github.com/h2non/bimg/issues/63

Hi Julius, the preinstall.sh script is almost-deprecated for users of sharp as only 32-bit x86 Linux users still require it (and Centos 6 users, but only for a few more months).
I do however realise it's being promoted and used by other libraries that rely on libvips such as bimg and friends.
If pango is required for bimg, we should be consistent and install it on all flavours of Linux, not just for the use case of a specific Ubuntu version when also installing openslide.
Are you also able to add the correct pango-dev, pango-devel etc. packages for the other Linux OSs?

@lovell So what is the 'new' way of setting up libvps for use with libraries like sharp, bimg etc? I am sorry, I didn't notice, that I made the pull request to sharp and not bimg.

sharp ships with pre-compiled binaries (Linux and Windows, x64 and ARM architectures) to allow non-root installation and keep up with the latest libvips improvements. These won't work out-of-the-box for the golang-based bimg.
You should be able to get away with using sudo apt install libvips-dev on Ubuntu 14.04 if the claimed libvips v7.40.0+ requirement of bimg is correct.

Closing as I believe this is no longer required - feel free to re-open if I've got this wrong.


Update node cause current version set seems to be incompatible

Describe the bug
Running ./gradlew -PnodeInstall=true -Pprod -Pwar clean bootWar outputs:
error<EMAIL_ADDRESS>The engine "node" is incompatible with this module. Expected version "^10 || ^12 || >=14". Got "13.12.0"

Where 13.12.0 is installed because it is set so in gradle.properties. So it seems either should autoprefixer we downgraded or node upgraded. The latter seems to make more sense.
To Reproduce
Running ./gradlew -PnodeInstall=true -Pprod -Pwar clean bootWar
Expected behavior

Environment

 - OS: Ubuntu
 - Branch: develop

Updating it fixes it

I can make a PR but since changing the node version might have a lot impact I would suggest some maintainer does so

please provide a PR, thanks

I guess we can go to node 14. It's out there since quite some time and the support in the build infrastructure should be available so that we do not get any error messages.
We just have to be careful, that there are not issues in Github actions or in Bamboo, so please do not choose the latest node version

I updated it to 14.0.0 on develop


Error 20 and then bot stops running

Hi
I run two bots, one with email authentication and one with mobile authentication.
Over the last 3 days, around 5 times now my bots have gone offline with error 20 which I assume is steam updates. But when I look at bots other people have made, they don't stop and I think it's because they aren't using this API but a newer one.
Does this API not support reconnecting after steam updates? If so, how should I implement this for both of my bots? Thanks

20 is ServiceUnavailable, which is normal. Just keep reconnecting.

Thanks for the fast reply. When you say, just keep reconnecting, I have been doing that but it's really frustrating when I wake up and they're offline and I have to get my auth code out again and restart it. Is there no way to keep it running automatically after error 20?

If you're using Steam Guard, you can specify sha_sentryfile when logging in to avoid having to specify auth_code, see https://github.com/seishun/node-steam/wiki/Protobufs#cmsgclientlogon. No clue about two-factor auth, I don't use it in my bots.

Yes that is what I have been doing with my email bot, it's the mobile bot that is the problem because it loses its login session and dies on error 20 and then has no way to reconnect automatically because it requires the mobile two-factor auth code to re-log in. Other bots aren't affected by this though - they don't even go offline so they must use a different API that handles the event differently(?) Either that or I need to do something different which is where I wasn't sure

If the Steam server goes offline, the bot will go offline and will need to reconnect (manually or automatically), there's no way around it.
There's https://github.com/DoctorMcKay/node-steam-totp but I haven't used it. You might want to ask in https://github.com/steam-forward/node-steam-forum


ROX-23570: Factor out mailing lists from vulnerability reporting

Description
Prerequisite to reuse form and view for NotifierConfigurations (also known as delivery destinations) from Vulnerability Reports in Compliance Reports.
export type NotifierConfiguration = {
    emailConfig: {
        notifierId: string;
        mailingLists: string[];
        customSubject: string;
        customBody: string;
    };
    notifierName: string;
};

Add src/Components/NotifierConfiguration folder following previous refactoring:

src/Components/EmailNotifier
src/Components/EmailTemplate

Changes in EmailTemplate folder

Edit EmailTemplateForm.tsx file.

Add isReadOnly prop to reuse form for view.
Add readOnlyVariant prop to TextInput and TextArea elements.
Add conditional rendering for Reset links.

Edit EmailTemplateModal.tsx file.

Use onChange={null} to reuse modal for view.

Changes in NotifierConfiguration folder

Create NotifierConfigurationForm.tsx file.

Caller provides name of property in formik values via fieldIdPrefix prop.
Caller updates other form state if needed via optional onDeleteLastNotifierConfiguration prop.
Vulnerability Reports resets the schedule.
Component updates formik state via setFieldValue prop.

Move and rename NotifierConfigurationView.tsx file.

Replace formValues: ReportFormValues with notifierConfiguraton: NotifierConfiguration prop.
Replace references to reportParameters with optional renderTemplatePreview prop for presentation that is specific to vulnerability reports.
Replace description list with table to solve alignment problems.
Replace generic Modal with specific EmailTemplateModal element.

Move and rename NotifierMailingLists.tsx file.

Caller updates form state via setMailingLists and setNotifier props.
It was a judgement call for NotifierMailingLists component to delegate its updates to NotifierConfigurationForm componen
Separate updateNotifierList from useEffect hook to solve timing problem (see testing step 1).

Changes in VulnerabilityReporting folder
Use optional props to separate vulnerability-specific dependencies on form values other than notifier configurations.

Edit DeliveryDestinationsForm.tsx file.

Factor out reusable code into NotifierConfigurationForm component.

Edit ReportReviewForm.tsx file.
Delete useEmailTemplateModal.ts file and use conditional rendering instead.
Edit ReportJobs.tsx file.
Edit ViewVulnReportPage.tsx file.

Checklist

[x] Investigated and inspected CI test results
[x] ~Unit test and regression tests added~

Testing Performed

yarn lint in ui
yarn build in ui and then compute branch - master for the following

wc build/static/js/*.js
main.js 0 = 4633606 - 4633606
total -3731 = 11574067 - 11577798
ls -al build/static/js/*.js | wc
files 0 = 172 - 172

yarn start in ui

Manual testing
Prerequisite: create at least one collection.
Create e-mail notifiers on the fly.

Create a report and create email notifier

Verify payload and response of POST /v1/notifiers request
See that notifier is selected automatically. Baseline code intended but did not succeed because of timing problem filter method call before response to GET /v1/notifiers request.
Verify review details.
Verify payload and response of POST /v2/reports/configurations request

Edit the first report and ditto items 2 and 3 above.
Clone the first report and ditto items 2 and 3 above.

ReportReviewForm
Misaligned in description lists before changes

Aligned in table after changes

ViewVulnReportPage

Integration testing
Uh, oh! No tests. Yet.

I noticed Configure report scope and Email notifier both show id's briefly when switching between steps in the wizard. I understand this has to do with a selection being made but the component not yet having the name since it needs to be re-fetched each time the step is visited. There are at least a couple solutions worth thinking about:

Storing name in formik – I know we prefer keeping formik state as close to the api as possible but I still see value in separating the two in some cases. There might be a clean way to keep the actual form input values in formik state while also maintaining what's important for the api.

Lifting notifiers and state up – This might add complexity but it would give us preexisting notifier data as we switch back and forth between steps. We could still update notifier data when adding a new notifier, switching steps, or adding a new delivery destination. React context could be really useful in this case.


UI Issues on Course Page and Missing "Load Less" Button

UI Issue on the Course Page:
The UI of the course page is not well-designed and has several responsiveness issues. The layout breaks on smaller screens, with elements misaligned and text overflowing.
Specific areas affected:
Course titles and descriptions overlap or break layout on mobile devices.
Images are distorted and not resizing correctly.
Buttons are inconsistently placed, leading to a poor user experience on different screen sizes.
Missing "Load Less" Button:
There is no "Load Less" button to collapse the course list after expanding it. This creates an issue where users cannot reduce the number of displayed courses once they have expanded the list, leading to a cluttered interface.
Expected behavior: After expanding the list of courses, there should be a "Load Less" button to allow users to collapse the course view and show fewer items. However, this functionality is missing.
@vivekvardhan2810  Assign this issue to me under gsscoc and hactober fest.

I would like to work on this issue pls assign me this @vivekvardhan2810

As i have a strong background in UI/UX i request u to assign it to me , i would be really glad to solve this issue

for now we don't required load less button, So i am closing this issue.
Please Understand.


Contributing! Remember.

Come on. License it!

Please

I'll shut up and make a real pull request


Compatibility with input element?

Hi !
The doc says this lib is compatible with textarea and contenteditable, but I need it on a regular input (type=text).
At first everything seems to run perfectly fine (fiddle: http://jsfiddle.net/r59oty6c/ ) but I am wondering if I'm going to run into some trouble in the long run ?
I found it weird that the doc doesn't state support for inputs.
Thanks !

The problem is if the input's content is too long, and you have scroll, the view position will be wired.
And I think autocomplete right under the input will be batter interaction for short input, so that I didn't state that.


Update examples for the 4.0 release

@jasongrout @SylvainCorlay
closes #114

This mostly reflects the new API to interact and link right?

Yes, and a couple bigsplit import changes.  I also uploaded some more images, removed some broken image links, and uploaded the low level widget talk from scipy.

This looks generally okay to me (I didn't look at everything in detail).

Jason and I are running all the notebooks.

The date picker' s up and down arrows in the examples are not quite aligned with the textbox.

The "factoring" polynomial does not use MathJax

WRT the date time picker, that's a browser control IIRC.  It could be that some of the Bootstrap CSS is causing it to render strangely.

ok

def f(x:True): in Using Interact

def f(x:True): in Using Interact

@jasongrout , what about it?  It works for me- are you using Python 3?

oh, right.  It does say it's for python 3.  I'll add a comment too.

@jdfreder, I'm correcting a bunch of grammar in at least the Using Interacts notebook.

@jasongrout open a different PR?

sure, I can, or put in a PR to your PR

Yeah @jasongrout if you could open a PR against this one, or open one after this one, whichever you prefer, that would be excellent !

Updates here: https://github.com/jasongrout/ipywidgets/tree/updateexamples (I'll open a PR when I'm done)