Gas meter not discovered/no sensors generated for gas.

Describe the bug
Missing gas meter sensors
To Reproduce
Added integration, current and previous electricity meter added, no gas meter
A gas meter may be seen to be present when running an API curl command https://api.octopus.energy/v1/accounts/A-xxxxxxx/ (the same key and acount added to the integrations. The meter is SMETS1 but is adopted )I tried ticked and unticked).
Expected behavior
Gas meter sensors to be added.
Home Assistant Logs
I can't find anything useful.

Sorry to hear you're having issues. I've just released a new version with some additional debugging information around adding sensors. To get this information, the first thing to do is increase the log levels for the component. This can be done by setting the following values in your configuration.yaml file.
logger:
  logs:
    custom_components.octopus_energy: info

If you don't have access to this file, then you should be able to set the log levels using the available services.
Once done, you'll need to reload the integration and then check the "full home assistant log". You should then see entries associated with this component stating either sensors were added, skipped or no sensors were available at all.
The identifiers of the sensors should then be checked against your Octopus Energy dashboard to verify the correct sensors are being picked up.

Thanks for the prompt response.
I deleted the integration, downloaded the latest version, installed and there are now no sensors.
The major error in the log is now;
Logger: homeassistant.components.sensor
Source: custom_components/octopus_energy/sensor.py:132
Integration: Sensor (documentation, issues)
First occurred: 20:15:42 (3 occurrences)
Last logged: 20:16:17
Error while setting up octopus_energy platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
await asyncio.shield(task)
File "/config/custom_components/octopus_energy/sensor.py", line 97, in async_setup_entry
await async_setup_default_sensors(hass, entry, async_add_entities)
File "/config/custom_components/octopus_energy/sensor.py", line 132, in async_setup_default_sensors
_LOGGER.info(f'Skipping electricity meter due to no active agreement; mpan: {point["mpan"]}; serial number: {meter["serial_number"]}')
UnboundLocalError: local variable 'meter' referenced before assignment

I'll try again using branch main (it defaulted to develop. Oops.

Nope, that looks like my dodgy logic. There is a new version available at https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/v4.1.1.
Based on where it's erroring, it looks like you have some meters without active agreements. Have you had sensors replaced? Do you know what kind of tariff you're on?

On 8.1.0 - I can see 8 sensors, all related to the electricity meters, I am unable to find the word "gas" in the log.
I am on the TEP, that might entertain you for a bit, they just added a second tariff to the one meter.
From Octopus;
Gas: Octopus 12M Fixed
Electricity: Sorry, we weren't able to get the details for your tariff. (I'm betting it will be "Tesla Energy Plan Export" tomorrow)
Electricity: Tesla Energy Plan Import.

The newly added information doesn't seem to appear in the HA log panel. You'll need to click "Load Full Home Assistant Log" to get the debug logs. As mentioned, 4.1.1 should fix your previous error.

On 4.1.1 -
Logger: homeassistant.components.sensor
Source: custom_components/octopus_energy/sensor.py:151
Integration: Sensor (documentation, issues)
First occurred: 21:15:43 (1 occurrences)
Last logged: 21:15:43
Error while setting up octopus_energy platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
await asyncio.shield(task)
File "/config/custom_components/octopus_energy/sensor.py", line 97, in async_setup_entry
await async_setup_default_sensors(hass, entry, async_add_entities)
File "/config/custom_components/octopus_energy/sensor.py", line 151, in async_setup_default_sensors
_LOGGER.info(f'Skipping gas meter due to no active agreement; mpan: {point["mpan"]}; serial number: {meter["serial_number"]}')
KeyError: 'mpan'

From the full log;
2022-03-22 21:15:43 INFO (MainThread) [custom_components.octopus_energy.sensor] Skipping electricity meter due to no active agreement; mpan and serial number:
2022-03-22 21:15:43 INFO (MainThread) [custom_components.octopus_energy.sensor] agreements: []
2022-03-22 21:15:43 INFO (MainThread) [custom_components.octopus_energy.sensor] Adding electricity meter; mpan: and; serial number:
2022-03-22 21:15:43 INFO (MainThread) [custom_components.octopus_energy.sensor] Adding electricity meter; mpan: and; serial number:
Still no Gas. Much like half the word at the moment.

🙄 Another fix inbound. At some point I should deploy some code that works in this area.  Once deployed, it should indicate the gas sensor that is being skipped and the agreements associated with the sensor which should indicate why it's being skipped.

Fast coder!

It looks like it's not picking up the final agreement for your gas despite it still being active, so that's where I'll take a look.
In regards to logs, what you have provided is sufficient. However, I think I'm going to expose an overarching device so that the relevant information can be extracted more easily using https://www.home-assistant.io/integrations/diagnostics/

I've just released https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/v4.1.3, which should fix your issue.

4.1.3
Unable to get past Setup your basic account information screen.
"Account information was not found"

4.1.3
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Skipping electricity meter due to no active agreement; mpan: 002; serial number: 207
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] agreements: []
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Adding electricity meter; mpan:<PHONE_NUMBER>768; serial number: 639
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Adding electricity meter; mpan: 768; serial number: 207
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Adding gas meter; mprn: 101; serial number: 171
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Adding gas meter; mprn: 101; serial number: 800
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyGasCurrentRate
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyGasCurrentRate
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityPreviousRate for '768/639'
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityCurrentRate for '768/207'
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityPreviousRate for '768/207'
2022-03-23 19:28:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityCurrentRate for '768/639'
2022-03-23 19:28:58 INFO (MainThread) [custom_components.octopus_energy.sensor] Calculated previous gas consumption for '101/800'...
2022-03-23 19:28:58 INFO (MainThread) [custom_components.octopus_energy.sensor] Calculated previous electricity consumption for '768/207'...
2022-03-23 19:29:59 INFO (MainThread) [custom_components.octopus_energy.sensor] Calculated previous gas consumption cost for '101/800'...
2022-03-23 19:30:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityPreviousRate for '768/639'
2022-03-23 19:30:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityCurrentRate for '768/207'
2022-03-23 19:30:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityPreviousRate for '768/207'
2022-03-23 19:30:57 INFO (MainThread) [custom_components.octopus_energy.sensor] Updating OctopusEnergyElectricityCurrentRate for '768/639'
So ... Lots of meters to look at, the gas is present. Some ghosts (I'll just hide them. Sorted, Thanks for a very prompt fix.

Glad it worked :)


Concurrency debug exceptions thrown inside RestKit

I'm debugging an app using the -com.apple.CoreData.ConcurrencyDebug 1 option with Xcode 6 beta 2 and after fixing all the concurrent exceptions I can find in our code I now have some being thrown in RestKit.
The first one is thrown in RKManagedObjectRequestOperation.m at -saveContext: (line 832).
    if ([self.privateContext hasChanges]) {
I fixed that by wrapping the hasChanges call in a performBlockAndWait. I can submit a pull request for this if it would be helpful.
__block BOOL hasChanges = NO;
[self.privateContext performBlockAndWait:^{
    hasChanges = self.privateContext hasChanges];
}];
if (hasChanges) {

Now the next exception is thrown in RKManagedObjectRequestOperation.m at -saveContextToPersistentStore:error: (line 774).
                success = [contextToSave save:&localError];
I checked and it fails at the first call (this may be important since the code is in a loop going back up its parent context chain, saving each one).
This brings me to the end of my Core Data debugging expertise. I don't understand how this would throw a concurrency exception here, since the code is operating inside a performBlockAndWait. Could it be something upstream in our app? I feel this is unlikely since the particular call chain is processing a JSON response from our server at this point (i.e. we got to this point in the code through NSURLConnection's callbacks).
However I'm interested to know
a) Should I be concerned about the fact that we get exceptions here? I'm testing with the Xcode 6 beta and I know it may raise false positives. Unfortunately this exception prevents me from checking the rest of the app.
b) If this is a genuine issue, how to fix it. Are there any tools that allow one to know the thread that a private context was originally created on? The puzzle here is that this code is already executing inside performBlockAndWait, which I thought was provided specifically to prevent the exception being thrown.
As an aside, I should say that I have also tried loading up the RestKit project into Xcode 6 B2 to see if the unit tests would pass and/or shed light on the problem but the test target won't compile (it claims OCMock is missing). I'll see if I can resolve that issue on my own but any hints would be appreciated.

Many of these errors have been cleaned up in #2022 and newer issues. Are you still seeing these problems?


Higher order gradients through DCN layer

When I try to get higher order derivatives on architectures using your implementation of DCNv2, I get the following error:
"RuntimeError: DCNv2FunctionLegacyBackward is not differentiable twice".
Is there a way around this?

What I did is comment out all #@once_differentiable on dcn_v2.py.


Cancelling automate-all doesn't exit

For a default Heroku deployment, cancelling the automate-all run gives a cancellation message but doesn't actually exit. It fails quickly because heroku create hasn't been called.
This is a one-line fix, adding sys.exit() in _prep_automate_all(). This will be fixed as soon as support_azure is merged.

[ ] Make release that includes this fix.


Dynatrace Agent Not Detecting Communication Endpoints and Tenant Information

Expected Behavior
The agent should be capable of correctly identifying communication endpoints, along with the tenant ID and tenant token.
Current Behavior
Currently, the Dynatrace agent is not utilizing (or not using at all) the data retrieved from the connectioninfo API to appropriately configure the token, tenant ID, and communication endpoints for establishing a connection with the Dynatrace backend.
Although the connectioninfo API provides all the required details, such as the token, tenant ID, and communication endpoint, the agent is not utilizing this data to set up the connection parameters. Instead, it defaults to attempting to connect to http://localhost:8020/communication for communication, which results in unsuccessful connections.
This behavior contradicts the intended functionality, where the agent should be able to autonomously configure itself using the information from the connectioninfo API.
Possible Solution
At this point, a potential solution has not been identified. It's essential to further investigate the agent's handling of the connectioninfo data and how it processes the token, tenant ID, and communication endpoint details.
Or does it make the call to connectioninfo API at all?
Steps to Reproduce

Set up a Spring Boot application.
Add the necessary bindings for Dynatrace configuration.
Build an image from the application created in step 1.
Run the resulting image.

Motivations
We have a strong motivation to adopt the Dynatrace buildpack to enhance our application's startup time. The current Dynatrace operator adds a significant delay due to its injection of an init container for agent download. By utilizing the buildpack, we aim to mitigate this added delay and streamline our startup process.

Currently, the Dynatrace agent is not utilizing (or not using at all) the data retrieved from the connectioninfo API to appropriately configure the token, tenant ID, and communication endpoints for establishing a connection with the Dynatrace backend.

Can you expand on this? The buildpacks team is not experts on all the software we integrate, it's just far too much stuff. If you can provide more details on how this works in Dynatrace, we can likely point you in the right direction or get some code changes to help.
A good way to do this is to explain how you'd normally do this on a VM or with Dockerfiles. Be as specific as possible. We can usually then translate to how it should work with buildpacks.
Thanks

Actually I just found this logic in this code repo here: https://github.com/paketo-buildpacks/dynatrace/blob/1f8c8cf74b1d7266b32330055201094f0881148c/helper/properties.go#L60-L92
Somehow it doesn’t work as expected, i.e. the env vars are not set to the values returned from /v1/deployment/installer/agent/connectioninfo.
It’s a bit hard to explain if you’re not familiar with Dynatrace. But I hope the above code snippet can give you some idea.

Gotcha, thanks. I'm on the same page now.
So this code runs in what we call a helper (the buildpack spec calls them exec.d binaries). The helpers/exec.d binaries run prior to your application starting in the launch or runtime container. The main purpose is to do things like this and configure the environment in which your application code will ultimately run.
A couple of things can commonly go wrong here:

Are you also setting your bindings at launch/runtime? Bindings for DT need to be set both when you build and when you run your application image. If the bindings are not set, then the helper won't run.

Which brings us to the next question. Can you confirm that the helper runs? Do you see the line Configuring Dynatrace properties when you start your app image? It should run if you've set bindings, but if it's not running that could signal something isn't right or expected with the bindings.

The info we get back from the connect info request get's set in env variables. It would be good to peek and see what those are being set to. It would be helpful to override the start process and execute env instead. That will dump the env variables as set by the buildpack and helper. You can then see the tenant, token and connection point being set. That'll help understand more about what is going on here.

Hope that helps.

Indeed, the helper wasn't executed due to the absence of bindings at runtime. I had initially assumed that the bindings were only necessary during the build time.
It might be beneficial for the instructions in the readme to provide more clarity on this aspect.
I appreciate your explanation. Thank you.

Another observation: When I omit runtime bindings for ca-certificates, the helper encounters difficulties using buildtime-provided certificates to connect to /v1/deployment/installer/agent/connectioninfo, due to our private CA.
Strangely, our Spring Boot app successfully utilizes these buildtime CA certs, which contrasts with the behavior of the dynatrace buildpack helper. Is this intended?

Glad that helped!

It might be beneficial for the instructions in the readme to provide more clarity on this aspect.

Good feedback. I'll add a story to improve our docs. Thanks

Strangely, our Spring Boot app successfully utilizes these buildtime CA certs, which contrasts with the behavior of the dynatrace buildpack helper. Is this intended?

Hmm, it could be a difference in the system CA certs vs the Java CA certs. The JVM keeps its own keystore with trusted certs, where the Helpers are written in Go and utilize the OpenSSL system CA certs. I'd have to see the output of build & run. The bindings can will add the certs at build time but that's only for build time (unless you use the option to embed the CA certs). If you omit them at runtime, I'd expect them to not be available. There's a lot of variables though.
Pro tip: for trusted certs, using the BP_EMBED_CERTS is really convenient. It'll take the ca-certificates bindings at build time and embed them into the image so you don't need the bindings at runtime. Since trusted certs are just public keys and not secret, it is safe to embed them. The only reason you may not want to do this would be if your trusted certs change often & you want to be able to update them without needing to rebuild the image.
Hope that helps!

Thank you for the suggestion regarding BP_EMBED_CERTS, it has resolved my issue.
Nonetheless, it appears that the behavior of importing the certificates to the JVM trust store and the ability of Spring to utilize build-time certificates might not have been intentionally designed. I've checked various documents from both buildpacks and Paketo, but I haven't come across any documentation covering this aspect.

@dmikusa shouldn't the bindings at build time be used again at run time by default, unless specifically overridden by runtime bindings?

No. That's the purpose of bindings. They contain secret information so it cannot be included into the container information, or you'll leak the secrets. That is why you need to specify them at both build and runtime in this case.
Having said that, there are some bindings that only apply at build time or only at runtime. If that's the case, then you only need them during the given phase of the lifecycle. In this case, with Dynatrace, they're needed at build time to trigger the agent to be installed and at runtime to actually configure the agent.


⚠ Samsung has degraded performance

In 4c2bc39, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2412 ms

Resolved: Samsung performance has improved in 5a22f76.


Chat consumes more VRAM than Generate

Bug description
Hi there 👋
While working on integration of Gemma 2 (9b variant) I noticed that while a regular generate script barely fits into L4 with 24 GB, a chat script throws an OOM error.
It looks like it's not a Gemma 2 9b specific error.
I tried a couple of other models, in a regular and a quantized form, with a single prompt:

What is the distance between Earth and the Moon?

Machine: Lightning Studio with 1xL4.

Model
Chat
Generate
$Δ$
Chat (bnb.nf4)
Generate (bnb.nf4)
$Δ$

Phi-3
9.29
7.78
1.51
4.33
2.83
1.5

TinyLlama (chat)
2.60
2.30
0.3
1.38
1.07
0.31

Gemma 1 7b-it
20.58
18.83
1.75
11.45
9.69
1.76

Gemma 2 9b-it
OOM
20.58
-
16.48
10.98
5.5

* memory is in GB
Chat is essentially a generate script running in a loop. It should not consume more memory, at least if a single prompt is provided.
Since the difference in memory consumption between a regular and a quantized model stays the same, I assume, without even looking at the code, there is something wrong with memory preallocation (kv cache?).
What operating system are you using?
Linux
LitGPT Version
Version: 0.4.3.dev0

Thanks for reporting. Yes, this is weird. My first thought would also be that it's something with the KV cache. I assume the maximum new tokens length is the same, right?

It looks like in generate script we provide how many tokens to generate, and the value is calculated as the length of the prompt + max_new_token:
https://github.com/Lightning-AI/litgpt/blob/3a4526ef9f1f107e78b8ab9dc537a144beb2d680/litgpt/generate/base.py#L261
While in chat script we don't do that and simply use the model's max sequence length:
https://github.com/Lightning-AI/litgpt/blob/3a4526ef9f1f107e78b8ab9dc537a144beb2d680/litgpt/chat/base.py#L126-L128
Still, it shouldn't affect VRAM consumption, since the output has the same length, meaning that the generation is stopped by eos token, rather than this limit.
I'll investigate further after I finish with the Gemma 2.

Good observation, this could be related. We can debug this by first making these two consistent.

could be that in generate/base.py, we set
# set the max_seq_length to limit the memory usage to what we need
model.max_seq_length = max_returned_tokens

and we don't do this in Chat!

Yes, @aniketmaurya, that is exactly the reason.
As I stated above, in the generate script the code overrides model.max_seq_length to be len(prompt) + max_new_tokens, and then this value is used when preallocating space for kv-cache.
In the chat script this is not done and kv-cache always has the size of the max_seq_length which is equal to block_size from the config.
I guess it was done in this way because in chat mode you don't know what is the length of all prompts beforehand.
But it's easily fixable. We can preallocate kv-cache for the first turn in the same fashion as in the generate script and then, if in the current turn the length of the prompt is greater than all the previous ones had, we can recreate kv-cache.
I'll create a PR on Monday with the fix.

Good point.
I guess an additional message saying that it can lead to a higher memory consumption due to a large kv-cache will be helpful.

This reminds me, at some point we discussed an option like optimize="compute" | "memory" for less advanced users, and this could be a good trade-off for such a setting


📝 update readme

Current dependencies on/for this PR:

main

PR #45   👈

This stack of pull requests is managed by Graphite.


[GEARPUMP-305][Doc] Use MkDocs plugin to validate html

Get rid of ruby tools
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the commit message is formatted like:
[GEARPUMP-<Jira issue #>] Meaningful description of pull request
[ ] Make sure tests pass via sbt clean test.
[ ] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality.

Codecov Report

Merging #252 into master will decrease coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #252      +/-   ##
==========================================
- Coverage   68.54%   68.49%   -0.05%
==========================================
  Files         195      195
  Lines        6234     6234
  Branches      541      541
==========================================
- Hits         4273     4270       -3
- Misses       1961     1964       +3


Move grpc/test/cpp/qps/qpstest.proto to a shared location

qpstest.proto will be used for other languages than just cpp, it should move to a common location (grpc/test/proto) before people start depending on it.

Friendly ping. I think it's time for wrapped languages to start implementation work on qps_client and qps_server (if that's what they they are called) and this is a prerequisite.


Exception during iterator yield caused by multiple issues in LookupTask

Running version 1.9.3, observed an exception in the tserver logs from Tablet (now in TabletBase) --> "Underlying iterator yielded to a position that does not follow the last key returned: "  The exception didn't make sense because the last key returned (results.get(results.size() - 1)) was not in the seek range.  While writing a unit test to reproduce the issue, three separate issues in LookupTask were discovered.

In LookupTask.run(), session.queries is a HashMap (therefore not sorted), so when List<KVEntry> results = new ArrayList<>(); which contains the accumulated results for all subsequent KeyExtents gets passed to successive tablets, a yield could easily happen at a key that is before the last entry in results. ("Underlying iterator yielded to a position that does not follow the last key returned: ")  LookupTask.run() should pass an empty List of results and add the returned value to the accumulated results.

LookupTask.run() uses an iterator (Iterator<Entry<KeyExtent,List<Range>>> iter = session.queries.entrySet().iterator()). If there are unfinished ranges on a tablet that is not closed, it calls session.queries.put(entry.getKey(), lookupResult.unfinishedRanges) which is modifying the container that is being iterated over inside iteration loop.  This results in a ConcurrentModificationException the next time iter.next() is called.

After 1 and 2 are corrected and the unfinished ranges are re-added to session.queries for a subsequent call, there are issues with partScan/partNextKey/partNextKeyInclusive
a. These variables are not used until after the loop exits, so they will be overwritten any time there is a lookupResult with unfinishedRanges on a tablet that is not closed.
b. Once we complete scanning the extent/ranges that were previously unfinished, the last one encoutered (is this desired) will then be in partScan/partNextKey/partNextKeyInclusive and fullscans.  Both are returned in the thrift response..... and this causes a NullPointerException in TabletServerBatchReaderIterator at line 554 in trackScanning()
ListIterator<Range> iterator = unscanned.get(ke).listIterator()

sequence of events:

Map<KeyExtent,List<Range>> unscanned starts with all requested values
remove all failures from unscanned
remove all fullScans from unscanned
then

  if (scanResult.partScan != null) {
      KeyExtent ke = new KeyExtent(scanResult.partScan);
      Key nextKey = new Key(scanResult.partNextKey);

      ListIterator<Range> iterator = unscanned.get(ke).listIterator();

NullPointerException because the scan on ke was completed and it is in fullScans, thus removed from the unscanned container
Encountered in version 1.9.3, but these issues exist in much older versions up to the current.
Reproduced by adding another test to YieldScannersIT that added splits to the table prior to adding data, ensuring that LookupTask received a request that included more than one KeyExtent -> List<Range> entries.

Updated the original description to use markdown a bit better. In particular, there were issues displaying the generic parameters for the collections classes, because < and > are reserved characters in markdown, as in HTML, and need to be escaped using < and > or quoted using backticks to create a literal code block, as in `List<String>`


Finding secrets in a commit history is flawed

When you search through a commit you are not actually finding all the secrets.

If a file is moved to a different path within the repo in a later commit it will not be searched in the earlier commits. Wraith will throw a file not found error because it is not fetching the commits, it is simply scanning them based on the file list from the top level commit.

There are duplicate findings being displayed. In part this is due to how the commit history is being scanned. Beyond that I am not sure the root cause yet.

Findings with a line number of 0 are being displayed. This is related to 2 but I am not sure again what the root cause is, though the zero is being set via hard coding for some reason.

The secret ID's are not unique. Due to how the ID's are being calculated, even across commits the same secret in the same file should have a unique secret ID. I am suspecting this is also related to how we are searching through the commit history and the duplicated findings.

@codeEmitter

Closed as these are no individual issues for easier tracking


Swift 3 updates for AnyObject -> Any

This just updated the AnyObject identifiers in the documentation that should have been changed to Any for the new Swift 3 usage  patterns.

🙇

Happy to help!


AniJS work in Angular? (Question)

Hello,
I used the following label to encourage an item and works:

But when my templates to integrate " Angular " is not working.
I have been researching and found the following:

Unfortunately it does not work either, I could help
Source: http://regularjs.github.io/blog/2014/07/26/2014-7-26-animation-rethink/#Angular

Hi Juan,
The AniJS behavior is the below:
1 - The library script is loaded.
2 - By default AniJS travel through the current DOM nodes.
If you add another DOM elements dinamically(using angularJS) with new data-anijs sentences after this moment , you need to call the AniJS run function using Javascript.
//Traveled the DOM scope parsing all nodes with data-anijs tag.
AniJS.run();
Maybe the next link can help.
Best Regards, Dariel

Thank you! Dariel

HI, I'm five years late for this, but can you give me the solution or example or any code that I can clone. I really want to use the aniJS on my Angular but I am too noob to do this and the solution that you give, I did not manage to do it. Hope you can help. Thank You


Debug logging in HttpPostEmitter and Batch

This addition should help to debug #5338, but it could also remain in the source code, because normally debug logging is not enabled.
@pjain1 @niketh @himanshug

Setup logging so that debug logging is enabled only in io.druid.java.util.emitter.core package, or just in HttpPostEmitter and Batch classes.

Setup logging so that the thread name is printed.

👍

@pjain1 could you please review and merge this if it is ok?


Issue/44/clean compile

The code now compiles cleanly under OSX and Linux with Clang. There was a lot of long vs. size_t warnings that I solved with static_cast however it might be a better idea to use the correct type. I would also suggest at one point or another that member variables used by subclasses or aggregates are exposed through member functions instead.

hiding the problem is not the best solution; if the code is going to be patched then can we improve it
instead?

I agree. However such a fix will be more intrusive then the fix I did. Its not my codebase so I do not want to changes that affects it too much. My development (emacs) is built around google code style which makes it a pain to use with a non, google-code-style, conformant code base.

Thank you, but try avoid replacing whitespaces. They make the commit somewhat unreadable. I try not being strict with use of whitespace (as long as the current ones stay as is) to avoid making commits without real purpose. If everyone cleaned up their whitespaces it would not be a problem, but not all do. Sorry, but would you mind making a new PR?
Also, no need to change the current CMakeLists.txt, at least in my opinion, as it became somewhat bloated with the warning flags? That is, the goal is to get rid of warnings when using the headers in your and others projects. Maybe just replace -Wall with -Weverything for this repository? Then it's the same with gcc and clang as well (I would recommend using gcc for release build by the way; more optimised binaries).
I agree with using the same type.
When it comes to setters and getters, I'm not a huge fan of such functions unless it is reasonable to assume that these are needed now or in the future. In larger projects, unneeded setters and getters are making the code much less readable in my opinion.

I have a macro in my editor that removes all trailing white-spaces when I save and this is what shows up in the diff where white-space has been removed. The only way for my do redo the pull-request is do get a fresh copy and start-over again turning off the macro in my editor.

I'll accept it this time then:) I'm a bit busy Tonight, but hopefully Tomorrow I'll be able to go through the changes. Thank you again.

Thanks. I will ensure that I have the macro turned off the next time. Sorry for the inconvenience.

I decided that -Wshadow restricts the programming style too much (I'm not a huge fan of the google style guide actually, I think I'm more partial to: https://github.com/isocpp/CppCoreGuidelines), but I made the following changes instead:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 836aa7c..41db0f6 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,6 +1,6 @@
 cmake_minimum_required (VERSION 2.8)
 project (Simple-Web-Server)
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -O3 -Wall")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -O3 -Weverything -Wno-shadow -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-weak-vtables -Wno-padded -Wno-unused-macros")

 if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
     if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4.8)
diff --git a/client_http.hpp b/client_http.hpp
index 65eb63c..6b24a72 100644
--- a/client_http.hpp
+++ b/client_http.hpp
@@ -14,6 +14,8 @@ namespace SimpleWeb {
     template <class socket_type>
     class ClientBase {
     public:
+        virtual ~ClientBase() {}
+
         class Response {
             friend class ClientBase<socket_type>;

@@ -43,7 +45,7 @@ namespace SimpleWeb {
         private:
             boost::asio::streambuf content_buffer;

-            Response(): content(&content_buffer) {};
+            Response(): content(&content_buffer) {}
         };

         std::shared_ptr<Response> request(const std::string& request_type, const std::string& path="/", boost::string_ref content="",
@@ -86,7 +88,7 @@ namespace SimpleWeb {
                 corrected_path="/";

             content.seekp(0, std::ios::end);
-            size_t content_length=content.tellp();
+            auto content_length=content.tellp();
             content.seekp(0, std::ios::beg);

             boost::asio::streambuf write_buffer;
@@ -135,7 +137,7 @@ namespace SimpleWeb {
             }
             else {
                 host=host_port.substr(0, host_end);
-                port=(unsigned short)stoul(host_port.substr(host_end+1));
+                port=static_cast<unsigned short>(stoul(host_port.substr(host_end+1)));
             }

             asio_endpoint=boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(), port);
@@ -191,7 +193,7 @@ namespace SimpleWeb {
                     boost::asio::streambuf streambuf;
                     std::ostream content(&streambuf);

-                    size_t length;
+                    std::streamsize length;
                     std::string buffer;
                     do {
                         size_t bytes_transferred = boost::asio::read_until(*socket, response->content_buffer, "\r\n");
@@ -199,7 +201,7 @@ namespace SimpleWeb {
                         getline(response->content, line);
                         bytes_transferred-=line.size()+1;
                         line.pop_back();
-                        length=stoull(line, 0, 16);
+                        length=stol(line, 0, 16);

                         size_t num_additional_bytes=response->content_buffer.size()-bytes_transferred;

@@ -208,7 +210,7 @@ namespace SimpleWeb {
                                 boost::asio::transfer_exactly(2+length-num_additional_bytes));
                         }

-                        buffer.resize(length);
+                        buffer.resize(static_cast<size_t>(length));
                         response->content.read(&buffer[0], length);
                         content.write(&buffer[0], length);

@@ -257,4 +259,4 @@ namespace SimpleWeb {
     };
 }

-#endif	/* CLIENT_HTTP_HPP */
\ No newline at end of file
+#endif	/* CLIENT_HTTP_HPP */
diff --git a/client_https.hpp b/client_https.hpp
index f6fb865..1a15747 100644
--- a/client_https.hpp
+++ b/client_https.hpp
@@ -28,7 +28,7 @@ namespace SimpleWeb {
                 asio_context.load_verify_file(verify_file);

             socket=std::make_shared<HTTPS>(asio_io_service, asio_context);
-        };
+        }

     private:
         boost::asio::ssl::context asio_context;
@@ -49,4 +49,4 @@ namespace SimpleWeb {
     };
 }

-#endif	/* CLIENT_HTTPS_HPP */
\ No newline at end of file
+#endif	/* CLIENT_HTTPS_HPP */
diff --git a/http_examples.cpp b/http_examples.cpp
index 664763e..d1119ca 100644
--- a/http_examples.cpp
+++ b/http_examples.cpp
@@ -102,7 +102,7 @@ int main() {

                     if(ifs) {
                         ifs.seekg(0, ios::end);
-                        size_t length=ifs.tellg();
+                        auto length=ifs.tellg();

                         ifs.seekg(0, ios::beg);

@@ -111,14 +111,14 @@ int main() {
                         //read and send 128 KB at a time
                         const size_t buffer_size=131072;
                         vector<char> buffer(buffer_size);
-                        size_t read_length;
+                        streamsize read_length;
                         try {
                             while((read_length=ifs.read(&buffer[0], buffer_size).gcount())>0) {
                                 response.write(&buffer[0], read_length);
                                 response.flush();
                             }
                         }
-                        catch(const exception &e) {
+                        catch(const exception &) {
                             cerr << "Connection interrupted, closing file" << endl;
                         }

diff --git a/https_examples.cpp b/https_examples.cpp
index 2d05146..2e9d8b5 100644
--- a/https_examples.cpp
+++ b/https_examples.cpp
@@ -102,7 +102,7 @@ int main() {

                     if(ifs) {
                         ifs.seekg(0, ios::end);
-                        size_t length=ifs.tellg();
+                        auto length=ifs.tellg();

                         ifs.seekg(0, ios::beg);

@@ -111,14 +111,14 @@ int main() {
                         //read and send 128 KB at a time
                         const size_t buffer_size=131072;
                         vector<char> buffer(buffer_size);
-                        size_t read_length;
+                        streamsize read_length;
                         try {
                             while((read_length=ifs.read(&buffer[0], buffer_size).gcount())>0) {
                                 response.write(&buffer[0], read_length);
                                 response.flush();
                             }
                         }
-                        catch(const exception &e) {
+                        catch(const exception &) {
                             cerr << "Connection interrupted, closing file" << endl;
                         }

diff --git a/server_http.hpp b/server_http.hpp
index 51f4a4f..557c219 100644
--- a/server_http.hpp
+++ b/server_http.hpp
@@ -17,6 +17,8 @@ namespace SimpleWeb {
     template <class socket_type>
     class ServerBase {
     public:
+        virtual ~ServerBase() {}
+
         class Response : public std::ostream {
             friend class ServerBase<socket_type>;
         private:
@@ -101,7 +103,7 @@ namespace SimpleWeb {
                     remote_endpoint_address=socket.lowest_layer().remote_endpoint().address().to_string();
                     remote_endpoint_port=socket.lowest_layer().remote_endpoint().port();
                 }
-                catch(const std::exception& e) {}
+                catch(const std::exception&) {}
             }
         };

@@ -195,16 +197,16 @@ namespace SimpleWeb {
         boost::asio::ip::tcp::acceptor acceptor;
         std::vector<std::thread> threads;

-        size_t timeout_request;
-        size_t timeout_content;
+        long timeout_request;
+        long timeout_content;

-        ServerBase(unsigned short port, size_t num_threads, size_t timeout_request, size_t timeout_send_or_receive) :
+        ServerBase(unsigned short port, size_t num_threads, long timeout_request, long timeout_send_or_receive) :
                 config(port, num_threads), acceptor(io_service),
                 timeout_request(timeout_request), timeout_content(timeout_send_or_receive) {}

         virtual void accept()=0;

-        std::shared_ptr<boost::asio::deadline_timer> set_timeout_on_socket(std::shared_ptr<socket_type> socket, size_t seconds) {
+        std::shared_ptr<boost::asio::deadline_timer> set_timeout_on_socket(std::shared_ptr<socket_type> socket, long seconds) {
             std::shared_ptr<boost::asio::deadline_timer> timer(new boost::asio::deadline_timer(io_service));
             timer->expires_from_now(boost::posix_time::seconds(seconds));
             timer->async_wait([socket](const boost::system::error_code& ec){
@@ -217,7 +219,7 @@ namespace SimpleWeb {
             return timer;
         }

-        std::shared_ptr<boost::asio::deadline_timer> set_timeout_on_socket(std::shared_ptr<socket_type> socket, std::shared_ptr<Request> request, size_t seconds) {
+        std::shared_ptr<boost::asio::deadline_timer> set_timeout_on_socket(std::shared_ptr<socket_type> socket, std::shared_ptr<Request> request, long seconds) {
             std::shared_ptr<boost::asio::deadline_timer> timer(new boost::asio::deadline_timer(io_service));
             timer->expires_from_now(boost::posix_time::seconds(seconds));
             timer->async_wait(request->strand.wrap([socket](const boost::system::error_code& ec){
@@ -266,7 +268,7 @@ namespace SimpleWeb {
                         try {
                             content_length=stoull(it->second);
                         }
-                        catch(const std::exception &e) {
+                        catch(const std::exception &) {
                             return;
                         }
                         if(content_length>num_additional_bytes) {
@@ -367,7 +369,7 @@ namespace SimpleWeb {
                 try {
                     resource_function(response, request);
                 }
-                catch(const std::exception& e) {
+                catch(const std::exception&) {
                     return;
                 }

@@ -375,7 +377,7 @@ namespace SimpleWeb {
                     try {
                         response.flush();
                     }
-                    catch(const std::exception &e) {
+                    catch(const std::exception &) {
                         return;
                     }
                 }
@@ -385,7 +387,7 @@ namespace SimpleWeb {
                 try {
                     http_version=stof(request->http_version);
                 }
-                catch(const std::exception &e) {
+                catch(const std::exception &) {
                     return;
                 }

@@ -408,7 +410,7 @@ namespace SimpleWeb {
     template<>
     class Server<HTTP> : public ServerBase<HTTP> {
     public:
-        Server(unsigned short port, size_t num_threads=1, size_t timeout_request=5, size_t timeout_content=300) :
+        Server(unsigned short port, size_t num_threads=1, long timeout_request=5, long timeout_content=300) :
                 ServerBase<HTTP>::ServerBase(port, num_threads, timeout_request, timeout_content) {}

     private:
diff --git a/server_https.hpp b/server_https.hpp
index 097ac25..3972475 100644
--- a/server_https.hpp
+++ b/server_https.hpp
@@ -11,7 +11,7 @@ namespace SimpleWeb {
     class Server<HTTPS> : public ServerBase<HTTPS> {
     public:
         Server(unsigned short port, size_t num_threads, const std::string& cert_file, const std::string& private_key_file,
-                size_t timeout_request=5, size_t timeout_content=300,
+                long timeout_request=5, long timeout_content=300,
                 const std::string& verify_file=std::string()) :
                 ServerBase<HTTPS>::ServerBase(port, num_threads, timeout_request, timeout_content),
                 context(boost::asio::ssl::context::sslv23) {

What are your opinions on this?

Very nice much, much better.
The shadowing issue has nothing to do with google C style and I believe that there is no standard or programming style that encourage shadowing even though CppCore is agnostic to the subject of naming. I respect if you do not want to use "_" or any other type of adornment of member variables but it would be really nice if we could find a solution. Where we do not have to allow shadowing.

I pushed the diff above, with some slight changes. For now its with still warning options -Wall, but that is to supress warning messages from boost on some platforms.
I'll have to think a bit more on the shadow issue, but in general I'm fond of as modern C++ code as possible. The prefixes/postfixes feels like something from the pre2000 era, and as far as I understand it, it is to avoid confusion. With modern IDE's (like juCi++) it should not be an issue in my opinion. Although, I'm open for further discussion!

I respect any decision you take. However shadowing can introduce bugs and it also quite nice from a readability perspective to know if a variable is declared in local scope or on a class level scope. We have about 80,000 lines of C++11/14 code developed during the last 3 years so I like to see our code base as pretty modern :-)

I also would prefer no warnings.
Regarding 3rd party libs and their warnings, I suggest if possible to:

use "-isystem " inplace of "-I " for the third party libs such as boost
in your source use "#include " for 3rd party and system headers (using < and >)
in your source use "#include "dir" for your own project headers (using ")

@jeffreyscottgraham thank you, that helped. Fixed in the commit above.

I think I'm pretty stuck on the issue on shadowing. There really is no good solution to this as far as I see it, unless of course you want to add pre/postfixes to either parameters or member variables, and thus enforcing a specific style. If https://github.com/isocpp/CppCoreGuidelines adds a section on this in the future, which I doubt they will, I'll of course follow them.
I instead propose you use include_directories(SYSTEM ...) when including this library in your projects.

Your proposed solution will work for us.

Closed maybe, but the shadow warnings, and most all compiler warnings in general, have absolutely nothing to do with STYLE. All warnings should be addressed and avoided. IMO shadowing leads
to increased maintenance and congnitive overhead because eventually the wrong-scoped version will be misused/mis-assumed.


How to set environment variable in container?

Hi I found this tool and wanted to test it, but can't unserstand how to set environment variables used by the container.
Any help would be much appreciated

this is now possible with the merge of #58


Imports cf-notifications from CF

Additions

Adds import for cf-notification from Capital Framework.

Removals

Removes notification.less

Changes

Adds role=alert when notification is warning or error in jinja template.
Converts notification JS to ES6 syntax.
Clean up unneeded/stale documentation.
Collapse CF Less paths in Gulp script to a glob call instead of calls to the individual modules.

Testing

./frontend.sh should pass.
Places where notification is used, such as on /blog/ should look like live.

Screenshots

Hold for the release of the renamed component.

@Scotchester ready!

I think something is still missing @anselmbradford


Regression: Clock example

Hi,
I have tried building the clock example on master and on version 0.1.1.
The clock in master has a regression. It is mirrored in the x axis and the hands of the clock run anticlockwise.
I did a git bisect and this appears to be the first commit that causes the issue:
6f52eb4d3a99565a6261eca3263fa8a5e2c3f63f is the first bad commit
Hope that helps and thanks for your time.

Seems like it is the dependency lyon v0.16 that is causing the regression. (graphics/Cargo.toml)
I tested reverting to 0.15 on master removes the issue,
Thanks

Thanks for catching this! #627 should fix the issue.

No sweat mate! What open source is for :) Sorry I couldn't be of more help. My first experience with the Iced code base  :/


Add new settings to captureCompleteSettings

Added engine and gamme to setting list

Perhaps more files need changing. See https://github.com/zivid/zivid-python-samples/pull/98/files


[CARBONDATA-2897][DataMap] Optimize datamap chooser

In this PR,

Remove code for merging into one datamap when some datamap hits both child nodes of And/Or expression in DataMapChooser. This aims to make datamap focus on pruning single index column without any logic process. Leave logic stuff to be done by AndDataMapExprWrapper and OrDataMapExprWrapper
Only extract ColumnExpression of Expression which our datamap can handle in DataMapChooser.
Add short circuit to return pruned result when result of left node is empty in AndDataMapExprWrapper.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

SDV Build Fail , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/6441/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/51/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/8131/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/60/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/15/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/222/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/390/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.3/8460/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8589/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/343/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8590/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/520/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/344/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8591/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/521/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/346/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8593/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/523/

retest this please

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8606/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/536/

since it changes the datamap chooser logic, I'd recommend you to test it with larger amount of data and observe the pruning

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/379/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8627/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/458/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/637/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8707/

I've verified this PR for another bug and it works fine.
see http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Issue-Bloomfilter-datamap-td63254.html

LGTM

@xuchuanyin In general if a datamap is created with a list of columns for suppose c1 and c2 then we create an index datamap on c1 and c2 and treat that as a composite index.  So if user queries with filters on both the columns then indexdatamap should combine both and make a single call of prune of that datamap because user has created composite datamap.

@ravipesala
It's a good idea to do multiple pruning at one time, but it also has shortcomings.

The datamap has to deal with the AND/OR logic of the composite expressions. If the expressions are AND, it needs to do intersection of the pruned result; If the expressions are OR, it needs to do union of the pruned result. It will makes the datamap complicated to handle.

Besides, you can see it  from the maillist that, current expressions forwarded to BloomDataMap contains too much unwanted expressions.

@xuchuanyin when user creates datamap on multiple columns then it should be composite index. But bloom creates index for individual columns even though user creates datamap on multiple columns. If user wants index for individual columns then he should create multiple datamaps not a single datamap.

@ravipesala What I want to say is that even if we support composite index, the current datamap chooser still failed to work, because it composites irrelevant expressions and forwards that to the datamap ignoring that the expression contains non-index columns. You can see it from the logs in the maillist that currently we have to deal 27 expressions while after applying this patch we only need to deal 4 expressions.

@xuchuanyin Datamap expression merging is the basic framework feature but not tested, so we might expect some issues from it as when it was implemented there were no index datamaps implemented. So we should fix those issues instead of removing it. It changes the very nature of index datamap itself if you remove it.
The expected feature should be as follows.
When the user creates index datamap on multiple columns then it should be treated as a composite index, not an individual index.
The user should create individual datamaps on the columns if he wants an individual index on the column.

@ravipesala  I do think assembling expressions and forward them to the corresponding datamap is a good idea. But current implementation needs more work to achieve this goal. We can not just pass the expressions to datamaps and let the datamap do the dirty work to do intersection or union of each expression.
As you mentioned we can create multiple individual datamaps, this will also be a problem now. For example bloomfilter datamap only support 'equalTo' and 'in', but current DataMapChooser will forward 'NotEqual','Greater','Less'... to the datamap also, which will cause performance waste.
Besides, for both BloomFilter datamap and Lucene datamap, we never treat specifying multiple columns in 'index_columns' as composite index, actually we just index them individually.

In addition, if the expression forwarded to bloomfilter datamap is indexCol1 = 1 AND indexCol2 = 2, bloomfilter datamap will prune the blocklets separately and giving result sets blockletSet1 and blockletSet2. The bloomfilter has to do intersection of the two result sets based on the 'AND' relation of the two expressions. ---- That's what I called 'dirty work'.

@ravipesala At last, I think current PR:

do fix the bug
has limited impact on other features (only impact bloomfilter datamap)
and it can be accepted.

Maybe in the next version, we can optimize the frame work by

assembling expression that exactly can be supported by the index datamap
supporting pipeline pruning (use a bitmap to indicate the hitted blocklet number in previous pruning procedure and use the bitmap in current pruning procedure)

@xuchuanyin Please check my fix in the PR https://github.com/apache/carbondata/pull/2767

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/714/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8781/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/536/

@kevinjmh I accepted another PR #2767 which also intends to fix this problem.
You can close this PR now.
Thanks for your working all the same!

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/624/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/816/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8885/


4: defer rendering menu buttons

This PR doesn't render the ... / reaction buttons until you hover over the box. This reduces a ton of rendering we normally never see
time to load storybook thread
base:
653.65
494.55 (with cache)
647.06
pr:
530.39
384.80 (with cache)
521.07

@keybase/react-hackers

@keybase/react-hackers bump

i think this is unrelated to this pr actually but i have a fix

@buoyad updated

talked IRL about trying to leave the menubuttons mounted after they've been mounted once + using css to control the hover color / button visibility

@buoyad updated. i pulled in the css structure from https://github.com/keybase/client/pull/14026


Azure DDoS L7 inline protection?

This article do not include clear message nor details about WAF integration, so I suggest the following:

clarify if there is any integration between Azure DDoS and Inline WAF (even via Sentinel analytics rules), or they are complementary (Azure DDoS for L3-L4 and WAF for L7),
explain how Azure WAF can protect against L7 DDoS attacks (slowloris attack, HTTP flooding...) and not just point to overview page

thank you
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 13e5c097-7992-261c-6961-3c5f2f34c264
Version Independent ID: 6aa116eb-54ab-f890-1dc3-6e89d3222aa1
Content: Inline L7 DDoS Protection with Gateway Load Balancer and partner NVAs
Content Source: articles/ddos-protection/inline-protection-glb.md
Service: ddos-protection
GitHub Login: @AbdullahBell
Microsoft Alias: abell

@Nabila-el
Thanks for your feedback! We will investigate and update as appropriate.

Hello @Nabila-el I relayed this to our internal team, and we are aware of this issue. An overhaul of this article is planned, but we do not yet have an ETA. Thank you for your input in the documentation!

#please-close


Added docs on how to hide a web part from the modern toolbox

Category

[x] New article

What's in this Pull Request?
Added documentation on how to hide a web part from the toolbox when editing a modern page.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/spfx/web-parts/basics/hide-web-part-from-toolbox.md
:white_check_mark:Succeeded
View

docs/toc.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

thx Mikael, appreciated.


add host domain parameter to google provider

Continuation of https://github.com/Vestorly/torii/pull/123; includes test.
Thanks!

Merged manually in 24a08c7ba1f820db1c728065243ba93aa496f56e


Support multi-arch pinot docker image

for both pinot base images and pinot docker image

Codecov Report

Merging #10429 (1ed5c34) into master (11b6bcd) will decrease coverage by 49.39%.
The diff coverage is n/a.

:exclamation: Current head 1ed5c34 differs from pull request most recent head 19fe1a9. Consider uploading reports for the commit 19fe1a9 to get more accurate results

@@              Coverage Diff              @@
##             master   #10429       +/-   ##
=============================================
- Coverage     63.23%   13.84%   -49.39%
+ Complexity     5068      259     -4809
=============================================
  Files          2028     1999       -29
  Lines        110632   108940     -1692
  Branches      16847    16647      -200
=============================================
- Hits          69955    15081    -54874
- Misses        35505    92637    +57132
+ Partials       5172     1222     -3950

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
?

unittests2
13.84% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1830 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Update .htconfig.php & additional preinstall option

Problem

.htconfig.php.sample needed an update
Disabling the directory before installing was requested by some users

Solution

htconfig.php.sample updated
An option now allows you to install a website with the directory disabled (actually not populated with your friend's friends)

PR Status

[x] Code finished and ready to be reviewed/tested
[x] The fix/enhancement were manually tested (if applicable)

Automatic tests
Automatic tests can be triggered on https://ci-apps-dev.yunohost.org/ after creating the PR, by commenting "!testme", "!gogogadgetoci" or "By the power of systemd, I invoke The Great App CI to test this Pull Request!". (N.B. : for this to work you need to be a member of the Yunohost-Apps organization)

!testme

We'll wait for the next major update of release branch in upstream before merging this in master.


Update RemoveFirstTwoChars.java

Using .delete(0, 2) method

Makes sense. However, I would not delete the original solution. It is better to ask learners to write both versions in two separate methods. Please update README.md and tests as well before merging.

Fixed according to the previous pull request

OK!


[BUG] OpenTelemetryExporter prevents setting the span name for Request telemetry

Library name and version
Azure.Monitor.OpenTelemetry.Exporter 1.2.0
Describe the bug
OpenTelemetryExporter overrides Activity.DisplayName with hard-coded logic for Request telemetry, which effectively makes it impossible to set the span name for requests. So, if a developer is able to figure out https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/1948 and determine that they have to set the Activity.DisplayName sometime after the request Activity has stopped, that value is ignored when the AppInsights telemetry is generated by the exporter - the exporter re-creates the default value that was replaced and ignores the overridden value.
Expected behavior
The value set in Activity.DisplayName should be used for request telemetry in the Request.Name and OperationName values. DisplayName should provide a good default (which it does), but the developer should be able to override it to make their telemetry more useful.
This is both the correct behavior, and I believe is consistent with App Insights classic.
Actual behavior
The value set in Activity.DisplayName is ignored; instead the logic used by the OpenTelemetry AspNetCore instrumentation is roughly repeated and the re-computed value is used for the Request.Name and OperationName.
Reproduction Steps
Here's a test (that runs in the Azure.Monitor.OpenTelemetry.Exporter.Integration.Tests project b/c there's no other reasonable way to validate the telemetry values):
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using Azure.Core.TestFramework;
using Azure.Monitor.OpenTelemetry.Exporter.Integration.Tests.TestFramework;
using Azure.Monitor.OpenTelemetry.Exporter.Models;
using Azure.Monitor.OpenTelemetry.Exporter.Tests.CommonTestFramework;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry;
using OpenTelemetry.Trace;
using Xunit;
using Xunit.Abstractions;

namespace Azure.Monitor.OpenTelemetry.Exporter.Integration.Tests.TelemetryCustomization;

public class WebServerTelemetryCustomizationTests : WebApplicationTestsBase
{
    private const string TestServerPort = "9996";
    private const string TestServerTarget = $"localhost:{TestServerPort}";
    private const string TestServerUrl = $"http://{TestServerTarget}/";

    public WebServerTelemetryCustomizationTests(ITestOutputHelper output)
    : base(output)
    {}

#if !NET462
    [Fact]
    public async Task ActivityDisplayName_CanBeSetForRequests()
    {
        List<TelemetryItem>? telemetryItems = null;

        var builder = WebApplication.CreateBuilder();
        builder.Services.AddOpenTelemetry()
            .WithTracing(builder =>
            {
                builder.AddAspNetCoreInstrumentation()
                    .AddProcessor<FixupDisplayNameProcessor>()
                    .AddAzureMonitorTraceExporterForTest(out telemetryItems);
            });
        builder.Services.AddRouting();

        await using var app = builder.Build();
        app.UseRouting();
        app.Map("{**path}",
                context =>
                {
                    // Here I'm setting the Activity.DisplayName to something more useful for this specific route
                    // However, I have to store a value for use by FixupDisplayNameProcessor due to https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/1948
                    var activity = context.Features.Get<IHttpActivityFeature>()?.Activity;
                    if (activity?.IsAllDataRequested == true)
                    {
                        var request = context.Request;
                        var spanName = $"{request.Method.ToUpperInvariant()} {request.Path}";
                        activity.AddTag(FixupDisplayNameProcessor.OverrideSpanNameTag, spanName);
                    }

                    context.Response.StatusCode = 200;
                    return Task.CompletedTask;
                });

        _ = app.RunAsync(TestServerUrl);

        // ACT
        using var httpClient = new HttpClient { BaseAddress = new Uri(TestServerUrl) };
        var res = await httpClient.GetAsync("/foo/bar");

        // SHUTDOWN
        var tracerProvider = app.Services.GetRequiredService<TracerProvider>();
        tracerProvider.ForceFlush();
        tracerProvider.Shutdown();

        // ASSERT
        Assert.NotNull(telemetryItems);
        WaitForActivityExport(telemetryItems);

        // Assert
        Assert.True(telemetryItems.Any(), "test project did not capture telemetry");
        var telemetryItem = telemetryItems.Last()!;
        telemetryOutput.Write(telemetryItem);

        AssertRequestTelemetry(
            telemetryItem: telemetryItem,
            expectedResponseCode: "200",
            expectedUrl: TestServerUrl + "foo/bar");

        // BUG: Both of these Asserts fail, b/c the value for both is "GET {**path}",
        // even though Activity.DisplayName was set to "GET /foo/bar" before the telemetryItem was created
        Assert.Equal("GET /foo/bar", (telemetryItem.Data.BaseData as RequestData)!.Name);
        Assert.Equal("GET /foo/bar", telemetryItem.Tags["ai.operation.name"]);
    }
#endif

    private class FixupDisplayNameProcessor : BaseProcessor<Activity>
    {
        public const string OverrideSpanNameTag = "override.span.name";

        public override void OnEnd(Activity activity)
        {
            // Have to set Activity.DisplayName here instead of where I want to due to https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/1948
            if (activity.IsAllDataRequested == true)
            {
                var overrideSpanName = activity.GetTagItem(OverrideSpanNameTag)?.ToString();
                if (overrideSpanName != null)
                {
                    activity.DisplayName = overrideSpanName;
                    activity.SetTag(OverrideSpanNameTag, null); // Delete the tag
                }
            }
        }
    }
}

Environment
PS > dotnet --info
.NET SDK:
 Version:           8.0.303
 Commit:            29ab8e3268
 Workload version:  8.0.300-manifests.34944930
 MSBuild version:   17.10.4+10fbfbf2e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.303\

Host:
  Version:      8.0.7
  Architecture: x64
  Commit:       2aade6beb0

.NET SDKs installed:
  6.0.424 [C:\Program Files\dotnet\sdk]
  8.0.303 [C:\Program Files\dotnet\sdk]

To provide a visual, here's what we get using Azure.Monitor.OpenTelemetry.Exporter :

and here's the equivalent view of requests in AppInsights classic:

and there's no reasonable way to fix it (there's at least 1 unreasonable way).

I'm consolidating all issues regarding overriding specific fields in the Application Insights schema into another issue to help with tracking. https://github.com/Azure/azure-sdk-for-net/issues/46021


Update cats-effect-laws, cats-effect-std to 3.3.13

Updates

org.typelevel:cats-effect-laws
org.typelevel:cats-effect-std

from 3.3.12 to 3.3.13.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.typelevel" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "org.typelevel" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #515.


✅ Extend Protractor End-to-End Tests for Quiz Exercises

Is your request related to a problem?
We want to prevent regression errors in ArTEMiS.
Describe the solution you'd like
We should add e2e tests for

[x] creation, and
[x] participation

of Quiz Exercises.
Additional context

We introduced Protractor End-to-End Tests in #140.
Right now, they cover Course creation and deletion, as well as login.

Closed in 3bf90611


Error while starting the container for initializing the database

description:
1、images from https://hub.docker.com/r/servicecatalog/oscm-deployer
2、run the   "   docker run --name deployer2 --rm -v /docker:/target -v /var/run/docker.sock:/var/run/docker.sock -e INITDB=true -e STARTUP=true servicecatalog/oscm-deployer  "
then I can't visit https://ip:8081/oscm-portal，but I can visit the tomcat console https://ip:8081/
screenshot:

Hi @wuweilue
Please check the value of BASE_URL_HTTPS you have given in the var.env file. Seems the it has an Invalid syntax.

Hi @GoebelL
I get the docker images again and
run the " docker run --name deployer2 --rm -v /docker:/target -v /var/run/docker.sock:/var/run/docker.sock -e INITDB=true -e STARTUP=true servicecatalog/oscm-deployer "
  then I Encountered the following problem

   Can you tell me, which version, I can run up?  Thank you.  or have  Complete deployment documentation, just follow the steps and Run smoothly。

    Think you!

Hi @wuweilue
This looks like some issue with openssl generating private keys on your host. In order help you we need more information. Please answer following questions.
a) Check var.env again. Is HOST_FQDN set correctly?
b) What is the name and version of your Linux distribution?
c) Which versions of docker do you use?
d) Which containers are running after running deployer2? Check with docker ps.
OSCM setup documentation can be found here.

Hi @GoebelL
Think you very much !
1、CentOS Linux release 7.4.1708 (Core)

2、Docker version 19.03.5, build 633a0ea

3、I check again the HOST_FQDN in var.evn，look like no problem. and  no any more BASE_URL_HTTPS errors

4、Run docker ps command, the results are as follows

Think you!

Hi @GoebelL
I want to ask a quesiton please.  which key in var.env？   Because run the last version I didn't see the problem。

Hi @wuweilue
Yes there was a small change on the deployer, that configures default certificates for new identity service.
Not totally sure, but I imagine that openssl fails to create the certificate CN name here, because the IP is beeing given instead of the full qualified domain name of the host. If not yet existing, I suggest to set a static full qualified hostname for your docker host. On CentOS 7 this should be possible from command-line - something like hostnamectl set-hostname ....
This could help to get OSCM starting up.

Hi @GoebelL
I  reinstalled the version for v18.02, the OSCM Started。

But it still doesn't work

Hi @GoebelL
I tried many times, but the problem still couldn't be solved.
1、hostnamectl

2、var.env

3、hosts

Hi @GoebelL
I found that errors is very likely to run the script of "openssl req -new -passin ".
So I look the script of oscm-deployer/start.sh,find the script "openssl req -new -passin file:/tmp/passphrase.txt -key /tmp/ssl.key -out /tmp/ssl.csr -subj "/CN=${HOST_FQDN}"" . Try to run alone it. But I cann't find erros.

Hi @wuweilue
Ok, I see there is a different issue now. I am looking to the first screenshot you posted yesterday. As shown there, the oscm containers up are up and running, tomee is there, but oscm-portal is not getting deployed. I also see that the oscm-deployer has pulled all containers from latest tag. Please note that latest OSCM also includes and requires the identity service and a few new settings. I fear you have a mixture of different versions in your environment now.
I suggest to install from scratch:

backup your var.env file
pull oscm-deployer from latest again
run deployer1 as described here
You can copy all env values from your backup file, except for may be three or four new settings. But the default values described in the comments inside the file should work well.

Hi @GoebelL
Thank you! I try again according to your method, The Result As Follow:
1、The application containers are all running  include oscm-identity.
2、But  run docker logs -f oscm-core，I find some errors，Var.env and docker-writer-initdb.yml key value have been modified。
The complete screenshot  is as follows，Could you tell me  How do I configure these incorrect keys in the tomcat log。
I've got to get the oscm running，thanks!

Hi @GoebelL
Continue to the previous question，Check the code to find the reason, and find that the key value is taken from the database, and the data source is configured in tomcat. var.env has been configured with key values, I don't know why the configuration setting table of the bss database has no data, and the writable in docker-compose-initdb.yml has been changed to true during initialization.

Hi @GoebelL
After trying many times, the reason why oscm-portal cannot be accessed is that the following key cannot be found in the tomcat log, but in fact var.env has been configured. Help. This needs to be resolved as soon as possible.
Thank you very much！
BASE_URL_HTTPS=https://<IP_ADDRESS>:8081/oscm-portal
LOG_FILE_PATH=/opt/logs
PSP_USAGE_ENABLED=false
TAGGING_MAX_TAGS=50
TAGGING_MIN_SCORE=10
WS_TIMEOUT=60000
AUDIT_LOG_ENABLED=false
AUDIT_LOG_MAX_ENTRIES_RETRIEVED=10
MAX_NUMBER_ALLOWED_USERS=100
TIMER_INTERVAL_USER_COUNT=100
AUTH_MODE=INTERNAL

Hi @GoebelL
thank you very much！
The problem has been solved, by looking at the code, insert the missing key into the database.


[FEATURE]

Description
How about adding some vector icons here and there on the site to make it look a little more appealing?
Screenshots
Something like this

Additional information
No response

Hi! I can work on this task. Can you assign me this.
Thanks.

Hey, can you elaborate a bit more on this issue? Which purpose are these icons serving? I'm not sure if we should add some random icons across the page 😅

hey @Cahllagerfeld  can elaborate that which type of icons u want to ,
in my opinion we can use FontAwesome library to use vector icons in the web-page t look more attractive


svdvals now returns a length-0 sequence of singular values given a size-0 2d array

asarray_validated?

Fixed.  Also modified svd to use this validation.

Why is it desirable to have svdvals and svd behave differently with zero-size arrays? I'm not a frequent user of either of these, so I might be missing something simple...
>>> from scipy.linalg import svd, svdvals
>>> import numpy as np
>>> a = np.zeros((0, 2))
>>> svdvals(a)
array([], dtype=float64)
>>> svd(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/br/virtualenvs/scipy-dev/local/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 97, in svd
    lwork, info = gesdd_lwork(a1.shape[0], a1.shape[1], compute_uv=compute_uv, full_matrices=full_matrices)
ValueError: On entry to DGESDD parameter number 5 had an illegal value

Why is it desirable to have svdvals and svd behave differently with zero-size arrays?

Thanks for looking at this!  I agree it's not so great that this PR would de-sync the behavior of svdvals and svd.
The svd implementation could probably be changed.  For example when compute_uv=0, svd should probably give the same thing as svdvals.  But then you will ask why svd with the compute_uv=0 option can deal with empty matrices whereas svd with the compute_uv=1 option cannot.  I haven't though as much about the U and V matrix outputs of svd for empty matrix inputs, but for svdvals this is a moot point.  Maybe the U and V outputs could be appropriately shaped, and maybe they can be identity matrices if orthogonal matrices are required.  There are also complications with left vs. right flags and with full vs. non-full matrix outputs.  All of these additional complications are moot when only svdvals is considered and not svd.

I'm not a frequent user of either of these, so I might be missing something simple...

I'll give some background.
There is pretty much a consensus agreement that the empty sum should be 0, the empty product should be 1, and that the concepts of empty lists and 'trivial' matrices with either zero rows or zero columns can be useful.
I would also suggest that empty vector norms and empty matrix norms should be zero, but I'm not 100% certain that this is a convention.  For example in numpy.linalg, the only tests for empty vector/matrix norms assert that these norms are zero, but not all norms are included in the tests, possibly for technical reasons (this is complicated by the fact that np.linalg.norm can be persuaded to compute functions that do not satisfy any of the inequalities that may characterize norms, for example the minimum absolute value of a vector).  Several matrix norms are defined in terms of singular values, and a slick way to get the 'empty matrix norm == 0' property could be to apply the appropriate function to the empty list of singular values.  For example if a particular matrix norm is defined as the of the sum of the singular values of a matrix, then if the singular values of an empty matrix were defined as an empty list, then this norm of an empty matrix would most naturally be the summation over an empty list, which is zero.
Therefore the natural implementation of empty matrix norms would be one use-case for defining the singular values of an empty matrix to be an empty list.  I don't know of other supporting or conflicting use-cases, and I'm also not sure if there is a mathematical convention.  Suggestions or comments are welcome!

Thanks for the backgound!
I can see why empty sum is zero and empty product is one. As for empty matrices I'm not so sure ---
I personally only met zero-sized arrays once when the cython code expecting a typed memoryview choked on it :-). But I can imagine it makes sense from some perspective, so take what I'm saying with a large grain of salt.
Basically, I'd think you'll need to special-case an empty matrix somewhere anyway. If the concept of an empty matrix is really useful, and empty matrices are allowed in the linalg routines, then I'd expect svd(..., compute_uv=True) with an empty input to return empty matrices for U and V. So that both the sum of the sigular values if zero, their product is one, and the norm of both U and V is zero. Oh, no, golly, then these matrices are not orthogonal. But if they are orthogonal, they cannot be empty, can they?
Basically, my gut feeling is that

svdvals being out of sync with svd is weird.
is there really a reason to keep svdvals as a separate function instead of recommending svd(.., compute_uv=False)?
Either linalg operations can be defined consistently for the empty matrices and then they should be allowed everywhere; otherwise they should not be allowed anywhere, and special-casing for the empty sum and empty product should be pushed up to the user level.

Ok, I'm hiding back behind a spoonful of salt to accompany whatever I said above.

That sounds pretty reasonable.  Maybe this PR should be replaced by a PR that gives more informative error messages for size-0 inputs for svdvals and svd, or even adds non-emptiness to the list of pre-conditions before calling lower level functions (this currently filters out sparse matrices, masked arrays, and object arrays)?

(Just noticed a possibly related numpy issue, https://github.com/numpy/numpy/issues/5419).
Don't let my gut feeling block progress :-). Maybe the right way is to proceed is to merge your changes for svdvals, and clearly document that svdvals only differs from svd(..., compoute_uv=True) in the handling of empty input --- how does this sound?
This only requires a one-line addition to the docstring of svdvals, it seems...

For example, here is a shot at phrasing it:
https://github.com/ev-br/scipy/commit/fc4a082ff91f9ca6807a2939aa3423dacf89869b
If you either take over this commit or come up with a better wording, I'm +1 for merging this PR.
Just to be clear: I'm changing my mind on the consistency of svdvals and svd (insert an obligatory Knuth quote here). Since svdvals exists, it makes sense to have it doing things which make sense for singular values but not for the orthogonal matrices from the full SVD decomposition.

Bump. @argriffing Maybe we should finish this for 0.16, what do you think?

@ev-br I've git cherry-pick evbr/pr/4365'd the commit

OK, it's been open for long enough and did not attract any further comments, so let's put it in. Thanks Alex and sorry for slowing this down.


[Feature Request] prepend tags with a specified string.

Hi and thank you for this plugin :).
I use nested tags to differentiate between tags from different integrations. I wanted to prepend every tag from pocket with 'pocket/'.
I did it for example in the obsidian-citations with {{#each entry.data.fields.keywords}}{{#if @last}}zotero/{{this}}{{else}}zotero/{{this}}, {{/if}}{{/each}} for zotero integration.
I wanted to use templater here but can't figure it out (probably can be done with executing JavaScript I think). It also would solve #82

For anybody else searching for this, I made it using templater like that:
---
Title: "{{title}}"
URL: {{url}}
Pocket URL: {{pocket-url}}
Pocket tags: {{tags-no-hash}}
---
tags: #pocket/<% await tp.frontmatter["Pocket tags"].replaceAll(', ', (', #pocket/'))%>


NgTableParams.$params gone?

Since 0.8.1, after instancing NgTableParams, there isn't $params anymore as part of it.
This was possible in 0.8.0:
var table = new NgTableParams(...);

table.$params.page = 1;

but not anymore. I didn't find this documented anywhere? What is the current approach to achieve above?

Here's the commit where $params was removed: https://github.com/esvit/ng-table/commit/e94ca5f7873673616e15a46ab8317595331ab6e1
The equivalent public api for readin/writing to $params$ is NgTableParams.parameters, or the more focused methods: filters, sorting, page, count
C


Updated ImageSource.FromStream() to explain lambda parameter.

A user on reddit was confused by this method, and the StackOverflow posts they referenced seemed to indicate the same confusion. It's not accurate that this method takes "a stream" as a parameter, it takes a factory method that returns a new, unique stream every time. I tried to add more information to make this clear.

Docs Build status updates of commit 12ad328:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

docs/Xamarin.Forms/ImageSource.xml
:x:Error

Details

:x:Error

Details

docs/Xamarin.Forms/ImageSource.xml

Line 0, Column 0: [Error-ECMA2Yaml_InternalError]

Intenal Several Error: System.Xml.XmlException: The 'see' start tag on line 233 position 63 does not match the end tag of 'param'. Line 233, position 208.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
   at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag)
   at System.Xml.XmlTextReaderImpl.ParseEndElement()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.Linq.XContainer.ReadContentFrom(XmlReader r)
   at System.Xml.Linq.XContainer.ReadContentFrom(XmlReader r, LoadOptions o)
   at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
   at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)
   at ECMA2Yaml.ECMALoader.LoadType(FileItem typeFile)
   at ECMA2Yaml.ECMALoader.LoadTypes(String basePath, Namespace ns)

Line 0, Column 0: [Error-ECMA2Yaml_File_LoadFailed] Failed to load 1 files, aborting...

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit f661ced:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs/Xamarin.Forms/ImageSource.xml
:warning:Warning

Details

docs/Xamarin.Forms/ImageSource.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'System.Threading.Tasks.Task<System.IO.Stream>'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Can somebody help me out with the current warning? It feels like that cross-reference should be valid. I copied it from a different part of the file. :(

Ahhhh I'm super excited this is my first PR to Xamarin stuff!


[Refactor] Optimize storage models

Regarding the long overdue refactor of the models and storage classes.
Thanks @saiy2k for putting this slideshow together!
https://docs.google.com/presentation/d/1VMN7MrgU9wXPgOSJjCKiFkKN7eMdfCe2PMhClawcJlI/edit#slide=id.p
This covers the core idea behind this refactor. It is absolutely necessary at this stage because we don't want to keep adding more technical debt by adapting all these new features into the old architecture.
This should be a comprehensive refactor and should not require such a major refactor again.

This issue is exclusively for model and pkg. Bloc updates can happen later as a separate issue.


clang-tidy-review cannot handle clang-diagnostics-* issues

In a build where e.g. some header files cannot be built, clang-tidy creates a clang-diagnostics error. This is quite helpful to see, however clang-tidy-review decides to crash when encountering those, as clang-tidy returns a non-zero errorcode in that situation:
https://github.com/continental/ecal/runs/6524850948?check_suite_focus=true#step:4:1955
Maybe it should be enough to simply remove the raise statement, when clang-tidy encounters a compile warning?
https://github.com/ZedThree/clang-tidy-review/blob/1cf54e5e9d293b1a5155a3ecda193519dcdfdd62/review.py#L546

I don't think removing the raise statement would be enough, it currently parses the output fixes YAML file, rather than the output of clang-tidy. So we'd need to parse the error in order to turn it into a comment or something.
I think also clang-tidy will stop as soon as it hits an error like this, so maybe it would also be helpful to do each file individually?

Hm, ok, i was afraid that it wouldn't be as simple as that 😅
Currently (due to the raise) a single compile error will also prevent clang-tidy-review to post information about other warnings, e.g. from clang-analyzer-*. I feel like those should be posted in any way. QtCreator does it the same way - the compile issues (clang-diagnostics-* you have to manually grab from the output, but everything else it will still display in the GUI.
Just not exiting but posting as much as it is able to would already be a proper solution in my mind.
Anyhow, I don't think clang-tidy exits directly in that case, I think it works just fine, but return 1 at the end as a hint that it wasn't able to build all files.

Fixed in #40 and released in v0.8.4
Thanks again @FlorianReimold !


Trying To Understand Why Certain Inputs Break Parser

I am using the T5 parser and noticed that the input
Making certain distinctions is imperative in looking back on the past
Causes the model to return the error
gid=x Unhandled token imperative
Failed to deserialize, snum=0, beam=0
gid=x Unhandled token imperative
Failed to deserialize, snum=0, beam=1
gid=x Unhandled token imperative
Failed to deserialize, snum=0, beam=2
gid=x Unhandled token imperative
Failed to deserialize, snum=0, beam=3

[None]

Two questions:

Why does this happen? In general, what kinds of inputs cause such an issue?
Suppose I need the sample parsed, is there anything I can do to get an output?

Thanks!

This is actually an implementation issue in the deserializer that I hadn't seen before.  In AMR 'interrogative', 'imperative', and 'expressive' are special reserve words used for attributes. Right now the deserializer is tagging them as attributes instead of nodes which they are in your sentence.  Since they're not in the right spot for an attrib it's skipping them.  The model is actually producing correct output going into the deserializer.
There should be a way to fix this but I'll have to spend a few minutes to figure out the right way to implement it.  I'll post a fix when I get it figured out, probably later today or tomorrow.
The model does occasionally produced improperly formatted output that doesn't deserialize but it's very rare, especially when you're using a beam of 4 (less than 1 in 1K).  If you want to see what's going into the deserializer you can use the flag ret_raw_gen=True (see below).  In this case the raw output is properly formatted AMR so you could just use it as is.  However, this won't always be the case.
Just a little more info to see this...
stog = amrlib.load_stog_model(model_dir='amrlib/data/model_parse_t5', device='cpu', ret_raw_gen=True)
outs = stog.parse_sents(['Making certain distinctions is imperative in looking back on the past'])
This produces... ( imperative :domain ( distinguish-01 :mod ( certain ) ) :time ( look-01 :ARG1 ( past ) :direction ( back ) ) )

Thanks! Appreciate the quick response. Will look out for your fix.

I pushed a fix (bc6acff).  beam=1 shows 0.815 smatch.
Need to retest with full beam=4 dataset and add the above sentence to the unit test cases.

After fix:
AMR Test Set (beams=4, no-wiki) SMATCH -> P: 0.836,  R: 0.803,  F: 0.819    (0 did not deserialize)
Unit test case added: https://github.com/bjascob/amrlib/commit/519b0af67f9cd57df5b86b85332bd2dd597d93cb

I have actually found a few more sentences that break the parser. Not sure why, but figured I'd pass these along.
Input: Researchers have also attempted to estimate the effect of employer-sponsored 401(k) plans on personal saving.

Err Message:
gid=x Initial node constructed when triples not empty.
Failed to deserialize, snum=0, beam=0
gid=x Initial node constructed when triples not empty.
Failed to deserialize, snum=0, beam=1
gid=x Initial node constructed when triples not empty.
Failed to deserialize, snum=0, beam=2
gid=x Initial node constructed when triples not empty.
Failed to deserialize, snum=0, beam=3

Input: Soap opera is now getting recognized as a fit topic for analysis.

Err Message:
gid=x Initial node constructed when triples not empty.
gid=x Missing starting paren for node o/opera
Failed to deserialize, snum=0, beam=0
gid=x Initial node constructed when triples not empty.
gid=x Missing starting paren for node o/opera
Failed to deserialize, snum=0, beam=1
gid=x Initial node constructed when triples not empty.
gid=x Missing starting paren for node o/opera
Failed to deserialize, snum=0, beam=2
gid=x Initial node constructed when triples not empty.
gid=x Missing starting paren for node o/opera
Failed to deserialize, snum=0, beam=3

These appear to have minor formatting errors coming off the T5 model and the deserializer can't figure out how to correctly reconstruct them.  Unfortunately there's a bit of random chance in the transformer model training and there's no way to guarantee it won't make small errors.  I tried parsing these with the previous model parse_t5-v0_1_0 and it parsed them without issue.  You could always use that as a fallback for any failures.
There is also parse_spring model which can still suffer from the same type of issue but it has a completely difference deserializer and anecdotally seems to be a little more resilient to formatting errors.