small Meckel's cartilage

New term:
small Meckel's cartilage (parent of short Meckel's cartilage MP:0013933)
DEF: reduced size of the cartilage bar in the mandibular arch that forms a temporary supporting structure in the embryonic mandible and gives rise to middle ear bones and sphenomandibular and anterior malleolar ligaments
SYN: small Meckel cartilage
SYN:  small mandibular cartilage
SYN: decreased Meckel's cartilage size
SYN: decreased Meckel cartilage size
SYN: decreased mandibular cartilage size
SYN: reduced Meckel's cartilage size
SYN: reduced Meckel cartilage size
SYN: reduced mandibular cartilage size
Anna

Added small Meckel's cartilage MP:0030026 but was unable to make it a parent of short Meckel's cartilage MP:0013933
Anna

This should be inferred provided axioms are in place, but I'm curious as
to why you were unable to assert this?
On 24 Jul 2017, at 7:08, anna-anagnostop wrote:

Added small Meckel's cartilage MP:0030026 but was unable to make it a
parent of short Meckel's cartilage MP:0013933
Anna
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/obophenotype/mammalian-phenotype-ontology/issues/2265#issuecomment-317433960

I believe the issue is the 's in the term label. I've run into the same thing with other terms with 's. I believe this is also why Anna was unable to create axioms for the Meckel's cartilage terms as well. There are existing terms with 's in the axioms but my guess is these were put in using a different version of Protege.

According to Cindy she had to create these axioms by making modifications in a text editor. @cmungall Is there anyway to work with terms containing apostrophes in the label without resorting to a text editor?

We ran into the same issue:
https://github.com/geneontology/go-ontology/issues/13881
We discussed the solution on a call but I don't remember what it was! You could pop over to @ukemi's office and ask him.

The solution is to change the View to 'render by entity IRI short name (Id)' then make the edit or add the axiom. You do need to have the IDs of the terms you want to use noted down but this works. (David popped up to show me how)


Support the inclusion of commit/version of test in ImageStats

Currently ImageStats only keep track of the GraalVM/Mandrel version (generator_version) used to generate them, but when it comes to the benchmark it only keeps it's name (img_name).
That way when inspecting the data if we observe something that needs investigation there is no way to know which specific version of the benchmark was used. The best we can do now is to look for the commit hashes in the corresponding repository based on the timestamp.
PerfStats include this information under quarkusVersion.
I believe we should introduce a similar entry for ImageStats as well (testVersion ?).
This would allow us to have the Quarkus CI (which uses a fixed/stable Mandrel image in contrast to our CI) push stats to the collector with every run of the CI on the main branch (or even for each PR) enabling us to (more) easily trace the commits that caused some regression.
cc @jerboaa

The point of ImageStats is to be a verbatim copy of the build-output.json files. Or at least it's supposed to be GraalVM/Mandrel specific. It's now lagging behind since that json output gained some more info since ImageStats got added. Nevertheless the way we feed this to the collector is here:
https://github.com/graalvm/mandrel/blob/82e6ba445d7c6dea8e4cdbdf2c8bf0ce8b527753/.github/import_stats.sh#L29-L41
So maybe we need some update for a specific ImageStat record to get a test_version added which would be amounting to the quarkus git sha retrieved elsewhere or something like that. It should be an optional field.


Add option with logo

For testing/community feedback: add options to provide badge as esip:term, logo:term or logo+esip:term.

Skipping the logo + text given the double-posting aspect of it.
The badge generator needs to be modified to deal with the differing widths and where to inject the image element.

See ec2e7504d6 and examples on the readme (and the notebook)


Evaluate bigcode/octogeex

https://huggingface.co/bigcode/octogeex
Something seems to be wrong with this model, I get absolute nonsense out:
5/28 SanitySecretIdentityMap python
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

 " " " " " ""  f Question Question Question

6/28 SanitySecretIdentityMap javascript
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

1233选民atasetett用了())()))()))()))()))()))()))(@(()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))()))            But ...       Hom  L W O P PO PT PA SAMSUMOEFILE为什么byNABBYB兴业 bang betweenRRSSSSAANAABABLAALF5램)),]),]),]((((((((((((((((((((((((((((((((((((())s00000000000000000000001111111111111111111111111111111111111111111111111111111111111

7/28 GCDZeroShot python
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

:::::::::::::::::::::111111111BLA11稳重mpty Inspired Interimated Is Is Is Is Is Is Is Is Is                               ) ) )

Need to look into whats up.

Doesn't work at all with latest transformers
  File "/root/.cache/huggingface/modules/transformers_modules/bigcode/octogeex/801fa5288006d64aaf93f68e69cff28cf80999ec/tokenization_chatglm.py", line 108, in vocab_size
    return self.tokenizer.n_words
AttributeError: 'ChatGLMTokenizer' object has no attribute 'tokenizer'


Make lxa-iobus-server compatible with python 3.10

I've encountered two issues when running lxa-iobus-server with Python 3.10:

A warning that pops up once when starting the server:
bin/lxa-iobus-server:106: DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
starting server on http://localhost:8080/

A warning that pops up very frequently:
WARNING:LXAIOBusServer:Exception during get_info() for node <LxaNode(address=00000507.00000002.00000003.00000001, node_id=1, driver=<lxa_iobus.node_drivers.Iobus4Do3Di3AiDriver object at 0x7f1d51224cd0>)>: TypeError("wait_for() got an unexpected keyword argument 'loop'")

This MR should fix both of these warnings.
The change removing the loop= parameter from wait_for works for python versions 3.7 and higher.

In Python 3.7 and Python 3.9 this change leads to the following exceptions:
RuntimeError: Task <Task pending coro=<LXAIOBusServer.flash_worker() running at /home/chris/work/Projects/github/lxa-iobus/lxa_iobus/server/server.py:122>> got Future <Future pending> attached to a different loop
ERROR:LXAIOBusServer:flashing failed
Traceback (most recent call last):
  File "/home/chris/work/Projects/github/lxa-iobus/lxa_iobus/server/server.py", line 122, in flash_worker
    shutdown, node, file_name = await self.flash_jobs.get()
  File "/home/chris/.pyenv/versions/3.7.3/lib/python3.7/asyncio/queues.py", line 159, in get
    await getter

The server seems to work but logs hundreds of messages a second.


in testing and contribution docs, use "main" instead of "master"; closes #272

I need to also actually rename the branch itself.

Now the branch has been renamed.


Use default import with latest Axios

With the axios upgrade, referencing the default export for require() is needed depending on the builds. This fixes that.

coverage: 30.234%. remained the same
when pulling 68d912eb5a2d1869d4260658586ab6782b149942 on hoijnet:main
into a349e945c80c53a638faefb3c33fddc3d2a7aa5e on terminusdb:main.


Tagging Jim Chiazzese for SURPASS work

@kazlauskis you added Jim to the repo on 21/05 but I cannot seem to tag him?
Please can you explain or tag Jim's account name?

He hasn't yet accepted the invitation.

I think the invitation may have come through prior to me having a github account? So that when trying to accept the invite I was prompted to first set up an account, which meant the original invite wasn't actually accepted? Could that be the reason I can't be tagged? @kazlauskis @CCarvell

Yes, that's probably the case, I've resent the invite now.

Thanks @kazlauskis That's sorted it now. Jim


Feature request : Foxx app code coverage

My Environment

ArangoDB Version:        3.6.4
Storage Engine:             RocksDB
Deployment Mode:         Single Server
Deployment Strategy:    ArangoDB Starter in Docker
Configuration:
Infrastructure:               Ubuntu / Docker
Operating System:        Ubuntu 20.04
Total RAM in your machine:        8G
Disks in use:        HDD
Used Package:            Docker - official Docker library

Component, Query & Data
Affected feature:
Foxx
Steps to reproduce

foxx test  -v /my-app

Problem:
I wish I could have code coverage while I run my integration tests
Expected result:
Any form of code coverage

probably related to https://github.com/arangodb/arangodb/issues/1538


Add Wheel of Fortune art

Zero To Mastery - Canvas 2022 🎉🎉
👉 Please remove the below and replace with your own values, leaving the headers where they are. 👈
PR contains:
A canvas depicting a spinning Wheel of Fortune
Breaking changes
None
Screenshots

Hey @royranger
Long time! How's it going?
Hope you are doing well 😄

Hey @royranger Long time! How's it going? Hope you are doing well 😄

Hey @MattCSmith !!! I'm doing good, hope you are too! I was sad to miss Hacktoberfest last year so popping in now 😁  Thanks for the merge! ⭐


generate_from_csv: adjust min_standing_time

The min_standing_time is a fixed value now. I thought about adding a feature that calculates the min_standing_time from the vehicle battery capacity and the charging power. We could then assume that there needs to be enough time to load the vehicle at least e.g. 80% of it's soc. This might just work for a constant loading curve though. @jakobgemassmer ,what do you think?
this is relevant for generate_from_csv.py and the bus_tool

I think the feature makes sense, but the question is "what is the reason for min_standing_time?"
Time for cleaning/loading etc. or time for little recharging? However I think this could be added at a later stage.

The min_standing_time (at depot) is very significant, because it sets the time for a vehicle to fully charge at the depot. If it is set to None / 0 the vehicles are assigned first in first out, so it might happen that a vehicle does another trip / rotation even though it has not been fully charged. The method above should help to define this time needed to fully charge (or at least load up to 80%)

new idea:
sum up consumption of all previous trips since last connect_cs = 1.
standing_time_deps_depb = sum_consumption / cs_power

Would recharge_fraction then be the same as min_soc?

No, he recharge fraction is just the fraction of the capacity that should be recharged. It does not say anything about the actual SoC of the vehicle at the time leaving.
One example:
capacity = 400 kWh
recharge fraction = 0.5
cs_power = 100 kW
-> the vehicle stands for at least 2 hours to recharge 200 kW.
If the vehicle comes in with an SoC of 0.1, this would mean it would leave with soc 0.6, if it comes in with 0.5 it would leave with SoC of 1

Ok, I understand your point. Making it the same as min_soc would ensure that min_soc can be reached. Bus is this always the case? Maybe one wants to give the vehicles more time to load (recharge_fraction = 1), so that the vehicles do not need the full cs_power, even though min_soc = 0.8? So the question is just, if we want another variable or only the min_soc.

You are right, another variable seems reasonable.


Pressure

We have an issue with pressure. If you run the simulation for a step, you get a huge negative number as pressure. Plus it is constant all over the place. Any Ideas?
With shear flow it works only a bit better - at least there's a red:blue profile, but it is still a negative number.
With Karman Vortex street, however,  it seems to be working.

closing for now...


Blocked 909 output package

Description
Create output module
Changes
Make generatedObject common
Make databag extend GeneratedObject
Move output config and guice to output module
Rename OutputFormat to OutputWriterFactory
Additional notes
Remaining work is moving the iteration visualiser from the generator to the orchestrator. this is the last remaining tie between generator and output
Issue
Resolves #909

I have noticed this branch introduced a bug into the velocity feedback...

I've fixed this: the code refactor had taken out the call to VelocityMonitor.rowEmitted() that was incrementing the output count.

I have noticed this branch introduced a bug into the velocity feedback...

I've fixed this: the code refactor had taken out the call to VelocityMonitor.rowEmitted() that was incrementing the output count.

Update: realised I have only fixed this for the reductive walker; it still occurs for the cartesian walker.  Still working on resolving.

Update: realised I have only fixed this for the reductive walker; it still occurs for the cartesian walker. Still working on resolving.

Velocity feedback now working for both walkers


what if "set token" dialog is shown always the first thing on the IDE start-up?

after all, users use IDE to submit code...
the reason is that if not, then even if an A+ project, then "Exercises" are empty

I can see the benefit, but I suspect showing a modal dialog at every start up could become very annoying to some users

a checkbox for "don't show this to me until needed (exercise submission)", because, all in all, we expect them to submit from the IDE aren't we?

Not a course project -> token missing (as a label)
refresh action shows setToken dialog when triggered (for exercises ;-) )


Add migrations for favorites  #28

Description
Databases: Create Migrations for Favourites Table
Fixes #28
How to test?
Please provide a short summary how your changes can be tested?
Checklist

[ ] I have performed a self-review of my own code
[ ] I have followed the name conventions for CSS Classnames and filenames, Components names and filenames, Style filenames, if you are in doubt check the the project README.MD and here https://github.com/HackYourFuture-CPH/curriculum/blob/master/review/review-checklist.md
[ ] I have commented my code, particularly in hard-to-understand areas, if you code was simple enough mark the box anyway
[ ] I have made corresponding changes to the documentation, if you code was simple enough mark the box anyway
[ ] This PR is ready to be merged and not breaking any other functionality

Nice effort @NataliaS1994  !
Have you tried using db:setup from package.json?
Does db:setup follow your changes successfully :) ?

Thank you @h09shais. I tried to npm run db:setup but I received an error that no user_id found. I think it's because there is no user database delivered yet.

Create the user table and set the relation then 😀!


Plotting in RTVS will disconnect session

Running any plot in RTVS will stop my R session.  Using RTVS version 1.3.40517.1016 in VS2017 Community.
From event viewer:
Faulting application name: Microsoft.R.Host.exe, version: 1.3.31012.1601, time stamp: 0x5afdb984
Faulting module name: Microsoft.R.Host.exe, version: 1.3.31012.1601, time stamp: 0x5afdb984
Exception code: 0xc0000005
Fault offset: 0x000000000007b038
Faulting process id: 0x1cb0
Faulting application start time: 0x01d40c26ad8903b2
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\R Tools for Visual Studio\Microsoft.R.Host.exe
Faulting module path: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\R Tools for Visual Studio\Microsoft.R.Host.exe
Report Id: cabb97fa-b6f1-484e-b30c-fd60f39e865b
Faulting package full name:
Faulting package-relative application ID:
What I see in R:  plot(1:100)
Interactive Window is disconnected from R session.
Click "Reset" to start a new session.
R session is stopped. Click "Reset" to start a new one.
Starting R session...

What if you open a plot window (R Tools->Plots->New Plot Window) after the Interactive window started, but before issuing the plot command?
What distro+version of R are you using?

I just tried that, and actually just opening the plot window disconnects my session.  I've also had it where I keep the plot window docked, then try issuing a plot command, same result.
I'm using R version 3.5, though I've also tried it with 3.4.3 (which was the last version I used in 2015 when it worked).  I've installed VS2015 to see if that was the issue, then tried Microsoft R Client 3.3.2.  Neither worked.
To make sure my version of R works, I've issued the same commands in RStudio, with the plots working fine.

Is there anywhere else I can post this issue?  I've exhausted options, from here on github, to the VS community.  Any help would be greatly appreciated.

Update: I see now that any new plot window will only ever display "R Plot - Device 0" when I open a new plot window or try plotting from R Interactive.  On systems where I have RTVS working (different laptop), it always has a positive integer.  Is there a way to make RTVS use a correct plotting device?

Hi. I’ll ping the dev to see if he can provide a fix or workaround. Stay
tuned.
On Sun, Jul 22, 2018 at 6:42 PM jichavez3<EMAIL_ADDRESS>wrote:

Update: I see now that any new plot window will only ever display "R Plot

Device 0" when I open a new plot window or try plotting from R
Interactive. On systems where I have RTVS working (different laptop), it
always has a positive integer. Is there a way to make RTVS use a correct
plotting device?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/RTVS/issues/4325#issuecomment-406915730,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtUVM4xOS9UKTtQ1nz8B4ikvORgJT87ks5uJSoUgaJpZM4U1a0D
.

Ah, I see Hugues has already replied. I’ll reping.

You can use the standard dev.* functions to manipulate the graphics devices.
dev.new() will create a new plot window in the ide. dev.list() and dev.cur(num) can change the active. dev.off() to close a graphics device / window. This is what the UI plot windows use internally. A device number of 0 is bad news, as you've noticed. It means it wasn't able to translate the internal device id to the public device number.
You can experiment with those commands and compare with your working machine to see if there is a clue there.
Note that you can instantiate a regular RGui graphics device:
windows()
Then any plot operation will draw in the external window.
You can also change the default to always use that instead of RTVS plot window (best done right after reset, or after you've closed all graphics devices with dev.off()).
options(device='windows')
or to return to the RTVS plot window as the default for the session:
options(device='ide')
I doubt it is size related issue at this point, but there's Window -> Reset Window Layout that you can try.


Add instructions to set up integration testing to contributor docs

It looks like the integration tests need these variables set in order to run properly. I didn't see it documented anywhere so I added it to the setup guide.
Also, is my project misconfigured or have the integration tests been broken for a a few weeks now? It looks like they have been broken since this commit, where you add new arguments to the function invalidate_edges, but don't update its uses in test_temporal_operations_int.py.

[!IMPORTANT]
Add environment variable setup instructions for integration tests to CONTRIBUTING.md.

Documentation:

Adds instructions in CONTRIBUTING.md for setting environment variables required for integration tests: TEST_OPENAI_API_KEY, TEST_OPENAI_MODEL, NEO4J_URI, NEO4J_USER, NEO4J_PASSWORD.

This description was created by  for c97a0666ba8a69f37ae3c7eb8e513e007e1380b1. It will automatically update as commits are pushed.

I have read the CLA Document and I hereby sign the CLA

thanks for contributing, @kylediaz . We're working to fix the int tests.


Bugged resolution or UI size

When installed the UI was too small for me so I unclicked the auto and changed the font to 10. This made the font so big I cant pull any menu up or see it when I try. I unistalled this add on and it is still too big in game I now have no idea what to do to get this back to being playable.

Can you provide more info on this? Maybe a picture?


Alert via CodeQL: DOM text reinterpreted as HTML

DOM text reinterpreted as HTML
(there's literature within the alert suggesting how to fix)

Even tho I added staging to codeql.yml on master, it looks like this isn't gonna register until the next staging->master sync because https://github.com/QURIresearch/squiggle/pull/137 went into staging. leaving issue open until codeql automatically updates/removes advisory.

#141 went in and staging is now default branch, this alert disappeared :)


Documentation improvement: HTML documentation

While it looks like most functions are classes are documented with numpy-style docstrings, I would highly encourage setting up HTML documentation. The Scientific Python Cookiecutter has an excellent template and guide to help you do this relatively quickly. This would not only automatically render the docstrings into an HTML API guide, but would also allow narrative documentation to be included, which would help users better understand the software. Regrettably, for reasons I don't understand, the CI portion of the Scientific Python Cookiecutter guide are focused on Travis, which has largely been dropped as the standard for open-source projects in favor of GitHub Actions. But, GitHub Actions can generally be readily set up (for automated testing and documentation publishing) by copying and pasting workflows from other projects (for example, you could copy the workflows from mikeio).

This is a great idea and definitely something I will work on in the future. At this stage, I feel like the package is sufficiently documented by the examples and the numpy-style docstrings


Improve default styling and top-level content layout

As noted by @baronfel in #27, the default styling and top-level layout needs improvement.
What is here is the current default styling for FSharp.Formatting using the standard _template.html, which is tight for iteration but we can do better. If necessary we can add a _template.html or fsdocs-styles.css in here  though to be honest I'd love it if we could make the default FSharp.Formatting _template.html good enough to be both configurable for the needs of this repo and to satisfy and please.
Please make a note below everything that you do/don't like about the current design, layout and content.  If possible give specific (preferably visual) examples of what you want improved, and, if possible, give links to how those improvements might be made in terms of CSS improvements and JS framework changes.
The things I've noticed so far:

The scrollbar on the left appears even if its not needed.  This is I believe because of horizontal overflow.

Agreed with @baronfel the basic look is very sparse (though I quite enjoyed browsing it on my phone earlier).  Also this is tricky as the site is not like most other doc sites as it is only doing FSHarp.Core API docs and is not a general tech doc site.

The repetition of parameter XML doc with parameter types is a bit painful (this is an F#F issue)

There is a glitch iwhere function names and parameters run together for active patterns and some module functions

Agreed with @cartermp that the link on the logo should go to https://fsharp.org

@cartermp @Krzysztof-Cieslak @baronfel
I copied over the default templates and styling and you guys can hack away on the design, fonts, JS frameworks used and so on to your heart's content.
Once you're done we can copy the results back into FSF as a default template
You may need to improve the style tagging in the generated HTML. I'd like FSF to document the style tags emitted and make sure we don't change them too much

I've also added a front page which I think greatly helps the site, though it all needs good styling


Does not load even though dependencies are satisfied

Hello. I installed the program and have python 3.5.2 & requests library installed, however, starting up the program gives an error:
Traceback (most recent call last):
	File "/usr/local/bin/ai-dungeon-cli", line 18, in <module>
		cfg = yaml.load(cfg_raw, Loader=yaml.FullLoader)
AttributeError: module 'yaml' has no attribute 'FullLoader'

It appears that there is an error with the yaml package. However, pip3 doesn't want to install yaml, as it doesn't appear to be a package that can be installed, so it's probably a python-inbuilt package like math. Does anyone know what might be causing this?

#5
This PR should solve this issue

Merged https://github.com/Eigenbahn/ai-dungeon-cli/pull/5, issue should be solved.

It didn't work, until I deleted the pyyaml and reinstalled it (it appears that the distro-installed pyyaml was broken).

It might be due to older versions not accepting a secondary argument for load().
I'm waiting final edits to https://github.com/Eigenbahn/ai-dungeon-cli/pull/4 in order to merge and will then tackle this issue.
It's only a matter of precising the version in requirements.txt.
For information, here is the version I've got on my system:
$ pip freeze | grep -i yaml
PyYAML==5.1.2

Should be fixed by https://github.com/Eigenbahn/ai-dungeon-cli/commit/e16b8af29a28852c252f2031710c6e24cc81c8dc (+ https://github.com/Eigenbahn/ai-dungeon-cli/commit/f253c374e06668f37d687a70fce7b91c215899be).

Yes, the new installation system fixed it. However, I have a new error, it appears to be in the code: user_input has no method starts_with. I will make a PR to fix this.

Edit: looks like it's fixed in the newest comment, nevermind about the PR.


fix undefined behavior in use of MaybeUninit

The previous commit used MaybeUninit in a way that was undefined behavior. This should be a fix.
(See https://doc.rust-lang.org/nightly/std/mem/union.MaybeUninit.html)

Here's a better plan though: don't use MaybeUninit at all.
Just put let mut val = 0;

Thanks!


Little problem

Hello.
Sometimes i have error in my node.js srcript.
app-0 (err): Error: Redis connection to /tmp/redis.sock failed - write EPIPE
app-0 (err):     at exports._errnoException (util.js:907:11)
app-0 (err):     at WriteWrap.afterWrite (net.js:785:14)

Before that, the redis begins to work very slowly.
nginx + PHP-FPM + nodejs + redis + laravel

Is this problem specific to the hiredis library at all?


GPHeteroscedasticRegression: RuntimeError: maximum recursion depth exceeded

Dear all,
I'm running into an error I wasn't able to fix when trying to create a new GPHeteroscedasticRegression model with the following input::
import GPy
import numpy as np
X = np.array([[  3.,   2.],[  3.,   3.],[  3.,   4.],[  3.,   5.],[  3.,   6.],[  3.,   7.],[  3.,   8.],[  3.,   9.],[  3.,  10.],[  3.,  11.],[  3.,  12.],[  3.,  13.],[  3.,  14.],[  3.,  15.],[  3.,  16.],[  4.,   1.],[  4.,   2.],[  4.,   3.],[  4.,   4.],[  4.,   5.],[  4.,   6.],[  4.,   7.],[  4.,   8.],[  4.,   9.],[  4.,  10.],[  4.,  11.],[  4.,  12.],[  4.,  13.],[  4.,  14.],[  4.,  15.],[  4.,  16.],[  5.,   1.],[  5.,   2.],[  5.,   3.],[  5.,   4.],[  5.,   5.],[  5.,   6.],[  5.,   7.],[  5.,   8.],[  5.,   9.],[  5.,  10.],[  5.,  11.],[  5.,  12.],[  5.,  13.],[  5.,  14.],[  5.,  15.],[  5.,  16.],[  6.,   1.],[  6.,   2.],[  6.,   3.],[  6.,   4.],[  6.,   5.]])
Y = np.array([[ 1.26009194],[ 1.11705572],[ 1.13733145],[ 0.90361428],[ 0.31151429],[-0.00558046],[-0.45493481],[-1.86474115],[-2.26985665],[-1.99614905],[-2.03313866],[ 0.14434941],[-0.85062491],[ 1.16469004],[ 1.08151982],[ 1.67174103],[ 1.1953489 ],[ 1.20362062],[ 0.91167117],[ 0.79200784],[ 0.12435543],[-0.66541966],[-0.3485895 ],[-2.39563241],[-2.08714596],[-1.99207315],[-2.17094947],[-0.0557339 ],[-0.29852262],[ 1.15457807],[ 0.9359406 ],[ 1.64350245],[ 1.06345797],[ 1.11817778],[ 0.61620098],[ 0.84620316],[ 0.19535039],[ 0.55819459],[-0.1007079 ],[-0.94839847],[-1.16807178],[-1.40530524],[-0.92938125],[ 0.86822105],[ 0.80256377],[ 0.84409863],[ 0.69201983],[ 1.64394879],[ 1.00750241],[ 1.09532067],[ 1.06837191],[ 0.86088544]])
kernel = GPy.kern.RBF(X.shape[1], ARD=True)
kernel += GPy.kern.White(X.shape[1])

This works::
m = GPy.models.GPHeteroscedasticRegression(X[:-1,], Y[:-1,], kernel)

This fails::
m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel)

This is the (repetetive) error::
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 964, in _name_changed
self._remove_parameter_name(None, old_name)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 961, in _remove_parameter_name
self._connect_parameters()
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 259, in _connect_parameters
self._add_parameter_name(p, ignore_added_names=ignore_added_names)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 940, in _add_parameter_name
warn_and_retry(param, _name_digit.match(other.name))
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 935, in warn_and_retry
param.name = match.group('name') + "_" + str(int(match.group('digit'))+1)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 315, in __setattr__
object.__setattr__(self, name, val);
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 243, in name
self._parent_._name_changed(self, from_name)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 964, in _name_changed
self._remove_parameter_name(None, old_name)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 961, in _remove_parameter_name
self._connect_parameters()
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 259, in _connect_parameters
self._add_parameter_name(p, ignore_added_names=ignore_added_names)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 940, in _add_parameter_name
warn_and_retry(param, _name_digit.match(other.name))
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 935, in warn_and_retry
param.name = match.group('name') + "_" + str(int(match.group('digit'))+1)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 315, in __setattr__
object.__setattr__(self, name, val);
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 241, in name
self._name = name
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 309, in __setattr__
pnames = self.parameter_names(False, adjust_for_printing=True, recursive=False)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 706, in parameter_names
else: names = [adjust(x.name) for x in self.parameters]
RuntimeError: maximum recursion depth exceeded

I've checked that the error is not related to the values of the last entries, but
rather to the lengths of X and Y...
I'm using GPy v.0.6.0 installed from Pypi.
Any idea what I'm doing wrong here?
Thanks a lot,
Elke

I got similar error before. It seems the current setting only allow the
length of X or Y smaller than 50. I cannot remember exact limits.
but if you just want to assign fixed known error or variance to the
observed data point, Ricardo and Alan has provided a temporary solution but
just need to tune GPy a little bit:
Go to GPy/inference/latent_function_inference/exact_gaussian_inference.py
In line 48, add a flatten(), so that the line changes from:
diag.add(Ky, likelihood.gaussian_variance(Y_metadata))
to:
diag.add(Ky, likelihood.gaussian_variance(Y_metadata).flatten())
Then, you won't use the heteroscedastic model any more, but you will build
your model like this:
heteroscedastic_variance = np.array([....]) # Here you define your variances
g = GPy.likelihoods.Gaussian(variance=heteroscedastic_variance)
m = GPy.core.GP http://gpy.core.gp/(X=X, Y=Y,
kernel=GPy.kern.RBF(X.shape[1]), likelihood=g)
Now you fix the variance (you have to fix it, the model won't be able to
optimize it otherwise)
m.likelihood.variance.fix()
m.optimize() # Everything else should work as before.
cheers to Ricardo and Alan
On 30 March 2015 at 17:00, Elke Schaper<EMAIL_ADDRESS>wrote:

Dear all,
I'm running into an error I wasn't able to fix when trying to create a new
GPHeteroscedasticRegression model with the following input::
import GPy
import numpy as np
X = np.array([[  3.,   2.],[  3.,   3.],[  3.,   4.],[  3.,   5.],[  3.,   6.],[  3.,   7.],[  3.,   8.],[  3.,   9.],[  3.,  10.],[  3.,  11.],[  3.,  12.],[  3.,  13.],[  3.,  14.],[  3.,  15.],[  3.,  16.],[  4.,   1.],[  4.,   2.],[  4.,   3.],[  4.,   4.],[  4.,   5.],[  4.,   6.],[  4.,   7.],[  4.,   8.],[  4.,   9.],[  4.,  10.],[  4.,  11.],[  4.,  12.],[  4.,  13.],[  4.,  14.],[  4.,  15.],[  4.,  16.],[  5.,   1.],[  5.,   2.],[  5.,   3.],[  5.,   4.],[  5.,   5.],[  5.,   6.],[  5.,   7.],[  5.,   8.],[  5.,   9.],[  5.,  10.],[  5.,  11.],[  5.,  12.],[  5.,  13.],[  5.,  14.],[  5.,  15.],[  5.,  16.],[  6.,   1.],[  6.,   2.],[  6.,   3.],[  6.,   4.],[  6.,   5.]])
Y = np.array([[ 1.26009194],[ 1.11705572],[ 1.13733145],[ 0.90361428],[ 0.31151429],[-0.00558046],[-0.45493481],[-1.86474115],[-2.26985665],[-1.99614905],[-2.03313866],[ 0.14434941],[-0.85062491],[ 1.16469004],[ 1.08151982],[ 1.67174103],[ 1.1953489 ],[ 1.20362062],[ 0.91167117],[ 0.79200784],[ 0.12435543],[-0.66541966],[-0.3485895 ],[-2.39563241],[-2.08714596],[-1.99207315],[-2.17094947],[-0.0557339 ],[-0.29852262],[ 1.15457807],[ 0.9359406 ],[ 1.64350245],[ 1.06345797],[ 1.11817778],[ 0.61620098],[ 0.84620316],[ 0.19535039],[ 0.55819459],[-0.1007079 ],[-0.94839847],[-1.16807178],[-1.40530524],[-0.92938125],[ 0.86822105],[ 0.80256377],[ 0.84409863],[ 0.69201983],[ 1.64394879],[ 1.00750241],[ 1.09532067],[ 1.06837191],[ 0.86088544]])
kernel = GPy.kern.RBF(X.shape[1], ARD=True)
kernel += GPy.kern.White(X.shape[1])
This works::
m = GPy.models.GPHeteroscedasticRegression(X[:-1,], Y[:-1,], kernel)
This fails::
m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel)
This is the (repetetive) error::
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 964, in _name_changed
self._remove_parameter_name(None, old_name)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 961, in _remove_parameter_name
self._connect_parameters()
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 259, in _connect_parameters
self._add_parameter_name(p, ignore_added_names=ignore_added_names)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 940, in _add_parameter_name
warn_and_retry(param, name_digit.match(other.name))
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 935, in warn_and_retryparam.name = match.group('name') + "" + str(int(match.group('digit'))+1)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 315, in setattr
object.setattr(self, name, val);
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 243, in name
self.parent._name_changed(self, from_name)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 964, in _name_changed
self._remove_parameter_name(None, old_name)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 961, in _remove_parameter_name
self._connect_parameters()
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 259, in _connect_parameters
self._add_parameter_name(p, ignore_added_names=ignore_added_names)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 940, in _add_parameter_name
warn_and_retry(param, name_digit.match(other.name))
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 935, in warn_and_retryparam.name = match.group('name') + "" + str(int(match.group('digit'))+1)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 315, in setattr
object.setattr(self, name, val);
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 241, in name
self._name = name
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.py", line 309, in setattr
pnames = self.parameter_names(False, adjust_for_printing=True, recursive=False)
File "/lib/python2.7/site-packages/GPy/core/parameterization/parameter_core.py", line 706, in parameter_names
else: names = [adjust(x.name) for x in self.parameters]
RuntimeError: maximum recursion depth exceeded
I've checked that the error is not related to the values of the last
entries, but
rather to the lengths of X and Y...
I'm using GPy v.0.6.0 installed from Pypi.
Any idea what I'm doing wrong here?
Thanks a lot,
Elke
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/187.

@mu2013
I tried your suggestion but I am experiencing problems with the predict function
I got the following error:
  File "/usr/local/lib/python2.7/dist-packages/GPy/core/gp.py", line 222, in predict
    mean, var = self.likelihood.predictive_values(mu, var, full_cov, Y_metadata)
  File "/usr/local/lib/python2.7/dist-packages/GPy/likelihoods/gaussian.py", line 88, in predictive_values
    var += self.variance
ValueError: operands could not be broadcast together with shapes (200,1) (100,1) (200,1)

where the training is done on 200 samples and I want to make a prediction on 100 samples.
Any suggestions how can I resolve that issue?

The solution I suggested is just a temporary fix which you can use to get
the optimised kernel parameter. But if you already have the optimised
kernel parameter, you can write a bit of code to do the prediction which is
just conditional multi variant normal.
If you insist to use Heteroscedastic model to do prediction, Ricardo is the
expert.
cheers
On 5 August 2015 at 14:57, borislavsm<EMAIL_ADDRESS>wrote:

@mu2013 https://github.com/mu2013
I tried your suggestion but I am experiencing problems with the predict
function
I got the following error:
File "/usr/local/lib/python2.7/dist-packages/GPy/core/gp.py", line 222, in predict
mean, var = self.likelihood.predictive_values(mu, var, full_cov, Y_metadata)
File "/usr/local/lib/python2.7/dist-packages/GPy/likelihoods/gaussian.py", line 88, in predictive_values
var += self.variance
ValueError: operands could not be broadcast together with shapes (200,1) (100,1) (200,1)
where the training is done on 200 samples and I want to make a prediction
on 100 samples.
Any suggestions how can I resolve that issue?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/187#issuecomment-128005104.

@ric70x7 , could you take a look at this please?

Try:
kernel = GPy.kern.RBF(X.shape[1], ARD=True)
kern   =  kernel + GPy.kern.White(X.shape[1])
m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel.copy())
The regression with kernels Fixed and White fails. Make sure to use the trained kernel.
Ref: https://github.com/SheffieldML/GPy/issues/196#issuecomment-114246090

Hi,
@borislavsm are you defining a new Y_metadata for the predictions? I would
need to see a piece of your code to be sure what is going on.
In any case, at the moment we don't have a routine to estimate new noise
terms for points outside the training set. Otherwise, for making
predictions at new points, it would be needed to pick a noise term from
those learnt in the training. Estimates of the latent variable at new
points are still available using m._raw_predict.
Cheers,
Ricardo
2015-08-17 15:24 GMT+01:00 Roc Reguant<EMAIL_ADDRESS>
Try:
kernel = GPy.kern.RBF(X.shape[1], ARD=True)
kern = kernel + GPy.kern.White(X.shape[1])
m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel.copy())
The regression with kernels Fixed and White fails. Make sure to use the
trained kernel.
Ref: #196 (comment)
https://github.com/SheffieldML/GPy/issues/196#issuecomment-114246090
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/187#issuecomment-131838201.

Hi,
@ric70x7, I define a new Y_metadata for the predictions. I just fixed the new noise terms for the points outside of the training dataset but I agree that their estimation is an interesting problem. The code that I used:
import numpy as np
import GPy

Ns = 100
#Generate data
X = 10*np.random.random((Ns,1))
y = np.sin(X) + np.random.random((Ns,1)) - 0.5
Xtest = np.linspace(np.min(X), np.max(X), 121)
Xtest = Xtest[:,None]

# Estimate model
heteroscedastic_variance = np.random.random((X.shape[0],1)) # Here you define your variances
g = GPy.likelihoods.Gaussian(variance=heteroscedastic_variance)
m = GPy.core.GP(X=X, Y=y, kernel=GPy.kern.RBF(X.shape[1]), likelihood=g)
m.likelihood.variance.fix()
m.optimize()

# Make predictions
noise_dict = {'output_index':1*np.ones_like(Xtest).astype(int)}
ytest = m.predict(Xtest, Y_metadata=noise_dict)[0]

The error is :
ValueError: operands could not be broadcast together with shapes (121,1) (100,1) (121,1)

Thanks

Ok, I see where the problem is. The Heteroscedastic model uses a different
kind of likelihood, than Gaussian, it is in essence a list of Gaussian
likelihoods -why is that? well at some point we might want to extend it to
a list of mixed noise models-.
When making predictions, this type of likelihood access each of the noise
terms through the Y_metadata dictionary. This does not happen with the
Gaussian likelihood that uses a vector of variances, not a single variance
term is extracted and there is when the incompatibility of the array shapes
appears.
So, the fastest solution at this point is the following:
ytest_mean, ytest_var = m._raw_predict(Xtest) # This is the prediction
of the latent variable (i.e., no noise added)
ytest_var += m.likelihood.variance[j] # Where y is the index of the
noise term you want to use.
Cheers,
Ricardo
2015-08-26 12:16 GMT+01:00 borislavsm<EMAIL_ADDRESS>
Hi,
@ric70x7 https://github.com/ric70x7, I define a new Y_metadata for the
predictions. I just fixed the new noise terms for the points outside of the
training dataset but I agree that their estimation is an interesting
problem. The code that I used:
import numpy as np
import GPy
Ns = 100
#Generate data
X = 10*np.random.random((Ns,1))
y = np.sin(X) + np.random.random((Ns,1)) - 0.5
Xtest = np.linspace(np.min(X), np.max(X), 121)
Xtest = Xtest[:,None]
Estimate model
heteroscedastic_variance = np.random.random((X.shape[0],1)) # Here you define your variances
g = GPy.likelihoods.Gaussian(variance=heteroscedastic_variance)
m = GPy.core.GP(X=X, Y=y, kernel=GPy.kern.RBF(X.shape[1]), likelihood=g)
m.likelihood.variance.fix()
m.optimize()
Make predictions
noise_dict = {'output_index':1*np.ones_like(Xtest).astype(int)}
ytest = m.predict(Xtest, Y_metadata=noise_dict)[0]
The error is :
ValueError: operands could not be broadcast together with shapes (121,1) (100,1) (121,1)
Thanks
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/187#issuecomment-134950922.

Fixed, use the HeteroscedasticGaussian likelihood.


Fix: correct legacy endpoint for get ISM policies

Signed-off-by: Ravi Thaluru<EMAIL_ADDRESS>Issue #, if available:
Description of changes:

Support the legacy endpoint - GET _opendistro/_ism/policies

CheckList:
[X ] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #24 (2a9e6a8) into main (06fbf04) will decrease coverage by 0.16%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main      #24      +/-   ##
============================================
- Coverage     78.02%   77.86%   -0.17%
+ Complexity     1860     1858       -2
============================================
  Files           255      255
  Lines          9988     9988
  Branches       1515     1515
============================================
- Hits           7793     7777      -16
- Misses         1375     1389      +14
- Partials        820      822       +2

Impacted Files
Coverage Δ

...statemanagement/resthandler/RestGetPolicyAction.kt
96.66% <100.00%> (ø)

...statemanagement/model/destination/CustomWebhook.kt
65.21% <0.00%> (-28.99%)
:arrow_down:

.../action/explain/TransportExplainTransformAction.kt
58.82% <0.00%> (-7.85%)
:arrow_down:

...dexmanagement/transform/model/TransformMetadata.kt
86.74% <0.00%> (-3.62%)
:arrow_down:

...earch/indexmanagement/transform/model/Transform.kt
90.10% <0.00%> (+1.04%)
:arrow_up:

...nt/rollup/action/stop/TransportStopRollupAction.kt
67.34% <0.00%> (+2.04%)
:arrow_up:

.../rollup/action/start/TransportStartRollupAction.kt
64.00% <0.00%> (+6.00%)
:arrow_up:

...ent/rollup/action/explain/ExplainRollupResponse.kt
85.00% <0.00%> (+10.00%)
:arrow_up:

...nt/indexstatemanagement/model/destination/Slack.kt
54.54% <0.00%> (+13.63%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06fbf04...2a9e6a8. Read the comment docs.


ARROW-5289: [C++] Move arrow/util/concatenate* to arrow/array

The array/ subfolder is a more natural place to organize algorithms and other code relating to manipulating columnar array objects

Codecov Report

Merging #4445 into master will increase coverage by 1.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4445      +/-   ##
==========================================
+ Coverage   88.42%   89.44%   +1.02%
==========================================
  Files         793      643     -150
  Lines      101335    89542   -11793
  Branches     1253        0    -1253
==========================================
- Hits        89602    80090    -9512
+ Misses      11486     9452    -2034
+ Partials      247        0     -247

Impacted Files
Coverage Δ

cpp/src/arrow/array/concatenate.cc
91.5% <ø> (ø)

cpp/src/arrow/array/concatenate-test.cc
85.98% <ø> (ø)

cpp/src/plasma/thirdparty/ae/ae.c
70.75% <0%> (-0.95%)
:arrow_down:

cpp/src/arrow/util/thread-pool-test.cc
97.66% <0%> (-0.94%)
:arrow_down:

go/arrow/ipc/writer.go

go/arrow/math/uint64_amd64.go

go/arrow/memory/memory_avx2_amd64.go

go/arrow/ipc/file_reader.go

js/src/enum.ts

go/arrow/array/builder.go

... and 144 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3379ec2...002dfc9. Read the comment docs.

+1, LGTM


zstream trigger: fix trigger logic, add logs and test

Fix logic of trigger job when we have new zstream version.
Add more logs.
Add a test to cover new zstream version trigger.

Report bugs in Issues
The following are automatically added:

Add reviewers from OWNER file (in the root of the repository) under reviewers section.
Set PR size label.
New issue is created for the PR. (Closed when PR is merged/closed)
Run pre-commit if .pre-commit-config.yaml exists in the repo.

Available user actions:

To mark PR as WIP comment /wip to the PR, To remove it from the PR comment /wip cancel to the PR.
To block merging of PR comment /hold, To un-block merging of PR comment /hold cancel.
To mark PR as verified comment /verified to the PR, to un-verify comment /verified cancel to the PR.
verified label removed on each new commit push.
To cherry pick a merged PR comment /cherry-pick <target branch to cherry-pick to> in the PR.

Multiple target branches can be cherry-picked, separated by spaces. (/cherry-pick branch1 branch2)
Cherry-pick will be started when PR is merged

To build and push container image command /build-and-push-container in the PR (tag will be the PR number).
To add a label by comment use /<label name>, to remove, use /<label name> cancel
To assign reviewers based on OWNERS file use /assign-reviewers
To check if PR can be merged use /check-can-merge

Supported /retest check runs

/retest tox: Retest tox
/retest build-container: Retest build-container

Supported labels

hold
verified
wip
lgtm

/build-and-push-container

New container for quay.io/redhat_msi/ci-jobs-trigger:pr-84 published

New container for quay.io/redhat_msi/ci-jobs-trigger:pr-84 published

/verified

New container for quay.io/redhat_msi/ci-jobs-trigger:latest published


bond docs, js docs, better install instructions

Signed-off-by: Gregory Hill<EMAIL_ADDRESS>

Should js/README.md be removed? Also the top-level README.md links to js/ maybe that should be js-api.md


docs: add relay definition to glossary.

Add relay definition to glossary.
Use cases will expand on specific examples.

Related Issue

Other information

@cedoor Please review and merge. Thanks


Only use cookies for context

Frankly, I don't fully remember why we had the context store in Rails.cache - There are random uids and they point to data in Redis. We store the uid in the cookie and reference the data. But that data is only a "1" meaning it should stick to master. I assume we had bigger plans for it have more info, but it never happened.
Given that's all it is, it would be smart to be able to encode the "stick to master" bit in the Rails session or it's own cookie. Maybe a hash in case we need some other bit. This would remove the need for the backend store and be generally more manageable.

@bleonard What about API calls that do not use cookies?? https://github.com/taskrabbit/makara/issues/175

The old method requires cookies, too. It has to store the "session context" guid somewhere. One way or another, that contet has to be saved. In our iOS app (using AFNetworking), for example, there is a way to actually send the right cookies headers back and forth.

Hey @bleonard! This week I've been working on a CookieStore for Makara, since right now we are using it without stickiness because we don't set any kind of backend cache (we just use the NoopStore) and wanted to use cookies for this. While doing that, I realised that I didn't quite understand what the real intended behaviour for stickiness was, as it's implemented now.
Without looking much as the code, I understood that once you stick to master, you want that all operations happening within the set master_ttl for that config go to master as well. That's why you set that ttl larger than the replication lag you can tolerate. However, once I was deep in the code, what I saw is that only the first request after sticking (or the first two requests if the first one is a redirect) goes to the master as well. Let me explain:
These are the relevant lines, when deciding whether to use the master, and whether to stick or not for the next request https://github.com/taskrabbit/makara/blob/e95d4303d350c9ab9dfc41b1e5c2743a1849b9c4/lib/makara/proxy.rb#L201-L214
It's easier to see with an example. Suppose we start without a context, that is, without a cookie set or anything, and make 3 requests that take less than the TTL we have configured for sticking to the master. In each request we keep track of the previous context and the current context, and after the request we set the cookie with the current context.
Request 1, this one needs the master because it writes:
cookies = {}
previous_context = "P1" # Generated, since the cookie is not set
current_context = "C1" # Generated. If the previous request was a redirect, we'd set it to P1.
cache = {}

appropriate_pool = primary # Since it's a write operation
# Stick to master
cache["C1"] = true
# Response cookies: _mkra_ctxt=C1--201

Request 2, that doesn't need the primary because it only reads:
cookies = { "_mkra_ctxt" => "C1--201" }
previous_context = "C1" # Read from the cookie
current_context = "C2" # Generated, as the previous request was a 201
cache = {"C1" => true}

appropriate_pool = primary # It's not a write operation, but we are stuck to master: cache.read(previous_context) = true
# In this case we don't write to cache ⚠
# Response cookies: _mkra_ctxt=C2--200

Request 3, that doesn't need the primary because it only reads, and it's still within the master TTL:
cookies = { "_mkra_ctxt" => "C2--200" }
previous_context = "C2" # Read from the cookie
current_context = "C3" # Generated, as the previous request was a 200
cache = {"C1" => true} # This hasn't expired yet.

appropriate_pool = replica # ⚠ This is not a write operation, and cache.read(previous_context) = false, since we didn't write it before

Did I get this wrong? This means that we wouldn't be sticking to master for the duration of master TTL, only for the first request done during that TTL after sticking, or the first two if the write operation happens to redirect, which is usual in typical actions in Rails apps. I'm not sure if this was the initial idea, to stick only for one request, up to the TTL.
Still, even if that was the initial idea, there is another scenario in which this doesn't work: if you have several configurations at once (like having both Redis and MySQL using Makara). Cache keys account for this possibility, but _mkra_ctxt cookie is only one and you can only remember one context at a time. This means that the "stuck" context might get overwritten with a different context from a different config and server before you have the chance to "stick" a subsequent read request.
I hope I explained myself properly, sorry this was quite long! 😅
I decided to comment here in this issue because what you describe is the solution I had gone for, 2 days ago. Then I realised that if you have several configs at once, as it's our particular case, we can't rely only on the presence, as that'll stick all configs. Then, we'd need to ignore completely the context from the cookie, and use only the config_id, extracting it from the cache key. Then I just realised I'm not sure why you'd need the context at all, if the desired behaviour is to stick to the master for all requests received for the duration of master_ttl. All you'd need is to track whether a specific config_id is currently stuck or not. If you had only one config, then all you need is the presence of the data, as you said, to indicate that is stuck. If you have several configs, then you need to know what the current config is, and whether is stuck or not (two different cookies), but you don't need to differentiate between contexts for the same config.
I'll be really happy to submit a pull request with my idea, using only cookies, and not tracking context, just config id, but first I wanted to check whether I was missing anything about the stickiness behaviour based on what I described above, and anything else you might have in mind!

That was a fairly impressive opening comment :-)
Thank you.
If I'm reading it right, there are few things in there.

Would it be better to stick to master for whole TTL even if that's a few requests?
Should makara have multiple contexts or is one enough?

Number 1 seems right to me. We're not doing too many redirects and it's mostly API calls so have no obviously run into that before. In all cases, that seems more intuitive to me.
For number 2, I'd love to understand your config better and how Redis and makara are working. It seems possible that preserving the ability to have two contexts could be smart. It's not always the same data being written to each so Redis and MySQL could have different invalidation patterns.
This would influence the cookie storage. Maybe the cookie name has the ability to have a key, otherwise it defaults to having a single cookie. So, it could be "_mkra_ctxt_mysql" or something. Or we keep _mkra_ctxt and it has a hash with the keys and their status.
If you consider Number 1 while using cookies, the whole system actually gets much simpler because cookies already have TTLs. We could update the TTL when a write happens. Of course, maybe that's why you just want 1 context. Having both contexts in the same cookie actually moves TTL management to a timestamp in the cookie. So maybe the (optional) cookie naming ("_mkra_mysql") is the way to go. If the config has a cookie_key or something to that effect, load it from there, otherwise use the default ("_mkra_default") (probably a new default so we don't have parsing issues with the old _mkra_ctxt cookie).
What you store in the cookie is an in interesting question. It's tempting to store just a "1" or something, but we also want to think about what would happen if we want to store something else later. Does that mean do a hash now? Maybe or maybe not. I suppose would could still add some crazy delimiter later if there was new data: 1|futureproof? What you think?
If we do get this cookie thing going, I can't think of a decent reason to continue with the Redis context storage at all. It already needs cookies to make it work, so there's no win there. It's just a waste of network and storage. Of course, I didn't make that so maybe there's something brilliant I can't see. I have the feeling it just never expanded as predicted (as noted above).

Thanks for your reply and your notes @bleonard! You are correct, I was talking about two different things in my comment, and I didn't explain myself very well for the second! Number 1 is right, the behaviour of sticking to the master for all requests happening during master_ttl was what we personally understood from the README. The problem of sticking only for one request or two is that you might not get the behaviour you want if there are other requests happening on a page load (XHR requests to load other pieces separately), and they might be equally affected by replication lag.

If you consider Number 1 while using cookies, the whole system actually gets much simpler because cookies already have TTLs. We could update the TTL when a write happens.

Yes! This was what I had implemented, very simple, until I realised this wouldn't work if you had more than one config around! In our case it was MySQL + Redis, but it could have been two different databases, or two or more different Redis instances.

For number 2, I'd love to understand your config better and how Redis and makara are working. It seems possible that preserving the ability to have two contexts could be smart. It's not always the same data being written to each so Redis and MySQL could have different invalidation patterns.

Sorry! I didn't explain number 2 very well. We use Redis together with MySQL in a web context (we also use Redis and MySQL in the context of ActionCable, but that's not relevant in this case, as we wouldn't be using the middleware and reading and setting the cookies). This means that we might read/write from MySQL and Redis in the same request. A write from MySQL would result in setting the sticking state to true in our new cookie, and that would affect Redis as well. This is what I mean by "if you have several configs at once, as it's our particular case, we can't rely only on the presence, as that'll stick all configs". That's why the simple solution of relying on cookie presence and cookie max_age didn't work 😭
Notice that when using the backend cache this does work because the config id is part of the cache key. Moreover, if we stick to master for the whole duration of master_ttl, that's the only part of the key that matters! That's why I was very curious about needing the context at all, I don't think it's necessary if you stick for the whole master_ttl.

I didn't make that so maybe there's something brilliant I can't see. I have the feeling it just never expanded as predicted

Ah, yes! That sounds like a very plausible explanation.

What you store in the cookie is an in interesting question. It's tempting to store just a "1" or something, but we also want to think about what would happen if we want to store something else later. Does that mean do a hash now? Maybe or maybe not. I suppose would could still add some crazy delimiter later if there was new data: 1|futureproof? What you think?

Right! I've been thinking about what we'd need to store in the cookie so that this works with several configurations. I considered two alternatives:

Add the config id to the cookie name. As you said, something like "_mkra_ctxt_#{config_id}" or "_mkra_#{cookie_key}" where the cookie_key is specified in the config. I didn't like this too much because parsing the cookie before starting to process the request in the middleware became too complex, since you have to parse all possible cookies, so you'd need to keep all these config ids somewhere. They could be    _mkra_ctxt_mysql and _mkra_ctxt_redis, but that wouldn't allow for much flexibility in the future.
Store the config ids and a value in a hash, in a single cookie. This is what I wanted to try, and I didn't think you need keep anything else apart from the config id. That is, no previous_context or current_context anymore. About the value, 1 vs. something else, my idea was to store the timestamp until which the setting was valid. The reason is that If we store a hash in a single cookie, we can no longer rely on max_age to handle the expiration time for us. Each value can have potentially different expiration times.

Let me show how I thought this could work, with an example. Imagine we have a database.yml config for our main database, with id: mysql_main and master_ttl: 2, and another shared.yml config for the Redis instance we use in web, with id: redis_shared and master_ttl: 1. Imagine also that we get similar requests to what I used in my previous example: request 1 writes to MySQL and reads from Redis, request 2 reads from both MySQL and Redis, and request 3 reads from MySQL and writes to Redis.
Current time: 2018-02-05 07:51:09 UTC<PHONE_NUMBER>.483005)
Request 1 (2018-02-05 07:51:09 UTC<PHONE_NUMBER>.483005):
cookies = {}
(mysql) appropriate_pool = primary # Since it's a write operation
(redis) appropriate_pool = replica # Since it's a read operation
# Stick to master in mysql
# Response cookies: _mkra_ctxt=mysql_main:1517817071.483005

The value for mysql_main is basically current timestamp + 2 (master_ttl). That's when the stickiness will expire.
Request 2 (2018-02-05 07:51:10 UTC<PHONE_NUMBER>.000000)
cookies = { _mkra_ctxt=mysql_main:1517817071.483005 }
(mysql) appropriate_pool = primary # value from mysql_main in the cookie is > than current time => We stick
(redis) appropriate_pool = replica # it's a read operation and we don't have the config id in the cookie
# Response cookies: {}

Request 3 (2018-02-05 07:51:11 UTC<PHONE_NUMBER>.000000)
cookies = { _mkra_ctxt=mysql_main:1517817071.483005 }
(mysql) appropriate_pool = primary # value from mysql_main in the cookie is > than current time => We stick
(redis) appropriate_pool = primary # it's a write operation
# We must stick to master in Redis
# Response cookies: _mkra_ctxt=mysql_main:1517817071.483005|redis_shared:1517817072.000000

We keep mysql_main in the response cookies because it hasn't expired yet, and we add the new value for redis_shared, current time + 1 second (its master_ttl).
When we check stickiness and one of the keys in the cookie has expired, we can get rid of it. For the response cookies max_age we need to set one that allows all the timestamps stored to "survive".
What do you think?

Oh! A concern I had with the above solution was the possibility of exceeding the cookie size limit for browsers (assuming only modern browsers that support more than 4096 bytes per domain, something like 4096 bytes per cookie, and > 20 cookies per domain). But if we used MD5 hashes in hex for config ids by default, and store the timestamp with the above precision, we'd have something like this per config: 0aa1ea9a5a04b78d4581dd6d17742627:1517818126.148078. That is 50 bytes, then one byte more for the separator "|". 80 keys and values like that would take 4080 bytes. I think this gives plenty of room.

That's great @rosa - I'd love to see a PR and talk about more specific spots.
It looks like you have something going already :-) so let's go from there.
Very exciting.

I personally wouldn't worry too much about the config ids - can probably use the ones from the config file with a gsub(/weird stuff/, "_") or something. Of course maybe you have more experience with crazy configs :-)
It would be nice if they were readable in the cookie and likely be shorter than a MD5.

Thanks @bleonard! 😄

Of course maybe you have more experience with crazy configs :-)

Haha! Not really, we weren't defining any ids ourselves, just relying on the internal MD5 hash
that is set by default
https://github.com/taskrabbit/makara/blob/c6b158e847468379da8cc2eb69aee455bef386fb/lib/makara/config_parser.rb#L151-L156

It would be nice if they were readable in the cookie and likely be shorter than a MD5.

Oh, yes. Hmm... Perhaps we can truncate the MD5 and get a substring of size 16 (that'd be using 64 bits instead of 128), I think the probability of collisions is still super low, specially for this case in which we are talking about 2 or 3 different ids in most cases with different configurations. For specified ids with strange characters that wouldn't be readable in the cookie, I imagine we can use the MD5 of the id itself? In that way we'll always have the same format for all identifiers, but it'd lose the disadvantage of having an id easy to identify in the cookie, for Makara users that set their own ids. Maybe it's better to just use whatever id is set in the config in that case, and let users set reasonable ids?

Yes, that sounds good. I would allow them to set that one and use that.
If we use the above as a default, though, it's pretty still that it would change every deploy. This would have the effect of not correctly sticking to master across a deploy.
So we could have this method use the passed in one || 'default'
If they want two contexts (least common case, they can set an id)
This does make me think that the cookie management code has to clean up both the "expired" things in it as well as "unknown" config ids.

Why would it change in every deploy? That is, assuming that the deploy doesn't include a change to database.yml, or redis.yml, or any other config that is using Makara? Not sure if I'm missing something that the ConfigParser does here.
I think that unknown config ids in the cookie might be fine as long as they expire in a short time 🤔  Well, all depends on the values set by the developers writing Makara config (that is, master_ttl).

Sorry, not every deploy. I read it wrong. But when the a config changes. I still think default is a nice idea and gives the expected behavior without a hash looking thing and wouldn't change even when adding a new read-only server.
Yes, unknown ones would likely be fine as long as the code looks at all keys and their expire times regardless of being known or unknown configs. That was just a note to write it that way. It's just as likely that we were to write it only looking at the known ones.

Ahh, got it, thanks!
I'll try to put together a pull request with this, hopefully I'll have time this week! 😄🤞

release candidate: gem 'makara', git: 'https://github.com/taskrabbit/makara.git', tag:'v0.4.0'

referenving #208 here.


Merging of T-Digests produce slightly different results depending upon the order of merge.

Hi Ted,
I have been running some tests with your T-Digest Java code to understand the algorithm. I am using MergingDigest implementation. The dataset has about 60k samples. I wanted to slice the samples into three different sets (e.g. set1 = 1-15000, set2 = 15001-45000, set3=45001-60000). I wanted to create partial digests from each of these sets and then merge them into a final set. In two different runs, I have changed the order in which partial digests were added to make final digests.
I found that the final median was slightly different in two different passes(i.e. the final median is sensitive to the order in which partial digests were merged). However, the sensitivity also depends on compression factor. If I increase that parameter, eventually the dependency on merge order disappears. Note that this dependency also has connection with input sample distribution. I was able to see this with one sample dataset and not with some other samples. Is this expected from MergingDigest algorithm?
Thanks,
Santanu
Sample unit test code is shown below :
        private int CF = 500;
	private BufferedReader reader;
	private List<Double> samples = new ArrayList<Double>();

	private void readSamples() throws Exception {
		String line = reader.readLine();
		while (line != null) {
			double d = Double.parseDouble(line);
			samples.add(d);
			line = reader.readLine();
		}
	}

	private void run_test() {
		//create a TDigest
		MergingDigest m = returnNewDigest();
		for (int i = 0; i < samples.size() ; i++) {
			m.add(samples.get(i));
		}

		System.out.println("Median without split "+m.quantile(0.5));
	}

	private MergingDigest returnNewDigest() {
		return new MergingDigest(CF);
	}

	private void run_test_split() {
		int s1 = 15000;
		int s2 = 45000;
		int s3 = samples.size();

		MergingDigest m1 = returnNewDigest();

		for (int i=0 ; i < s1 ; i++) {
			m1.add(samples.get(i));
		}

		MergingDigest m2 = returnNewDigest();
		for (int i = s1 ; i < s2 ; i++) {
			m2.add(samples.get(i));
		}

		MergingDigest m3 = returnNewDigest();
		for (int i = s2 ; i < s3 ; i++) {
			m3.add(samples.get(i));
		}

		MergingDigest m4 = returnNewDigest();

		List<TDigest> l1 = new ArrayList<TDigest>();
		l1.add(m1);
		l1.add(m2);
		l1.add(m3);
		m4.add(l1);

		System.out.println("median with split (s1+s2+s3) "+m4.quantile(0.5));

		List<TDigest> l2 = new ArrayList<TDigest>();
		l2.add(m2);
		l2.add(m3);
		l2.add(m1);

		MergingDigest m5 = returnNewDigest();
		m5.add(l2);

		System.out.println("median with split (s2+s3+s1) "+m5.quantile(0.5));
	}

	@Test
	public void test() throws Exception {
		reader = new BufferedReader(new FileReader("src/test/resources/up.txt"));
		readSamples();
	    System.out.println("Number of samples "+samples.size());
		run_test();
		run_test_split();
	}

The output of the test
Number of samples 60398
Median without split 30.101212824980962
median with split (s1+s2+s3) 30.585493893059724
median with split (s2+s3+s1) 30.232216458636376

If I select CF = 1500 , I get the following results :
Number of samples 60398
Median without split 29.98999977111816
median with split (s1+s2+s3) 29.98999977111816
median with split (s2+s3+s1) 29.98999977111816

It is impossible to precisely determine quantiles in a single pass through
the data, except in happy accidents.
As such, t-digest is an approximation algorithm and the errors are not
deterministic if you don't know the order of the data. As you increase the
compression factor, the errors should always decrease or at least not
increase.
The value you get will also vary somewhat if you rearrange how you add the
data via intermediate t-digests.
All of this is expected.
On Wed, Nov 1, 2017 at 1:44 PM, schatterjee10<EMAIL_ADDRESS>wrote:

Hi Ted,
I have been running some tests with your T-Digest Java code to understand
the algorithm. I am using MergingDigest implementation. The dataset has
about 60k samples. I wanted to slice the samples into three different sets
(e.g. set1 = 1-15000, set2 = 15001-45000, set3=45001-60000). I wanted to
create partial digests from each of these sets and then merge them into a
final set. In two different runs, I have changed the order in which partial
digests were added to make final digests.
I found that the final median was slightly different in two different
passes(i.e. the final median is sensitive to the order in which partial
digests were merged). However, the sensitivity also depends on compression
factor. If I increase that parameter, eventually the dependency on merge
order disappears. Note that this dependency also has connection with input
sample distribution. I was able to see this with one sample dataset and not
with some other samples. Is this expected from MergingDigest algorithm?
Sample unit test code is shown below :
    private int CF = 500;

private BufferedReader reader;
private List samples = new ArrayList();
private void readSamples() throws Exception {
String line = reader.readLine();
while (line != null) {
double d = Double.parseDouble(line);
samples.add(d);
line = reader.readLine();
}
}
private void run_test() {
//create a TDigest
MergingDigest m = returnNewDigest();
for (int i = 0; i < samples.size() ; i++) {
m.add(samples.get(i));
}
  System.out.println("Median without split "+m.quantile(0.5));

}
private MergingDigest returnNewDigest() {
return new MergingDigest(CF);
}
private void run_test_split() {
int s1 = 15000;
int s2 = 45000;
int s3 = samples.size();
  MergingDigest m1 = returnNewDigest();

  for (int i=0 ; i < s1 ; i++) {
  	m1.add(samples.get(i));
  }

  MergingDigest m2 = returnNewDigest();
  for (int i = s1 ; i < s2 ; i++) {
  	m2.add(samples.get(i));
  }

  MergingDigest m3 = returnNewDigest();
  for (int i = s2 ; i < s3 ; i++) {
  	m3.add(samples.get(i));
  }

  MergingDigest m4 = returnNewDigest();

  List<TDigest> l1 = new ArrayList<TDigest>();
  l1.add(m1);
  l1.add(m2);
  l1.add(m3);
  m4.add(l1);

  System.out.println("median with split (s1+s2+s3) "+m4.quantile(0.5));

  List<TDigest> l2 = new ArrayList<TDigest>();
  l2.add(m2);
  l2.add(m3);
  l2.add(m1);

  MergingDigest m5 = returnNewDigest();
  m5.add(l2);

  System.out.println("median with split (s2+s3+s1) "+m5.quantile(0.5));

}
@Test
public void test() throws Exception {
reader = new BufferedReader(new FileReader("src/test/resources/up.txt"));
readSamples();
System.out.println("Number of samples "+samples.size());
run_test();
run_test_split();
}
The output of the test
Number of samples 60398
Median without split 30.101212824980962
median with split (s1+s2+s3) 30.585493893059724
median with split (s2+s3+s1) 30.232216458636376
If I select CF = 1500 , I get the following results :
Number of samples 60398
Median without split 29.98999977111816
median with split (s1+s2+s3) 29.98999977111816
median with split (s2+s3+s1) 29.98999977111816
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tdunning/t-digest/issues/94, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPSejapbgOPf_zGXLW_7Mlv7f4os5exks5syNgmgaJpZM4QOy2M
.


Question - Is it possible to have window/door sensors?

Hi,
This is not an issue but a question. Couldn't find a place to ask a question so sorry to use this space for asking.
I'm new to HomeKit and been using zwave for some time and thanks to this great library, I'm now being able to use HomeKit to control switches, outlets, lights and door locks. However, my motion sensor and window/door sensors are not being exposed in HomeKit. Looking around, I see that Eve has a door/window sensor that are HomeKit compatible.
Looking at the list of accessories offered by the library, I cannot find one that fits into motion sensor or window/door sensors. Furthermore, looking at HomeKit Framework Reference I also do not see how those sensors can be supported by HomeKit. Am I missing something here? :) Is it possible to use HomeKit to see if my window/door is opened/closed or if my motion sensor detected movement? Should I extend the library to make this happen or is there an accessory that already offers this?
Thanks in advance,
Gaston

HomeKit does have a ContactSensor service type. This library doesn't have the definitions for it built-in yet, but you could always create one by implementing the Service interface directly. (You can find the UUID and required characteristics in Apple's HomeKit simulator)
If you do that, consider doing it within the library and submitting a PR. Otherwise, I do plan on adding the new service types when I get a chance.

Andy,
Thanks for the pointers. I think I'll have time this weekend to work on this. Thanks for the pointers. Once I have it running I'll create a PR.
Thanks again,
Gaston

Closed by #13


Feature: User configurable shortcuts

A new tab in Preferences where you can change/reset shortcuts like Keyboard Shortcuts in a table:

Command
Default Keybinding
User Keybinding
Reset

Close Tab
Ctrl+W

reset button

...
It can also act as documentation. The user configuration can be stored in a JSON file that can be imported/exported:
{
  {
       "command": "closeTab",
       "key": "ctrl+w",
       "key_user": ""
  }
}

I would love to migrate to Bruno, but not having shortcuts to switch recent requests and search for requests is block as I have pretty large collection.
Thx

Yes that feature would be super important! For example I am using keyboard shortcuts in all my apps to move between tabs. Unfortunately my Macbook keyboard does not have PageUp and PageDown so I can not set the currently fix shortcuts. A simple env file would be enough for me

I would love being able to change the current keybindings too, the Send Request actually requires to leave the hand from the mouse to press ENTER key, I would prefer to use a one-hand shortcut like CTRL+D or CTRL+SPACE.


SendGrid Subject and Body Inconsistent

Please provide a succinct description of the issue.
A function that is EventHub triggered and has SendGrid bindings added manually sometimes sends email with the default subject and body and other times with the programmatically set subject and body. The event hub has 4 partitions. The SendGrid API key is set via an App Config Setting.
Provide the steps required to reproduce the problem

Create an EventHub triggered function
Add SendGrid bindings to the function
Add an "app settings" variable containing the SendGrid Key (directly adding the key in Functions fails)
Add default Subject and Body (as required)
Add code to programmatically set subject and body in the function
Let the function execute messages from the EventHub

Expected behavior
Every message should be using the programmatically set subject and body
Actual behavior
Most messages get the default subject and body but some intermittently get the correct programmatically set ones.
Known workarounds
None known
Related information
Base template was C# EventHub triggered function. SendGrid code and bindings were added to that template.

Example Code:
#r "SendGrid"
using System;
using SendGrid.Helpers.Mail;
using Microsoft.Azure.WebJobs.Host;
public static void Run(string myEventHubMessage, out Mail message, TraceWriter log)
{
message = new Mail();
string [] parts = myEventHubMessage.Split(',');
if (parts.Length > 2)
{
if(parts[1] != "200")
{
log.Info($"The code : {parts[1]} and The reason : {parts[2]} ");
try
{
message.Subject = $"Azure API Management Security Event Trigger - Error {parts[1]}";
Content content = new Content
{
Type = "text/plain",
Value = $"Azure API Management Security Event Triggered - Error Code {parts[1]}, Error : {parts[2]}"
};
            message.AddContent(content);
        }
        catch
        (Exception ex)
        {
            log.Info(ex.Message);
        }
    }
    else
    {

    }
}

}

You have conditional logic in your function that only programatically sets the subject/content based on event properties. Are you sure it isn't the case that this conditional logic isn't the cause of the variability you're seeing? For an output message we only default the Mail properties if your code hasn't set them (code here).
@brettsam Any other ideas?

I think what @mathewc describes above is the likely cause. A good way to test this out would be to move the instantiation of your message to your try/catch block (initialize it with null) and perhaps log when your condition is not met. As in those cases no email would be sent at all.

I believe @mathewc is probably correct. I will check with my customer tomorrow and see if we can resolve this by either changing setting the message parameter to null, using an ICollection<> out param or just returning a value from the function instead.
Thanks for your help! I'll let you know how it goes.

The customer set the message to "null" by default and only set it as a new object right before setting properties to send. This has resolved their issue.
Thanks for all your help!


Make Databricks jobs output readable

Synopsis
As a developer
I want the (error) output from Databricks jobs to be readable
So that I can see what the problem is
Acceptance Criteria

[ ] Error output from Databricks is not showing ANSI escape codes

Tech Notes
Here is a site where you can quick and easy decode the output: https://maxschmitt.me/ansistrip/
Example:

According to Kasper (Halfspace): "[..] ved jeg at Databricks har rettet dette, og de håber at release til produktion inden jul." I anticipate we accept this and suggest that we close the story.


Overwriting get_url_parts makes page 404 under the new custom URL

Issue Summary
As far as I can find get_url_parts is supposed to return the full URL to the page. Overwriting this should, as far I understand it, make the page accessible via a custom URL rather than the tree-structure made through the Wagtail admin.
However although the Wagtail admin's "Live" button does link to the new custom URL, actually navigating to that page ends up in a 404.
Steps to Reproduce

Create a new page type
Overwrite get_url_parts as follows:

def get_url_parts(self, request=None):
    site_id, root_url, _ = super().get_url_parts(request=request)
    parent = self.get_parent().teampage
    return (site_id, root_url, "/test")

Create an instance of this new page via the Wagtail admin
Navigate to the site base URL with "/test" appended to it (for example by clicking the "Live" button in the Wagtail admin)

All the information I can find online suggests overwriting this method should be enough to make it available on a custom URL independent of the page tree structure of Wagtail but overwriting it just creates invalid URL's that 404.

I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: yes

Technical details

Python version: 3.12
Django version: 5.0.6
Wagtail version: 6.1.2
Browser version: Firefox 128.0

Overwriting this should, as far I understand it, make the page accessible via a custom URL rather than the tree-structure made through the Wagtail admin.

This isn't correct. Overriding get_url_parts will change the URL that a page reports as its own URL, but doesn't ensure that requests to that URL actually reach that page - for that, you need to override the routing logic, such as defining a route method on the site's root page.

Interesting, thanks for your swift answer. Reading that page shouldn't I be able to overwrite route() on the page then to accomplish this? In my (limited) testing it seems route() isn't called at all during/before matching the path to a page.
Also if overwriting get_url_parts() by itself just generates invalid URL's, shouldn't the docs mention somewhere that you should also overwrite some other function?

Reading that page shouldn't I be able to overwrite route() on the page then to accomplish this?

No - when a request comes in for a URL such as http://example.com/test/, it will start by looking up the root page of the site example.com, which will typically be of type HomePage or similar - you would need to override route on this class to give it special-case handling for the path /test/, rather than just following the default behaviour of looking for a child page with the slug test. Overriding route on the class where you defined get_url_parts won't work, because it will never reach a page of that type.

Also if overwriting get_url_parts() by itself just generates invalid URL's, shouldn't the docs mention somewhere that you should also overwrite some other function?

We'd be happy to consider a pull request to improve this part of the documentation. However, I suspect it's hard to give instructions that are broadly useful here, since there are many reasons that people might override get_url_parts, and a solution that works for a trivial case like return (site_id, root_url, "/test") may not be useful for real-world cases. Depending on how it's being used, the real solution might be to use RoutablePageMixin, or middleware to intercept the request, or a hard-coded URL path in urls.py.


Wysiwyg editor patch

Description
Implement discussed changes (with some executive decisions) to enable on demand editing of supported page types (Grid + Flex)
Backend

original PR did not work with node-red security (1/2 day figuring that out)

Needed to pick up the access token from the local storage

Changed call to deploy flows from V1 to V2 API

this helps with merge conflict resolution

Changed deploy mode from "full" to "nodes"

"full" edits trigger any nodes set to run on start. Also, with "full" mode, all nodes get destroyed and recreated that includes non-dashboard stuff (like PLC comms, MQTT comms, in-memory context etc) would all being taken down and rebuilt due to someone just tweaking a dashboard layout.

changed dashboard route to include api/v1/  path segment

this is to clarify intent (this is an API)

Added validation checks (server side)

return 400 + message if there was nothing to deploy (message is shown to user)

this was important since a bug or issue would make it appear flows have been deployed (but nothing was!)

check dashboard base id matches, returns 403 if not (plus message)
adds an ephemeral "edit key" that is verified upon applying edits, returns 403 if not (plus message)
Adds page and base IDs to the PATCH req and checks they match on the server, returns 403 if not (plus message)
Ensures every changed group still exists on server before posting flows

this is for integrity - before we do a deploy - ensures server flows have not changed while user edited dashboard

Ensures every changed groups page ID matches the original flow groups page ID

this is for integrity - before we do a deploy  - ensures server flows have not changed while user edited dashboard

Only perform POST of new flows IF changes are detected.

avoids interrupting other users if there is nothing actually changed.
although in theory this is handled at the client side, it is possible (due to string -vs- number differences to have differences that dont apply or make any change to the flow)

Added 2nd dashboard API endpoint for enabling edit mode

This is key to enabling edit mode and ONLY for one page.
If another user tries to edit a different page, the single edit-key will be changed and will invalidate/prevent the other edits. The aim is to avoid multiple edits

Front end (dashboard)

fixed tooltips (kinda*)

Could not get v-tooltip component with slots to work!!!
Changed the buttons to use v-tooltip attribute instead.

Implemented "Exit Edit Mode" functionality
Block changes being applied on every mouse move event

was getting really REALLY slow
was causing back pressure and browser lock ups / artifacts

Moved toolbar controls to own component
Added re-useable ConfirmDialog so that we could:

Prompt to Confirm losing changes when exiting edit mode (if unsaved/dirty)
Prompt to Confirm user is certain they want to deploy flows

avoid accidental deploys. clarifyies what is really happening (so no suprises when they see the "merge your changes" in node-red!

Feed back issues

like when server has changed or permissions are not met or there is network failure etc - user is informed that "save" did not work & thus get the ability to try again.

Moved lot of logic to the mixin to minimise repeated code on layout pages
Implemented necessary changes on the Flex and Grid vue components
Deleted the temporary WYSIWYG vue POC file

Front end (node-red)

added on-hover "Layout Editor" buttons

Removed text on buttons to make them all fit (used the original titles as tooltips)

This new button is conditioned to only appear on supported pages (Grid and Flex)

TODO
Toolbar issues

[x] Larger groups push the toolbar off bottom of screen. This makes it hard to distinguish edit mode is active. Also, not great UX having to scroll down to get toolbar.  should be change to position fixed
[x] No clear indication of saving / saved / save failure
* Added error dialog for non 200 response
* Added spinner to save button
* Disable toolbar while spinning
[x] Users page theme affects toolbar buttons visibility.

e.g. setting black background and light foreground makes the undo button effectively invisible
DONE - added color="secondary" and color="success" to undo and save. Also added filter: grayscale(1); to disabled buttons via CSS

Resize issues

[x] random issue - group collapsing to size 1 while resizing

resize my a few pixels, let go of mouse (i.e. don't resize) - it skips straight to size: 1

[x]  resize issue - cannot be dragged back to original size
[x] seizure inducing flip-flop when resizing and the position is between snap points
[ ]  resize skipping some snap points.

To recreate, start with a single size 6 group on a 12 grid. Drag it to size 1 and keep hold - now size up - it skips snap points 2,3,4 then starts working again.

[ ]  resize snap points are inconsistent

Currently, the group jumps ahead to the next snap point, other times it doesn't snap until you reach the snap point.
in short, it changes depending on how you operate and is quite disconcerting
the improvement could be to modulo the dx on stepX and snap at the halfway point - OR - calculate snap zones

snap zones should fix the flopflop issue

[ ]  Mouse cursor pointer gets changed to the halt pointer while resizing (should stay as the ew-resize pointer)

Ordering issues

[x]  Not obvious groups can be moved

DONE Set cursor to grab on :hover

[x] seizure inducing flip-flop when moving a group up over different sized group

DONE - don't snap into position until the moving component is within the area it would actually fit into. At this point, the groups get actually re-ordered but since we are holding the grid that just landed, it has the same index and so doesn't flip-flop

[ ]  Mouse cursor pointer gets changed to the halt pointer while moving (should be the "grabbing" pointer)

General issues

[ ]  when widgets associated with a group have a set size (non auto), they appear in error after refresh of node-red
[ ]  resizing sets the width to a number but flow source is a string. This means the undo "lights up" (is dirty) even when it is not.
[ ]  possible refactoring and de-duplication in grid+flex vue pages around edit controls events
* sortable.js not utilised (delete or implement)

For consideration:

set mouse pointer to grab open when on a place than can drag group - OR/AND - add a visible grabbable element
move the save/discard/cancel logic to mixin or EditControls to avoid duplicate code?
sometimes when user enters edit page it gets refreshed a few seconds after fully loaded

this interrupts the workflow and loses any sizing or moving user has done
This emanates from file:@vite-plugin-pwa/virtual:pwa-register/vue
It is related to workbox wb.addEventListener("activated"... and event.isUpdate is true causes window.location.reload(); to be called.

UX considerations

I really dislike that this posts to the FLOWS endpoint.
An alternative (that flexdash did) is to simply send edits to the editor & then let the user press deploy on Node-RED.
While editing, I would like an overlay on Node-RED that says "currently editing" and have a "cancel edits"
Add a pencil icon to the side bar entry when editing a page to make it clear page is being edited
Consider updating title of browser TAB to indicate edit mode.

Related Issue(s)

Checklist

[x] I have read the contribution guidelines
[ ] Suitable unit/system level tests have been added and they pass
[ ] Documentation has been updated

[ ] Upgrade instructions
[ ] Configuration details
[ ] Concepts

[ ] Changes flowforge.yml?

[ ] Issue/PR raised on FlowFuse/helm to update ConfigMap Template
[ ] Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production

Labels

[ ] Includes a DB migration? -> add the area:migration label

Discussed the following outstanding items on a call:

resize skipping some snap points.

resize snap points are inconsistent

Both okay for now

Mouse cursor pointer gets changed to the halt pointer while moving (should be the "grabbing" pointer)

This is a strange one, needs investigation before merging.

when widgets associated with a group have a set size (non auto), they appear in error after refresh of node-red

I see us having two options here:

We change the widget size validation, so that widths higher than the parent group are allowed, given that the UI handles there correctly anyway.
As part of the API deploy, we also modify any widths of widget's larger than the parenting group that was explicitly edited.

Add a pencil icon to the side bar entry when editing a page to make it clear page is being edited

I agree with this, how about a secondary-styled pill with "Edit Mode" text inside next to the page name? This won't be available when the side menu is collapsed, but it's better than nothing

Summarising Actions:
Cursor Icon

Mouse cursor pointer gets changed to the halt pointer while moving (should be the "grabbing" pointer)

Needs investigating
Widget Size Issue
Implement option 1 as stated in my comment above, disregard widget width > group width validation
Code Cleaning
Remove sortable.js
Action Icons
Clean the iconography & colours in the "Action" bar. @joepavitt to resolve this.

Merged into feature branch as discussed with @joepavitt
I'll take the action points and comments over to the original issue.


[ENT Support Request] ent77

Hi,
I think the link has changed because there is no longer a direct link to connect to https://ent.seine-et-marne.fr/
The link in ent.py which is https://ent77.seine-et-marne.fr/auth/login is redirected to https://ent.seine-et-marne.fr/
Thanks for you support.

Try this:
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0",
    'referer': 'https://ent.seine-et-marne.fr/',
}

def _open_ent_ng(
    username: str, password: str, url: str = "", **opts: str
) -> requests.cookies.RequestsCookieJar:
    """
    ENT which has an authentication like https://ent.iledefrance.fr/auth/login

    Parameters
    ----------
    username : str
        username
    password : str
        password
    url : str
        url of the ENT

    Returns
    -------
    cookies : cookies
        returns the ent session cookies
    """
    if not url:
        print("Missing url attribute")
        return

    print(f"[ENT {url}] Logging in with {username}")

    # ENT Connection
    with requests.Session() as session:
        payload = {"email": username, "password": password}
        r = session.post(url, headers=HEADERS, data=payload)

        if "login" in r.url:
            print(
                f"Fail to connect with Open NG {url} : probably wrong login information"
            )
            return

        return session.cookies

print(_open_ent_ng(url="https://ent77.seine-et-marne.fr/auth/login", username="oenvdhdkdv", password="odjvdjd"))

for information, the error I have with example.py
  File "g:\Mon Drive\Domotique\Pronote\pronotepy-master\pronotepy-master\example.py", line 5, in <module>
    client = pronotepy.Client('https://0772190m.index-education.net/pronote/eleve.html',
  File "g:\Mon Drive\Domotique\Pronote\pronotepy-master\pronotepy-master\pronotepy\clients.py", line 436, in __init__
    super().__init__(pronote_url, username, password, ent, mode, uuid)
  File "g:\Mon Drive\Domotique\Pronote\pronotepy-master\pronotepy-master\pronotepy\clients.py", line 96, in __init__
    self.attributes, self.func_options = self.communication.initialise()
  File "g:\Mon Drive\Domotique\Pronote\pronotepy-master\pronotepy-master\pronotepy\pronoteAPI.py", line 77, in initialise
    self.attributes = self._parse_html(get_response.content)
  File "g:\Mon Drive\Domotique\Pronote\pronotepy-master\pronotepy-master\pronotepy\pronoteAPI.py", line 264, in _parse_html
    raise PronoteAPIError(
pronotepy.exceptions.PronoteAPIError: Page html is different than expected. Be sure that pronote_url is the direct url to your pronote page.

with the config below:
import pronotepy
from pronotepy.ent import ent77

# creating the client and passing the occitanie_montpellier function to automatically get cookies from ENT
client = pronotepy.Client('https://0772190m.index-education.net/pronote/eleve.html',
                          username='*****',
                          password='*****',
                          ent=ent77)

yes it is because your ent changed their default login page from ent77 to just ent.
i just added the 'referer': 'https://ent.seine-et-marne.fr/' to the headers and it works.
In your client object you just need to use the function i provided above instead of the default ent77 from pronotepy.

Very thanks !!! was able to get my script for Home Assistant working.


Yatter2

@bhorinrb As the library management is now made with poetry, I think requirements.txt and setup can be removed. Additionally, coverage library needs to be installed with poetry for its workflow


Create README.md

Hope this works, still gathering resources on TC 39 meetings

Any final changes request before merging?


Add publish to up-streamer rust

Add support for tracking subscriptions and forwarding published messages in ustreamer.rs
Add usubscription-static-file to stub out subscriptions before usubscription is written
Add tests for mE (SOME/IP) subscribe/publish and uE (zenoh) subscribe/publish

NOTE: This PR also contains changes that @PLeVasseur has done for up-streamer-rust. His will be merged first and then I will rebase based on that merge.

#18

Closing this so I can open new one with clean commit structure.