Alternative to cloudformation

Our security policy prevents us from using cloudformation to deploy an elastic stack due to strict iam resourcing control.
We're currently using Terraform and would love to have an alternative to cloudformation to setup the elastic stack.

Thanks for the issue @KevinGrandon, Terraform is absolutely something we are considering. Will discuss this some more and update shortly.
Would it be possible to provide more detail on CloudFormation? Does "strict iam resourcing control" rule out ANY CloudFormation, or are there things about how we are using that are problematic? Would it be possible for us to address these security concerns and let you move forward with CloudFormation? Many folks have been wrapping it in a Terraform resource: https://www.terraform.io/docs/providers/aws/r/cloudformation_stack.html

We could for instance lock down the permissions to only work in the provided vpc / subnets:
https://aws.amazon.com/blogs/security/how-to-help-lock-down-a-users-amazon-ec2-capabilities-to-a-single-vpc/

@toolmantim did you have any other insight into this one?

@KevinGrandon how does Cloudformation differ to Terraform?

Do you mean in so far as it's security implications?
On Sun, 19 Feb 2017 at 11:04 pm, Adam Brady<EMAIL_ADDRESS>wrote:

@KevinGrandon https://github.com/KevinGrandon how does Cloudformation
differ to Terraform?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/buildkite/elastic-ci-stack-for-aws/issues/246#issuecomment-280914721,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA9jlHQ50KzsAWFjeVywqXCPpAUgfSeks5reC_agaJpZM4L9umd
.

I don't have any more insight I'm afraid!
I know @pvdvreede is ingesting this via Terraform. Have you had any luck?

@lox Yeah. You should just be able to create an IAM user with the necessary permissions and then execute CF as that user. I'm not seeing what else terraform would do, seeing as it would obviously use a subset of IAM

We are using Terraform as a wrapper around the Cloudformation. In Terraform we create a IAM policy and then pass that policy into the Cloudformation. However, the Cloudformation still creates the instance profile, role and then attaches this extra policy.
Would passing an IAM Role into the Cloudformation so that it only creates the instance profile, and connects it to the passed in Role help with your Security restrictions, @SomeoneWeird ?

@pvdvreede I think you mean @KevinGrandon there :)

I'd guess the issue here is the broad permissions that would be required for the stack in it's present form. It needs IAM creation for one, which is basically root access. We could look at locking this down, but I'd want to know if @KevinGrandon has other requirements beyond that.

Ahh, yes sorry I did mean @KevinGrandon !

I think that locking down IAM permissions would be a good step and allow many larger companies to consider using this stack in the first place. For us, we have a requirement to deploy this with Terraform so we're going to start investigating using that.

@KevinGrandon Just to be sure I understand what you mean by strict iam resourcing control: the roles/users you have access to are excluded from using the cloudformation service altogether? Or is it a resource created by the stack the blocks you?
If it's the former than I don't understand how Terraform will help you as it will be making calls into cloudformation - unless you dissect the stack into it's base components.

Yes, I don't think we're going to be able to use Cloudformation at all currently. It sounds like we're probably going to have to use base components for the time being. Has anyone successfully done that?

At one point I started trying to convert the stack to Terraform but it would be too much work to consume any updates so I decided to just consume the stack.

Hi, could someone done the setup with Terraform/Packer?
I would to try Buildkit service, but I don't want to introduce Cloudformation on my AWS account.
Any chance for this support or at least a complete tutorial of what is required for, @lox ?

@wbotelhos it would be a complete rewrite to exclude CloudFormation @wbotelhos, so not something we've looked at.
You can manage the CloudFormation resources with Terraform though!

Here's an example of how we manage Buildkite with Terraform:
resource "aws_cloudformation_stack" "buildkite_<QUEUE_NAME>" {
  name = "buildkite-<QUEUE_NAME>"

  parameters {
    KeyName                        = "XXXXXXXXXXX"
    BuildkiteAgentRelease          = "stable"
    BuildkiteAgentToken            = "XXXXXXXXXXXXXXX"
    BuildkiteAgentTags             = "os=linux"
    BuildkiteQueue                 = "<QUEUE_NAME>"
    AgentsPerInstance              = 4
    SecretsBucket                  = "XXXXXXXX"
    ArtifactsBucket                = ""
    BootstrapScriptUrl             = "s3://XXXXXXXXXXX/buildkite_boot.sh"
    VpcId                          = "XXXXXXXX"
    Subnets                        = "XXXXXXXXXX"
    AvailabilityZones              = ""
    InstanceType                   = "c5.2xlarge"
    SpotPrice                      = "0"
    MaxSize                        = 40
    MinSize                        = 0
    ScaleUpAdjustment              = 3
    ScaleDownAdjustment            = -1
    ScaleDownPeriod                = 1800
    RootVolumeSize                 = 250
    SecurityGroupId                = "sg-XXXXXXX"
    ImageId                        = ""                                                        # use default
    ManagedPolicyARN               = "arn:aws:iam::XXXXXXXXXXXX:policy/XXXXXXXXXXXX"
    ECRAccessPolicy                = "none"
    AssociatePublicIpAddress       = "false"
    EnableSecretsPlugin            = "true"
    EnableECRPlugin                = "false"
    EnableDockerLoginPlugin        = "true"
    EnableDockerUserNamespaceRemap = "true"
    EnableCostAllocationTags       = "true"
    CostAllocationTagName          = "X-Application"
    CostAllocationTagValue         = "buildkite"
  }

  template_url = "https://s3.amazonaws.com/buildkite-aws-stack/v${var.elastic_ci_stack_version}/aws-stack.json"
  capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"]
}

Thanks for your answer @lox .
Thanks for the template @tduffield :)

No worries! One day we'll get a pure Terraform alternative happening!

If anyone comes across this, it looks like template_url = "https://s3.amazonaws.com/buildkite-aws-stack/v${var.elastic_ci_stack_version}/aws-stack.json" doesn't quite work anymore. specifically, I latest is the only value I can find for v${var.elastic_ci_stack_version} that doesn't return access denied.

update on my previous comment: it appears part of my issue was using aws-stack.json rather than aws-stack.yml.

A reusable terraform module would be more convenient than CF so we can keep the entire IAM role, attached managed policies, and inline policies all in code.
At the moment, we run the bk cluster and attach policies to the role in terraform.

I'm going to take a crack at a native terraform module here - https://github.com/roverdotcom/terraform-aws-buildkite-elastic-ci-stack. I think a separate repo will work the best if it eventually wants to end up on the terraform public registry. I'll try to get a proof of concept up and running.
The cloudformation from this repo has worked really well for us so far, but we're also a 100% terraform shop, and it seems like there is interest in a native terraform solution. We're currently applying the cloudformation stack via terraform, but our interest in seeing mixed instance types in the ASG for spot (as mentioned in #651) has led us to thinking about a native terraform solution.
Hopefully I can have a proof of concept going in a week or so if all goes well. I'm planning on basing it off of the current stable v4.5.0 stack, only supporting the native lambda scaling. There may be some tricks to pinning the lambda and ami resources properly in a separate project, but I think I have some ways to get it working initially. I'll circle back when I have something if folks are interested.

@bshelton229 sounds good, we'll be interested to see how it goes!

only supporting the native lambda scaling

Our intention is to make this the primary scaling method in future releases, so this is a good choice.

I got a chance to get a proof of concept of a pure terraform module working in our test environment. We're going to go that direction for our shop for sure, and as it seems there is interest in a terraform module, I'm planning on publishing the module to an open source repo. I decided to base off of what's landing in master because there is some great stuff in there. Overall the AMI component built from this repo, and the autoscaling lambda published to s3, are easy to use to get a terraform stack working. The new autoscaler lambda is absolutely amazing, and we like how the SSM parameter holding the agent token is handled in master. I'll be starting on this next week and wanted to get some feedback if possible.
My main concern is staying up to date with the inputs to the cloudinit script used by the AMI built in this repo. If we start relying on this interface in external projects, it might be nice to document the env var inputs and maybe publish a changelog? There is one non-breaking issue with needing to pass BUILDKITE_STACK_NAME, which is required, and used for cfn-signal. Thankfully it exists OK right now, but if we start using this outside of cloudformation more formally, maybe we could check for that var being empty and not send cfn-signal?
I'm initially planning on creating a single terraform module that would act like the cloudformation stack, but would be composed of several modules that could also be used independently if desired. You would be able to either use the higher level module directly, getting a full stack experience, or you would be able to reach into the sub-components yourself and build your own stack. The component modules would likely be things like autoscaler-lambda, asg, iam-role, secrets-bucket, userdata. At the higher level, selectively using or not using entire sub-modules is finally much easier in terraform 0.13. I'd love feedback as to whether or not folks would want something that worked with terraform 0.12 as well. As soon as I have something that is more generally consumable I'll post back here.

@tduffield thanks a lot for sharing this.
We were going to wait on an official terraform module but we figured it might take a while for an official (or unofficial) one to get traction.
This is what we did

Create an internal module with every parameter as it's own input variable
Override the default values from the cf stack with our own values

if we don't have an override value then reuse the default from the stack

Take an input for the version with a default of v5.0.0 so it can be overridden with latest or master
Version the terraform module itself
Output all the stack outputs. The useful one is the IAM role name so additional inline and managed policies can be easily attached
Make sure disable_rollback=true or the imported stack will need to be deleted and recreated instead of updated

This solution works for us but allowing us to not have to copy around Makefiles or shell scripts around different repos and allows bk iam policies to be coupled with the queue.
The only issue I've noticed so far is that when importing into terraform, it doesn't copy the parameters correctly in the state so it will need to be applied (which results in a cloud formation update if disable rollback is true) before parameter differences can be seen. I created a ticket with the aws provider here https://github.com/hashicorp/terraform-provider-aws/issues/15992.
This should also have the way forward for terraforming build pipelines in the future.

@tduffield thanks a lot for sharing this.
We were going to wait on an official terraform module but we figured it might take a while for an official (or unofficial) one to get traction.
This is what we did

Create an internal module with every parameter as it's own input variable
Override the default values from the cf stack with our own values

if we don't have an override value then reuse the default from the stack

Take an input for the version with a default of v5.0.0 so it can be overridden with latest or master
Version the terraform module itself
Output all the stack outputs. The useful one is the InstanceRoleName so additional inline and managed policies can be easily attached.output "InstanceRoleName" {
  value = try(aws_cloudformation_stack.buildkite_queue.outputs.InstanceRoleName, "")
}

Make sure disable_rollback=true on the aws_cloudformation_stack or the imported stack will need to be deleted and recreated instead of updated.

This solution works for us by preventing us from copying around Makefiles or shell scripts to different repos and allows bk iam policies to be coupled with the queue. It gives us a single point of truth for sane defaults while allowing us to easily bump up versions. Only thing required would be bumping the version of the module reference source (if the HEAD of the branch is not used) and a subsequent reapply.
The only issue I've noticed so far is that when importing into terraform, it doesn't copy the parameters correctly in the state so it will need to be applied (which results in a cloud formation update if disable rollback is true) before parameter differences can be seen. I created a ticket with the aws provider here https://github.com/hashicorp/terraform-provider-aws/issues/15992 (please upvote if possible).
This should also have the way forward for terraforming build pipelines in the future.
provider "aws" {
  region = "us-west-2"
}

module "buildkite_queue" {
  # this can and should be pinned to a release tag using ?ref=tags/x.y.z
  source = "git@github.com:company/buildkite-elastic-ci.git//terraform?ref=main"

  # tag buildkite queue with team
  team = "my_team"
  # queue name
  BuildkiteQueue = "bananafanadingdong"
}

# Example: attach an inline-policy to the queue

## define the inline-policy

data "aws_iam_policy_document" "my_new_really_super_duper_cool_and_secure_iam_policy" {
  statement {
    effect    = "Allow"
    resources = ["*"]

    actions = [
      "ec2:DescribeTags"
    ]
  }
}

## attach the above inline-policy to the buildkite role from cloudformation

resource "aws_iam_role_policy" "test_policy" {
  name = "my_new_really_super_duper_cool_and_secure_iam_policy"
  role = module.buildkite_queue.InstanceRoleName

  policy = data.aws_iam_policy_document.my_new_really_super_duper_cool_and_secure_iam_policy.json
}

# Example: attach a managed IAM policy to the queue

resource "aws_iam_role_policy_attachment" "secrets_access" {
  role       = module.buildkite_queue.InstanceRoleName
  policy_arn = "arn:aws:iam::1234567890:policy/secrets-access"
}

A terraform stack isn’t on our roadmap currently, and supporting two independent AWS Elastic CI Stack implementations that offer the same functionality isn’t something we can reasonably commit to right now 😓
It may be that we end up with enough sufficiently compelling reasons to switch to terraform and do a re-write here, but we aren’t at that stage yet 😅 It may also be that ancillary efforts to component-ise the existing stack, like moving the Lambda scaler to the Serverless Application Repository, or potentially breaking out AMI building, may ease such a transition or make bootstrapping a community implementation more viable!
Speaking to the original issue of security policy preventing the use of CloudFormation, if still applicable, I would suggest looking into AWS Organisation Service Control Policies and IAM Permission Boundaries. These enable tight scoping of permissions and can limit the API actions CloudFormation can perform inside an account. These combined with using an isolated AWS Account may enable an acceptable use of CloudFormation.

@keithduncan I think that's a fair decision, but I have one proposal that might be a fairly small added surface area for you with a lot of benefit for the rest of us. I've been using the solution in https://github.com/buildkite/elastic-ci-stack-for-aws/issues/246#issuecomment-399426091 pretty successfully, but the annoying part is updating the parameters each time there's a new version and the stack's variables change. Would you consider publishing a terraform module (not a provider) that simply wraps the stack template and exposes each parameter as a module variable?

Would you consider publishing a terraform module (not a provider) that simply wraps the stack template and exposes each parameter as a module variable?

I’ve published a quick spike in this direction as a gist that will wrap a CloudFormation template with a terraform JSON template and expose module variables for each template Parameter.
This is not an official or supported project! 😅  I have only tested this with the latest version of the Elastic CI Stack for AWS template and successfully ran a terraform plan and apply for the generated module.
I hope this is useful for folks, if it is please do let us know.

@keithduncan that's basically what we are doing now. we have private modules that l wrap the cloud formation stack with the terraform resource. this works well but, like @ianwremmel said, if there's a new version with a new input, then we need to create a new input and update the module.
if there was an official terraform module maintained by buildkite within an official repo and updated upon each release (change of inputs), then we could stop having to maintain our private modules and instead point our code to the upstream official module.

Yea @nitrocode, I can absolutely see the value in that! I've added it to the backlog for us to tackle as the year kicks off.


Restored the ability to create subscriptions with plans

Create:

Update:

Before when creating one cannot select their plan.

Task linked: QF-3901 Read the plan field in the Subscriptions, make it read-only for editing

Thanks for fixing this 👍 - Important for support I guess.
Just fix the fields reported in the core test failure logs then feel free to merge some more suggested changes, see related PR...

Thanks for spotting, allowed myself to force push the two extra fields to keep the history cleaner. Will manage the merge myself later.


Add the ability to set width of the border of the candle and the wick

Is your feature request related to a problem? Please describe.
No.
Describe the solution you'd like
Would be great to allow to set the desired size of the border and the Wick on candles.
Additional context

I take it this is still unresolved? If pointed in the right direction, I'd be willing to contribute to this!

Thank you for your feature request for Lightweight Charts. In order to keep the library lightweight and maintain a narrow focus, we strive to include only strictly required features in the core library. As a result, we have decided that this requested feature should be implemented as a plugin instead, and will be closing this issue as a result.
We recently introduced the ability to write plugins for Lightweight Charts, allowing users like you to extend the functionality of the library without adding it to the core. We highly recommend implementing your feature request as a plugin to achieve the desired functionality while keeping the core bundle size smaller for all users.
To get started with plugins, please refer to our Plugins Documentation for a better understanding of what is possible and how plugins work. You can also explore our collection of plugin examples (with a preview hosted here) for inspiration and guidance on implementing specific functionality.
For your particular feature request, we suggest taking a look at the following examples (see above links):

Whisker Box Series (similar to a candlestick series)
Rounded Candle Series

To help you get started quickly, we have created an NPM package called create-lwc-plugin, which sets up a plugin project for you. This way, you can hit the ground running with your plugin development.
If you do create a plugin based on your feature request, we would appreciate it if you could publish it on GitHub or NPM and let us know. This way, we can share it with the rest of the community. It would be helpful if you could prefix your NPM package name with lwc-plugin- for easier discoverability by others.
Thank you for your understanding and support. We believe implementing your feature request as a plugin will greatly benefit both you and the Lightweight Charts community.


clr-icon shape property should be bindable

Select one ...
[ ] bug
[x] feature request
[x] enhancement

Expected behavior
<clr-icon [shape]=""> or <clr-icon   -shape="{{}}">
Actual behavior
dynamic shape cannot be used

Dynamic shapes can be used as follows:
<clr-icon [attr.shape]="shapeNameData">


Fix languages

When server-side rendering a component and then using the tool on the client-side, we should need to know which language is it.
A solution is to make a PR to OC itself https://github.com/opentable/oc and add the lang html attribute to the oc-container so that when doing client-side rendering we can attach the Accept-language string.

Other solution: allow (via specifying an option) oh-see to send the language baed on the pages' html's lang attribute


Make sure we normalize header types for test mock connection class

This pull request fixes the issue reported in https://issues.apache.org/jira/browse/LIBCLOUD-945.
We normalized header types inside the actual Connection class, but not inside the Connection class used by the tests. That's why some of the tests failed with newer version of requests which doesn't automatically cast header values to strings anymore.

Codecov Report

Merging #1111 into trunk will increase coverage by <.01%.
The diff coverage is 87.5%.

@@            Coverage Diff             @@
##            trunk    #1111      +/-   ##
==========================================
+ Coverage   85.33%   85.33%   +<.01%
==========================================
  Files         342      342
  Lines       65695    65709      +14
  Branches     5856     5855       -1
==========================================
+ Hits        56060    56072      +12
- Misses       7250     7252       +2
  Partials     2385     2385

Impacted Files
Coverage Δ

libcloud/http.py
89.31% <100%> (+0.16%)
:arrow_up:

libcloud/test/__init__.py
86.32% <100%> (+0.23%)
:arrow_up:

libcloud/storage/drivers/azure_blobs.py
66.97% <66.66%> (ø)
:arrow_up:

libcloud/test/storage/test_azure_blobs.py
91.63% <81.81%> (-0.25%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59b44a0...eafbc11. Read the comment docs.


Traffic plot improvement

Original issue by mateusccoelho on 2023-08-27 03:27:32+00:00

Initialize the traffic counts with yesterday is not right because it's misleading. If one doesn't know how the package works, he/she will think that the first day had no visitors.
If the computer running the Streamlit server is in a negative timezone, altair/javascript will parse dates wrong and "shift" dates by -1 day. This Stack Overflow thread explains the issue.

See plot below to visualize the two problems:

Notice the first day is 0 and that the counting started in 27th august, but there are no pageviews in this day.

@mateusccoelho  is this still relevant?

@mateusccoelho  I believe this is the current design that the first days metrics are not in the middle of the screen.
It starts from the previous day to make a relative comparison and to show the increase or decrease for the days afterwards
I do agree that this could be misleading.
This will be resolved once we introduce more granular timestamp data as requested in #31
Marking this issue as resolved - duplicate and planned work.
Please share insights and comments on #31


[ffi] Bool type

Should we add a Bool type that corresponds to the Dart bool?
How should it behave when passed to C?
/cc @sjindel-google @mraleph

Since C99, the C language has a bool type which we might want to support in dart:ffi.

/cc @mkustermann

Any updates in this?

There is any update on this? Have you any side-way to support c99 bool return types while we wait for this feature to be implemented?

Not yet. You can work around it with Int8 for now. Does that work for you?

Asking to google, I've found that it could be implemented as a uint8_t in some cases, but for me (gcc on linux) Int8 works like a charm! Thank you for your suggestion and the immediate response!
I hope you're all keeping well and safe

How should it behave when passed to C?

Maybe use Itanium C++ ABI spec? This should be correct for most targets.

If the base ABI specifies rules for the C99 type _Bool, then bool follows those rules. Otherwise, it has size and alignment 1.


Implemented transposed (de-) convolutions into Keras

Thanks for the PR @yaringal I only missed a unittest.
Also, the output_shape needs to be passed by hand, do you think we can infer that but just forcing the user to use batch_input_shape when a deconv layer is part of the model? In TF the shape can be recovered with input get_shape plus layer get_output_shape_for. I noticed some comments where you tried to catch output_shape while building the layer, that didn't work?

guys, there are lots of conflicts in this branch, I'm starting another one from scratch and making sure the vae deconv example works in tensorflow.
I'll leave the subsample -> stride modification out of this. Also, I'll keep the hand defined output_shape while we think of an automatic inference way to do this.
I'll make the PR as soon as the vae conv example finishes running (its much slower on TF, is it because of the transpose operations?)

To benchmark against TF, use the TF dim ordering.
On Jul 20, 2016 4:38 PM, "Eder Santana"<EMAIL_ADDRESS>wrote:

guys, there are lots of conflicts in this branch, I'm starting another one
from scratch and making sure the vae deconv example works in tensorflow.
I'll leave the subsample -> stride modification out of this. Also, I'll
keep the hand defined output_shape while we think of an automatic
inference way to do this.
I'll make the PR as soon as the vae conv example finishes running (its
much slower on TF, is it because of the transpose operations?)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/pull/3251#issuecomment-234116377, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AArWb0sRub0vovC2FsKbhXnlvuScDGnbks5qXrFZgaJpZM4JPAeJ
.

keras.backed also says deconv2d. Also, deconv is the most popular term, although not the most precise

This naming imbroglio is interesting. While this operation has been called "deconvolution"  in several papers and libraries, this is simply a convolution with fractional strides (i.e. interpolation is needed when applying the kernel). The "classical" meaning of deconvolution is to reverse the effect of a convolution which is not necessarily the case here. For convolutions with stride 1, the corresponding deconvolution would not need this deconv... This makes this naming quite miss-leading...

Apologies for not answering - I've been travelling the past few days with no internet access. How should we proceed? adapt code on this PR or on #3273?

@yaringal you can just get the code from the other PR and paste it here. This way you get proper credit for you contributions

Theano code seems to be working, but I can't debug the tensorflow code

@yaringal there is a None, defined in the test and TF deconv does not accept that. I think I know where it is... let me try to fix this. I'll make a PR to your PR

@yaringal I made you a PR,I think it should fix it

@yaringal sorry for being distracted, I "fixed" the wrong test last time. I made another corrections.
I can run the tests locally right now. If this doesn't work, I'll get back to this tomorrow

yay!!! we are good here @fchollet
let us know what do you think, specially regarding the output_shape inferecen

the failure looks weird, doesn't seem related to the PR, does it?

Build still seems to fail with error
The command "if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh; else wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh; fi" failed and exited with 8 during .
Your build has been stopped.

@EderSantana @yaringal I've tried to do an output_shape inference for this PR.
Any advice how can I make a PR with those changes into this PR? ;)

@lukovkin I think you can do a pull request at https://github.com/yaringal/keras/commits/master which I can merge into this PR

It seems the tests are failing due to an issue with Travis. I would expect it to go away; if it doesn't me may have to update our Travis config.

(Travis is fixed and the tests are passing)

@fchollet do you have any further suggestions for changes?

Yes, I added a series of comments on the code.

@yaringal I've created a PR in your fork.

@lukovkin thanks - let me know when it's working with theano and I'll merge (I tried to do something similar a few days ago but theano complained that conv cannot take symbolic variables)

@EderSantana are you working on the changes @fchollet suggested or shall I?


Point to next ODS version

Follow-up from https://github.com/opendevstack/ods-core/pull/212.
I anticipate another change to this which should make management of the version a bit easier, see https://github.com/opendevstack/ods-core/issues/225.

ping

@tjaeschke Thanks for reviewing.
I noticed now though that:

this will conflict a little with changes @kiwo will make, so in order not to disturb that work I'll not merge this now
It assumes a branch 2, which does not exist yet - not ideal. I guess on master you need to point to either production (as that will exist in BitBucket) or the last released branch (would be 1.2.x for now). I'll think a little more about this. Input welcome!

So for now I will not merge this, but. we'll have a similar PR around the release time for v2 where, at least for the released version, we definitely want to set all identifiers to that version.


Add Multi-client packing to the keras example.

This has to support model parallelism across clients. I tested this on the 8 client single-node mode, and it looked about right.

Thanks!


polkadot-v0.9.12

@shaunxw whenever you get the chance could you run ci

Sorry just noticed this PR, do we still need v0.9.12 support at this time?

No worries, the patch in our Cargo.toml workspace for acala makes it fine as no changes to the emulator code was needed. We probably should bump to the newest version.


Role / Group check

With the existing implementation of SAML authenticator is it possible to restrict access by roles / groups that are returned in the SAML response? If not any plans to add in the near future?

@itsnagaraj - I think this is a pretty large feature. It involves adding one or more new configurable fields on the base object, and changing the documentation, and gating the login based on one (or more) new XPaths. There are also going to be a lot of tests with this feature.
That being said, I think this is an important feature to add. I think it's obviously an oversight on my part to not have included this in the first version of the project, and I think that it's a good thing that people are calling out for a better user experience when using this software.
I think because of the reasons listed above, this should probably go into some sort of 0.1.0 version - this is probably not a bugfix-type operation, but it's important.
Since I'm not getting a lot of time to work on this project at work, this might be slow, but I'll definitely add this to the v0.1.0 milestone.

Thanks for your response. I am thinking along these lines
2 configurable fields required to allow for role check

one to specify the xpath which will used to read the roles the user belongs to
one to specify the allowed roles

`
xpath_role_location = Unicode(
default_value='//saml:Role/text()',
allow_none=True,
config=True,
help='''
This is an XPath that specifies where the user's allowed roles is located in the
SAML Assertion. This is to restrict users with certain roles granted by the administrator
to have access to jupyterhub:
    {
        'ds'   : 'http://www.w3.org/2000/09/xmldsig#',
        'md'   : 'urn:oasis:names:tc:SAML:2.0:metadata',
        'saml' : 'urn:oasis:names:tc:SAML:2.0:assertion',
        'samlp': 'urn:oasis:names:tc:SAML:2.0:protocol'
    }
    '''
)
allowed_roles = List(
    default_value=None,
    allow_none=True,
    config=True,
    help='''
    This values specifies what roles are allowed to have access to jupyterhub.
    '''
)

`
Rules for check:

Only perform the check if allowed_roles are specified
If allowed_roles are specified then read the roles form SAML response using the xpath
if roles is empty then fail authentication
if roles is not empty and does not have any of the specified allowed_roles then fail authentication

            if self.allowed_roles and self.allowed_roles.len() > 0:                 user_roles = self._get_roles_from_saml_doc(signed_xml, saml_doc_etree)                 if user_roles.len() == 0:                     return None                 if user_roles.len() > 0:                     role_check = any(elem in self.allowed_roles for elem in user_roles)                     if not role_check:                         return None

@distortedsignal I have added configurable role access but need advice on updating the test constants. It looks to be using responses from a cloud IAM provider. So changes can't be made that require the signature to be recalculated.
Any recommendations on the best way forward?

@0nebody if you want to open the PR, I'd be willing to work on getting the unit tests set up for you.

I think that after @0nebody's PR, this can be closed. I'll work on getting this out over the weekend.


Improve error with CI credentials that mislead user to think it's a binfmt error

Here is the log of the CI error:
Run ./.vscode/tasks.ps1 run run-torizon-binfmt
> Executing task: run-torizon-binfmt <
Container Runtime: docker
Run Arguments: --rm --privileged torizon/binfmt
Container Name: binfmt
Error response from daemon: No such container: binfmt
Container does not exist. Starting ...
Unable to find image 'torizon/binfmt:latest' locally
latest: Pulling from torizon/binfmt
8aabc323a91f: Pulling fs layer
26b478ce1ac7: Pulling fs layer
06f8d3fb0cee: Pulling fs layer
26b478ce1ac7: Verifying Checksum
26b478ce1ac7: Download complete
8aabc323a91f: Verifying Checksum
8aabc323a91f: Download complete
8aabc323a91f: Pull complete
26b478ce1ac7: Pull complete
06f8d3fb0cee: Pull complete
Digest: sha256:4341763f8efc917a9ca02dfb9f6f0fdf53116a6358dcab0b363a03c6a0f69788
Status: Downloaded newer image for torizon/binfmt:latest
Response status code does not indicate success: 401 (Unauthorized).
	at _getJonOsterToken, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.conf/torizonIO.ps1: line 55
	at <ScriptBlock>, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.conf/torizonIO.ps1: line 68
	at checkTCBInputs, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.vscode/tasks.ps1: line 264
	at _parseEnvs, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.vscode/tasks.ps1: line 475
	at runTask, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.vscode/tasks.ps1: line 550
	at taskArgumentExecute, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.vscode/tasks.ps1: line 448
	at <ScriptBlock>, /home/runner/work/videoTFLiteTorizon/videoTFLiteTorizon/.vscode/tasks.ps1: line 672
	at <ScriptBlock>, /home/runner/work/_temp/1b745540-d8b6-4c98-8821-f622e7af2e00.ps1: line 3
	at <ScriptBlock>, <No file>: line 1
Error: Process completed with exit code 1.

It looks like something went wrong with binfmt, but it was just wrong credentials.

Thanks for the report @leograba
Was added the call of docker-login before  all the container operations, so it will fail fast in case of wrong credentials.


custom tiles is overlapping

I am using this script for adding custom tile but it is not working , it is overlapping with google maps tiles .
<View>
          <MapView
          style={styles.container}
            initialRegion={{
              latitude: 27.78825,
              longitude: 77.4324,
              latitudeDelta: 0.0922,
              longitudeDelta: 0.0421,
            }}
          >
          <MapView.UrlTile
            urlTemplate="https://mt1.mapmyindia.com/advancedmaps/v1/key/still_map/{z}/{x}/{y}.png"
            zIndex={-1}
          />
          </MapView>
      </View>

I got the same problem on Android!

Merging to #1880

hi i am using below url of soil map tile
<MapView.UrlTile
urlTemplate=" ....../api/soil/wms/?x=5458&y=12859&z=15 " />
in x,y i pass converted lat long and in z i pass zoom level.
But my map overlay tile repeat, why i don't know?
Please help


canot open your code file

could be damaged, can you help

I've tried to update the file, please check again.
If not working, let me know and I'll try to create a ticket. Meanwhile, if you send an email address, I can send the code to you.

still the same error could not be<EMAIL_ADDRESS>Thanks a lot, appreciate

I found that if you click on "< >" button or "Raw" button after having the error, you can see the code.

I've changed the extension to .py (python) instead of .ipynb (jupyter notebook) and now the file is readable.


Font loader not working

If I embed the Font Loader into my BaseLayout.astro…
import AstroFont from<EMAIL_ADDRESS>
…I get the following error:

Cannot find module<EMAIL_ADDRESS>or its corresponding type declarations.

If I change the import to its path in node_modules…
import AstroFont from<EMAIL_ADDRESS>
…I get the following error:

Missing "./components/AstroFont.astro" specifier in "@gamesome/astro-font" package

The first one should be correct. Can you add a minimal reproduction example?

@jsve No, but my project is live at https://hawk-gt1191.de, the repository is open source at https://github.com/macx/hawk-gt1191.
I added the needed code lines for astro-font but commented it out (and added regulary fontsource-components to work. You should be clone and run this easily (with yarn).

I can't reproduce the issue in your repository. However, I was unable to yarn install, with the following error:
warning Pattern ["wrap-ansi-cjs<EMAIL_ADDRESS>is trying to unpack in the same destination "/home/.cache/yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern<EMAIL_ADDRESS>This could result in non-deterministic behavior, skipping.
error An unexpected error occurred: "Failed to fetch dependencies:
-<EMAIL_ADDRESS>
This seems to be comming from @astrojs/check. After removing that, I was able to install and run the project. Are you sure @gamesome/astro-font was properly installed?
Please set up a minimal reproduction if you need further assistance!

@jsve Sure, here is a blank blog template: https://github.com/macx/astro-font-test
Please run yarn build (which includes the check and has to be installed). It fails:

src/components/BaseHead.astro:2:23 - error ts(2307): Cannot find module<EMAIL_ADDRESS>or its corresponding type declarations.
2 import AstroFont from<EMAIL_ADDRESS>

I can see no issues in that repository either. After cloning and yarn build (using astro check) here is the output:
11:50:22 Types generated 5.32s
11:50:22 [build] output: "static"
11:50:22 [build] directory: /home/projects/gqainkapj.github/dist/
11:50:22 [build] Collecting build info...
11:50:22 [build] ✓ Completed in 5.33s.
11:50:22 [build] Building static entrypoints...
11:50:26 [vite] ✓ built in 3.18s
11:50:26 [build] ✓ Completed in 3.35s.

 generating static routes
11:50:26 ▶ src/pages/about.astro
11:50:26   └─ /about/index.html (+40ms)
11:50:26 ▶ src/pages/blog/index.astro
11:50:26   └─ /blog/index.html (+12ms)
11:50:26 ▶ src/pages/blog/[...slug].astro
11:50:26   ├─ /blog/markdown-style-guide/index.html (+13ms)
11:50:26   ├─ /blog/using-mdx/index.html (+22ms)
11:50:26   ├─ /blog/first-post/index.html (+5ms)
11:50:26   ├─ /blog/second-post/index.html (+5ms)
11:50:26   └─ /blog/third-post/index.html (+6ms)
11:50:26 λ src/pages/rss.xml.js
11:50:26   └─ /rss.xml (+2ms)
11:50:26 ▶ src/pages/index.astro
11:50:26   └─ /index.html (+4ms)
11:50:26 ✓ Completed in 173ms.

11:50:26 [@astrojs/sitemap] `sitemap-index.xml` created at `dist`
11:50:26 [build] 8 page(s) built in 8.86s
11:50:26 [build] Complete!

The only thing that sticks out is that you have moduleResolution set to "Node" in your tsconfig. Also, which version of node and yarn are you using?

Yep, @jsve, my tsconfig was misconfigurated. Removing "moduleResolution": "Node" did the trick.
Thanks for your investigation!


Crypto & Investment Research - Domain Knowledge Development

Oh, finally someone I know. Yes, I've been watching :)


Inconsistent use of 'Census Place'

FC1 shows Worksop as a Census Place in 1841 (results), when it should show Bassetlaw, if Census Place = District.
I don't see the issue (yet) so while I remember what I was doing and have the screen shots:
Worksop, Notts, is down on both as not being a census place for 1841 (although we have all the others).  So in an attempt to find out which Census Place it was in, in 1841, I searched FreeCEN1 for Smith, with Worksop as the Civil Parish

Then repeated this on FreeCEN2 (using Worksop as the Place)

The reason is because in 1841 Worksop was in the Bassetlaw Parish
If you look at the Stats page at you will see how the NA organised them as
https://freecen1.freecen.org.uk/coverage-NTT.html#1841
By selecting Bassetlaw then you would get 55 results in FC2.
FC1 lets you select 1841 Census but also lets you select Worksop as the
Census Place by typing in the name WORKSOP  and not selecting it from a
dropdown list.
As the FC2 census place list, also has the Census years alongside it, and
1841 is not listed Example it says (Worksop 1851/1861/1871/1891) FC2 does
not then search 1841 for Worksop.
Do we really need all the census years alongside each Census place in the
dropdown lists?
I know FC1 also has them as well, but  they also have the alternative that
lets you type in Worksop as I previously mentioned.
Well that’s my thinking of it anyway.
On Thu, 6 Feb 2020 at 12:50, PatReynolds<EMAIL_ADDRESS>wrote:

I don't see the issue (yet) so while I remember what I was doing and have
the screen shots:
Worksop, Notts, is down on both as not being a census place for 1841
(although we have all the others). So in an attempt to find out which
Census Place it was in, in 1841, I searched FreeCEN1 for Smith, with
Worksop as the Civil Parish
[image: image.png]
https://camo.githubusercontent.com/ed87b334de8b1465caa082b6aff1f66887cd3233/68747470733a2f2f696d616765732e7a656e68756275736572636f6e74656e742e636f6d2f3562363935613564616333613230303135356337363565332f39363932613664302d323864362d346263332d616263322d373439646439376533623637
Then repeated this on FreeCEN2 (using Worksop as the Place)
[image: image.png]
https://camo.githubusercontent.com/e532b996a29fa90cb998e1a7b0aad349bd988a13/68747470733a2f2f696d616765732e7a656e68756275736572636f6e74656e742e636f6d2f3562363935613564616333613230303135356337363565332f39303133326263312d316437612d346331342d393334392d343338373839646562393338
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/805?email_source=notifications&email_token=ADIL3VRBPF6D4VCFHQSW4L3RBQBSVA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4ILQCEMA,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADIL3VSA4RB3HLPDGZOOHV3RBQBSVANCNFSM4KQ4KVVA
.

Hi Brenda,
The FreeCEN1 results show Worksop, not Bassetlaw, as the Census place (I expected to see Bassettlaw, but that's not what FC1 is showing me.
Is 'Census Place' on FC1 the sub-district, and on FC2 the district?

Worksop in the 1841 census is under Bassetlaw if you look at the Stats page.
I only gave you the link to FC1 Statistics page to show you what the PARMS
has.
FC2 Stats page says the same as they are both derived from the PARMS file.
I just happen to have that link open on my iPad from FC1.
Shows the same herewith
https://www.freecen.org.uk/freecen_coverage/NTT
Not sure what you mean by the last bit  anyway
On Thu, 6 Feb 2020 at 16:42, PatReynolds<EMAIL_ADDRESS>wrote:

Hi Brenda,
The FreeCEN1 results show Worksop, not Bassetlaw, as the Census place (I
expected to see Bassettlaw, but that's not what FC1 is showing me.
Is 'Census Place' on FC1 the sub-district, and on FC2 the district?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/805?email_source=notifications&email_token=ADIL3VWH2BTZC2OZ2EKAGPDRBQ4XJA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK74ZMI#issuecomment-582995121,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADIL3VRAVCWE3HBTPT63K23RBQ4XJANCNFSM4KQ4KVVA
.

We are using 'census place' to mean two completely different, but associated, pieces of data: e.g. a search "Robinson" for 'Census place = Bassettlaw' in 1841 gives this result:

.
when we get down to detail

Conclusions
a) FreeCEN1 is inconsistent in its use of the term 'Census place' and documentation brought over from FC1 needs to be checked with this in mind.
b) FreeCEN1 provides more useful information on the search results than FreeCEN2 does - changing the 'Place' field for the 'Civil Parish' field on search results would redress this, if wanted.
c)  I may be related to an 85 year female wheelwright!!! I've reported this as a possible typo in the gender) - irrelevant to this issue but I'd love it to be true.

@PatReynolds
I hate to comment but we should not use the term place for any of the fields in CEN
The Census Place field that is used for searching in CEN2 is actually the Census District. That is taken from the piece field called district name.  We should use this term Census District instead of Census Place or Place. Specifically in the Search Form, The search Results and the Details.
Now there is a field on the census documents much closer to what people would refer to as a place and that is the "City, Borough, Town or County Corporate or Municipal Boundary or Island" information. This field is NOT presently transcribed. However @geoffj-FUG is suggesting its inclusion in future transcriptions for CEN2

Census Place was my term
There are a group of information boxes at the top of census sheets and only one is ever completed. I was therefore looking for a term that covered them all.
They can be Island, Borough, town, village, hamlet etc.
In the past it has been up to the transcriber to capture this information (or not) by entering it in the address. Hamlets for instance didn’t necessarily have addresses. They certainly were not the same as the Civil Parish.
I was looking for a way to formally capture the information in just one field. I used ‘Census Place’ as a term to cover them all. I have no objection to a different caption.
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Monday, April 20, 2020 6:18 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Inconsistent use of 'Census Place' (#805)
@PatReynolds https://github.com/PatReynolds
I hate to comment but we should not use the term place for any of the fields in CEN
The Census Place field that is used for searching in CEN2 is actually the Census District. That is taken from the piece field called district name. We should use this term Census District instead of Census Place or Place. Specifically in the Search Form, The search Results and the Details.
Now there is a field on the census documents much closer to what people would refer to as a place and that is the "City, Borough, Town or County Corporate or Municipal Boundary or Island" information. This field is NOT presently transcribed. However @geoffj-FUG https://github.com/geoffj-FUG  is suggesting its inclusion in future transcriptions for CEN2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-616218429 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFKCHBWC7HPEGGTSIBTRNNL6NANCNFSM4KQ4KVVA .  https://github.com/notifications/beacon/AKCPIFO2ZHXJRCAQGSCYKU3RNNL6NA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOES5L6PI.gif

How about Location or Census Location?
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Monday, April 20, 2020 6:18 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Inconsistent use of 'Census Place' (#805)
@PatReynolds https://github.com/PatReynolds
I hate to comment but we should not use the term place for any of the fields in CEN
The Census Place field that is used for searching in CEN2 is actually the Census District. That is taken from the piece field called district name. We should use this term Census District instead of Census Place or Place. Specifically in the Search Form, The search Results and the Details.
Now there is a field on the census documents much closer to what people would refer to as a place and that is the "City, Borough, Town or County Corporate or Municipal Boundary or Island" information. This field is NOT presently transcribed. However @geoffj-FUG https://github.com/geoffj-FUG  is suggesting its inclusion in future transcriptions for CEN2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-616218429 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFKCHBWC7HPEGGTSIBTRNNL6NANCNFSM4KQ4KVVA .  https://github.com/notifications/beacon/AKCPIFO2ZHXJRCAQGSCYKU3RNNL6NA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOES5L6PI.gif

@geoffj-FUG this story and my comment were not focussed on your proposed usage of that term. They were focussed on its use on the search page and the results.
WRT to the your use in the spreadsheet census region might be better.
To me a location implies grid coordinates

I don't think we should use the term 'location' either.  It needs to be
more specific.  Unless we are saying something like 'always use twys when
transcribing places mentioned in the census'.  I'm trying to come up with a
standard term list for each level of the hierarchy, using the original/TNA
term where possible.
On Mon, 20 Apr 2020 at 00:38, geoffj-FUG<EMAIL_ADDRESS>wrote:

How about Location or Census Location?
Geoff
From: Kirk Dawson<EMAIL_ADDRESS>Sent: Monday, April 20, 2020 6:18 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention <
<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Inconsistent use of 'Census
Place' (#805)
@PatReynolds https://github.com/PatReynolds
I hate to comment but we should not use the term place for any of the
fields in CEN
The Census Place field that is used for searching in CEN2 is actually the
Census District. That is taken from the piece field called district name.
We should use this term Census District instead of Census Place or Place.
Specifically in the Search Form, The search Results and the Details.
Now there is a field on the census documents much closer to what people
would refer to as a place and that is the "City, Borough, Town or County
Corporate or Municipal Boundary or Island" information. This field is NOT
presently transcribed. However @geoffj-FUG https://github.com/geoffj-FUG
is suggesting its inclusion in future transcriptions for CEN2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <
https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-616218429>
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AKCPIFKCHBWC7HPEGGTSIBTRNNL6NANCNFSM4KQ4KVVA>
. <
https://github.com/notifications/beacon/AKCPIFO2ZHXJRCAQGSCYKU3RNNL6NA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOES5L6PI.gif>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-616244613,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADK5GHJNJPX7FX2V6NVO673RNODODANCNFSM4KQ4KVVA
.

--
- -
Dr Pat Reynolds
Executive Director
A Charitable Incorporated Organisation registered in England and Wales,
number 1167484
VAT registration: 233 0105 70
Join us on social:
[image: : https://www.facebook.com/FreeUKGEN]
https://www.facebook.com/FreeUKGEN https://twitter.com/FreeUKGen
https://www.instagram.com/freeukgenealogy/
https://www.pinterest.co.uk/FreeUKGenealogy/
Please read our Privacy Notice
https://www.freeukgenealogy.org.uk/files/Documents/Privacy-Notice.pdf for
information on how we will protect and use your data.
https://www.easyfundraising.org.uk/causes/freeukgenealogy/?utm_campaign=raise-more
+44 1723 362616   +44 7943 145387
Westwood House,Westwood, Scarborough YO11 2JD, UK

Helena
I have plumped for Where Census Taken for the caption. There is a hotchpotch of boxes on the census forms.
I have come up with the following instructions:
“The place where the census was taken can be the name of the Island, Borough or Burgh, Municipal Borough, City, Town, Village, Hamlet or Tything. There are various options for the enumerator to record this information but only one was used for each place that the census was taken, depending on the circumstances. In 1841 it may have been recorded as a County Corporate.
Islands, Boroughs or Burghs are usually found on the cover sheet of an ED. Towns, Villages, Hamlets and Tythings are usually found at the top of each census page.
Recording the information in this field eliminates the need to capture it in the address, unless it has also been entered in the address column by the enumerator.
You only need to enter this information once at the beginning of the ED unless it changes.”
Regards
Geoff
From: Helena Hyde<EMAIL_ADDRESS>Sent: Thursday, April 23, 2020 9:34 PM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: geoffj-FUG<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Inconsistent use of 'Census Place' (#805)
I don't think we should use the term 'location' either. It needs to be
more specific. Unless we are saying something like 'always use twys when
transcribing places mentioned in the census'. I'm trying to come up with a
standard term list for each level of the hierarchy, using the original/TNA
term where possible.
On Mon, 20 Apr 2020 at 00:38, geoffj-FUG<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

How about Location or Census Location?
Geoff
From: Kirk Dawson<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Sent: Monday, April 20, 2020 6:18 AM
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Cc: geoffj-FUG<EMAIL_ADDRESS><EMAIL_ADDRESS>>; Mention <
<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [FreeUKGen/FreeCENMigration] Inconsistent use of 'Census
Place' (#805)
@PatReynolds https://github.com/PatReynolds
I hate to comment but we should not use the term place for any of the
fields in CEN
The Census Place field that is used for searching in CEN2 is actually the
Census District. That is taken from the piece field called district name.
We should use this term Census District instead of Census Place or Place.
Specifically in the Search Form, The search Results and the Details.
Now there is a field on the census documents much closer to what people
would refer to as a place and that is the "City, Borough, Town or County
Corporate or Municipal Boundary or Island" information. This field is NOT
presently transcribed. However @geoffj-FUG https://github.com/geoffj-FUG
is suggesting its inclusion in future transcriptions for CEN2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <
https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-616218429>
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AKCPIFKCHBWC7HPEGGTSIBTRNNL6NANCNFSM4KQ4KVVA>
. <
https://github.com/notifications/beacon/AKCPIFO2ZHXJRCAQGSCYKU3RNNL6NA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOES5L6PI.gif>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-616244613,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADK5GHJNJPX7FX2V6NVO673RNODODANCNFSM4KQ4KVVA
.

--
- -
Dr Pat Reynolds
Executive Director
A Charitable Incorporated Organisation registered in England and Wales,
number 1167484
VAT registration: 233 0105 70
Join us on social:
[image: : https://www.facebook.com/FreeUKGEN]
https://www.facebook.com/FreeUKGEN https://twitter.com/FreeUKGen
https://www.instagram.com/freeukgenealogy/
https://www.pinterest.co.uk/FreeUKGenealogy/
Please read our Privacy Notice
https://www.freeukgenealogy.org.uk/files/Documents/Privacy-Notice.pdf for
information on how we will protect and use your data.
https://www.easyfundraising.org.uk/causes/freeukgenealogy/?utm_campaign=raise-more
+44 1723 362616  +44 7943 145387
Westwood House,Westwood, Scarborough YO11 2JD, UK
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-618350156 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFM2HVHWZH7EPQE52YDROARTZANCNFSM4KQ4KVVA .  https://github.com/notifications/beacon/AKCPIFO6NWQOQQOB7LRFR5LROARTZA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOETNUMTA.gif

I like 'Where Census taken'
It may be helpful to use the place heirarchies described in https://www.visionofbritain.org.uk/expertsearch#tab04 in discussions to clarify what we are talking about.
So ... to rephrase my original point and add more detail
FC1 search (correctly) lists Worksop as a high-level district in all censuses except for 1841:

Testing proved dificult, as I couldn't remember which record I'd found, and when I tried to find a specific record John Smith, living in Worksop in 1841 - using: https://www.familysearch.org/ark:/61903/1:1:M7S3-3C9 this record was not found.  These records are found:

Therefore I am assuming that if the records for Worksop (the third level district aka Civil Parish) were to be transcribed, then they would appear here as the 'Census place'.
In FreeCEN2, similarly, the search is on the first level district:
 and so is the results page:

But at detailed results, the second and third level districts are shown, as on the details page of FreeCEN1.

For decision:
Do we replace the first level district on the search results page with the third level district (personally, I think this latter is more useful).
Do we explore, in the longer term, using the TNA data on the second level districts where they have this (aka the EDs) to present place names at this level in the search results.
E.g. https://discovery.nationalarchives.gov.uk/details/r/C3762855 shows that HO 107/401 level four district (the sub-district, which we call the ED) number 1 is called Worksop and number 2 is called Carlton.   I believe this information is also available on the initial images of the register, and could be transcribed.

I would welcome discussin/decision on my previous comment.

@PatReynolds You asked for discussion/decision on your comment?
We are currently using 'Where Census Taken' in the transcriber input field and in the display of that field.
The usage of appropriate terminology on the search page and its results remains open.
I think that we need to make a decision on the basis of census records for 1851 -1921 and fit 1841 into those. Why? 1841 is a unique and somewhat problematic structure.
I am somewhat hampered as my knowledge is based on the actual TNA  structures. The  extent to which Scotland does/does not follow that structure is unclear to me as I have never actually seen what their archives actually record. We know Ireland is something of an issue. Bottom line there fore is my comments are England/Wales centric.
More comments to follow.

I agree that it needs to be 1851-1821, with 1841 fitted in.

'Where census taken' as a descriptor works for me.
I now have a better grasp of the levels involved in the Census for location information
To give the example of North Scarle, the following hierachy of information is known (also some others, but these are the ones with names and I don't think people will want to search by numbers - if they do, then that's an enhancement):
Level/Example
[0 Year]
1 England
2  Lincolnshire
3 Newark
4 North Collingham
5 [Ed number and suffix]
6 [Ward]
7 North Scarle.
In the search results, FreeCEN1 shows 7 as the 'Census Place'
In the search results, FreeCN2 shows 4 as the 'Census Place'
I think that showing level 7 as 'Where Census Taken' is the most helpful.

To note (for no action at this point) In searching:
FreeBMD allows searching by their equivalents of 2 and 3 (all the data they have)  FreeREG allows searching by 2 and 7 (all the data they have).
FreeCEN1 allows searching by 2, or 4, or a free text search for 7
FreeCE2  allows searching by 2 and then by , but allocates each 4 to a level 2 which may be 'wrong' as a level 3 can cross county boundaries..

Hello Geoff, cdo you have a handy table showing the heirarchy of place names used in each census, including non heirarchical realationships (civil parish and church parish) which ones were recorded in each census (whether or not they were transcribed by FreeCEN)?
If not, don't worry, I can compile one.

Pat
That sounds like the PARMS.
It can be got by county/year by using
Manage County … choose county … County statistics.
That is set out in the structure of the PARMS.
CAP will show it for the whole county once it is done. Not too far away.
What do you want it for?
Geoff
From: Pat Reynolds @.>
Sent: Wednesday, April 10, 2024 8:24 PM
To: FreeUKGen/FreeCENMigration @.>
Cc: Geoff J @.>; Mention @.>
Subject: Re: [FreeUKGen/FreeCENMigration] Inconsistent use of 'Census Place' (#805)
Hello Geoff, cdo you have a handy table showing the heirarchy of place names used in each census, including non heirarchical realationships (civil parish and church parish) which ones were recorded in each census (whether or not they were transcribed by FreeCEN)?
If not, don't worry, I can compile one.
—
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/805#issuecomment-2047151440 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFJTZ2BCRA3CS7TF4I3Y4UHNBAVCNFSM4KQ4KVVKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUG4YTKMJUGQYA .
You are receiving this because you were mentioned.  https://github.com/notifications/beacon/AKCPIFLWQSB6R3DOKNYBEE3Y4UHNBA5CNFSM4KQ4KVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOPICQ2UA.gif Message ID: @.*** @.***> >

I think this is probably best discussed in a FreeCEN committee meeting once I've returned.  Or the geolocation work may have resolved the issue.  I will take a look again in December 2024.


lib: add the tmpDir parameter (again)

First, this revert a patch so that docker2aci compiles against appc/spec v0.5.2 rather than just git-master. It will be useful for rkt, see https://github.com/coreos/rkt/pull/929#issuecomment-104787060.
Then, add the tmpDir parameter to fix #50. It was partially done in #51 but not completely.

This breaks the build but we need a ref with the tmpdir parameter working and working against appc/spec v0.5.2. Please add a revert of Revert "common: rename App to ImageName" so the build doesn't break but you still can vendor the intermediary commit in rkt.
After that, LGTM.

I like reverting reverts. LGTM.


Path is Not Accurate (Much like displacement between two points)

Everything is working fine but path is showing like displacement route. It should looks like the path along with roads but is overlapping the roads and showing displacement.

Tested, it's working fine. I have tested with different scenarios. You were expecting the path through the motorway but unfortunately, Google doesn't provide any clue for the tolls or etc. closing the ticket, please feel free to reach me out through email. thanks!


althttpd: unstable-2022-08-12 -> unstable-2023-08-12

Description of changes
Fossil doesn't have an autoupdater, but so few packages use Fossil.
Tested by cat hello > index.html && nix run .#althttpd
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 260796 run on aarch64-darwin 1

1 package built:

althttpd


Made texture flow python test less strict to fix it on AARCH64

What does this PR change?
Texture flow python binding test fails on AARCH64 due to float result rounding issue in Sobel evaluation so test is made less strict

:+1:

Test fails on uniform part of the image there is no own gradient while small gradient magnitude produced by blurring yield unstable eigen vector estimation sensitive to float significand last bit differences caused by rounding


Yiran li/feature/edit unavailability test

Describe your changes
Write five test cases for edit function.

successfully edit  2. can not find the event which need to be edited  3. the updated information is wrong(the start time is later than the end time)  4. the updated information is meaningless(the start time is early than now)  5. the updated information is meaningless(the end time is early than now)

Issue ticket number and link
https://fireapp-emergiq-2024.atlassian.net/browse/FE-190

@BenMMcLean, could you please review this when you are free? It would be nice if we could get this done soon, as the front-end team needs the feature improvements from this branch.


[Mention The Feature]: Working on navbar and Home page navigation for projects, about us, contact, community support page

Is your feature request related to a problem? Please describe.

Navbar on home page does'nt look good, and text is also hidden.-> no proper styling of navbar

No option to navigate back to home page on other pages (home page navigation is only available for courses page)

Describe the solution you'd like.

Proper styling for navbar so that all the content is clearly visible.
adding home page navigation on other pages for better user experience and easy navigation on different pages.

Describe alternatives you've considered.

Proper styling for navbar so that all the content is clearly visible.
adding home page navigation on other pages for better user experience and easy navigation on different pages.

Additional context.
Attached screenshots in feature requests.
Show us the magic with screenshots

for navbar

Home page navigation

similarly for other pages, navigation to home page is made available.

Checklist

[X] I have checked the Existing Issues
[X] I have read the Contributing Guidelines
[X] I want to work on this issue. (optional)

I have seen many navbar issues, and i have mentioned it before for contributors to not create issues regarding navbar and there should be no more navbar issues, so i have closed this issue, if you have any queries, you can join discord server for further updates and communication.
we are not looking for navbar updates for now, as it was done and changes many times before, please understand.
and already it is in updation, so please understand.


Add support for a custom document in watchModals

When rendering into iframes, for example when using react-frame-component, the usual intention is to listen for events from that iframe's document/window and not from the ones in the frame in which the script is executed (those can be two different global environments).
This is a somewhat test PR - we wonder if you would be open to accepting similar ones for more places in the code. I believe that a useGlobals hook (or similar) could be introduced and those globals could be provided using React context so all hooks could receive that information from the context, without manual prop drilling using options etc.

This is a somewhat test PR - we wonder if you would be open to accepting similar ones for more places in the code. I believe that a useGlobals hook (or similar) could be introduced and those globals could be provided using React context so all hooks could receive that information from the context, without manual prop drilling using options etc.

That seems like a good idea. Would you expect that you could pass them in directly as options like in this PR in addition to via context, or would you only provide the globals through context.
In this case, watchModals is not a hook, so it wouldn't be able to use context anyway, so fine with merging it for now, but we should decide what the API should be in general.

Would you expect that you could pass them in directly as options like in this PR in addition to via context, or would you only provide the globals through context.

I would only expect those to be provided using context. IMHO this simplifies things because you can just configure it at the "app" boundary and be done with it.


feat: synchronize the inCodeValidator output to match the CLI -json

Purpose:

Both the inCodeValidator and CLI -json return objects, so we want to standardize and keep these formats in sync.
Make output changes to fewer places when introducing new features. Simplify the code base.

Changes:

Add infos and hints to the JSON output.
Split the object creation and the printing of the JSON object into separate functions, so the inCodeValidator may utilize the same object-creation function.

Tests:

Add a test utility that aggregates the inCodeValidator errors, warnings, etc. into a single array to convert output to a format that is still compatible with existing tests.
Remove checks that expect the infos and hints fields to be undefined.

:tada: This PR is included in version 0.36.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


admin page shows my site is not provisioned by cloudflare

Confirmation

[X] My issue isn't already found on the issue tracker.
[X] I have replicated my issue using the latest version of the plugin and it is still present.

WordPress version
5.9
Cloudflare-WordPress version
4.9.1
PHP version
7.4
Expected result
able to login to or link to my cloudflare account and config settings
Actual result
buttons not responding in the admin panel and the following message shown
It looks like your domain stg.i-cable.com is not provisioned with Cloudflare. Please continue to [Cloudflare](https://dash.cloudflare.com/?to=/:account/add-site) to secure and speed up your website.

Steps to reproduce

install the plugin
activate the plugin
go to setting -> cloudflare

Additional factoids
I have tried adding WP_HOME and WP_SITEURL into my wp-config.php according to #240 but the problem still not being solved
References
#240

I'm afraid this isn't an issue with the plugin but your site or usage and there is very little I can do to help other than direct you to the documentation we have. I'd recommend checking your site is setup on Cloudflare, and then try to install the plugin (keeping in mind things like domains have to match and be provisioned on Cloudflare to be used)


Failure to replicate results

Has anyone been able to successfully replicate the model from the paper? I've been training for about two weeks (over 240k iterations) using the published parameters, along with training an additional ~3k iterations with coverage. Here is what my training loss looks like:

Unclear what caused the increase starting around iteration 180k, but even then the output was not looking great.
Here are some REF (gold) and DEC (system) summaries. As you can see, they are qualitatively bad. Unfortunately, at the moment, I can't figure out how to get pyrouge to run so I can't quantify the performance relative to the published results.

000000_reference.txt
REF: marseille prosecutor says so far no videos were used in the crash investigation '' despite media reports . journalists at bild and paris match are very confident '' the video clip is real , an editor says .
andreas lubitz had informed his lufthansa training school of an episode of severe depression , airline says .
DEC: robin 's comments are aware of any video footage , german paris match .
he 's accused into the crash of germanwings flight 9525 flight .
prosecutor : `` it is a very disturbing scene ''
000001_reference.txt
REF: membership gives the icc jurisdiction over alleged crimes committed in palestinian territories since last june .
israel and the united states opposed the move , which could open the door to war crimes investigations against israelis .
DEC: palestinians signed icc 's founding rome statute of alleged crimes in palestinian territories .
israel says `` in the occupied palestinian territory to immediately end and injustice , she says .
it 's founding rome .
000002_reference.txt
REF: amnesty 's annual death penalty report catalogs encouraging signs , but setbacks in numbers of those sentenced to death .
organization claims that governments around the world are using the threat of terrorism to advance executions .
the number of executions worldwide has gone down by almost 22 % compared with 2013 , but death sentences up by 28 % .
DEC: it 's death sentences and executions 2014 '' is some we are closer to abolition , to advance executions '' number of deterrence , `` a number of countries are abolitionist '' amnesty says he would not be used for the death penalty .
000003_reference.txt
REF: amnesty international releases its annual review of the death penalty worldwide ; much of it makes for grim reading .
salil shetty : countries that use executions to deal with problems are on the wrong side of history .
DEC: soldiers who a china agreed to tackle a surge in death sentences to death .
jordan ended china 's public mass sentencing is part a china 's northwestern xinjiang region .
a sharp spike in december 2006 , 2014 , 2014 .
000004_reference.txt
REF: museum : anne frank died earlier than previously believed .
researchers re-examined archives and testimonies of survivors .
anne and older sister margot frank are believed to have died in february 1945 .
DEC: bergen-belsen concentration camp is believed death on march 31 , anne frank says .
four the jewish diarist concentration camp , margot , margot , violent , died at the age of 15 .
`` i am no more than a skeleton camp , '' witnesses say .

If anyone has had success reproducing the published model I would love to hear how you did it. I'm stumped.

Hi @hate5six. Yes, looks like your system is generating summaries that are quite nonsensical. I'm not sure what would cause this.

You say your 240k iterations took two weeks. Did you start by setting max_enc_steps and max_dec_steps to something small, then increase later? There's a note about it in the README. We found this necessary in order to do quicker iterations. That's how we did 230k iterations in the 3 days 4 hours reported in the paper. I think most of our training iterations for the pointer-generator model were with max_enc_steps=200 and max_dec_steps=50, then we increased to the full  max_enc_steps=400 and max_dec_steps=100 only for a short training period near the end. I think you could go even lower -- with the baseline model we experimented with starting at max_enc_steps=50 and max_dec_steps=50 and that was even more efficient.

Have you seen the note about chunking data? I'm not sure if this actually makes a difference to reproducibility, but we released the note in case it does.

Are you aware that you can look at the full output on the test data (there's a link to it in the README)? This gives you a way to compare with the published results, that's more comprehensive than just the ROUGE numbers / handful of examples in the paper.

Hi @abisee,
I've tried training the model in several different ways on multiple GPU machines--starting with max enc/dec steps at different configurations like 50/50 and scaling them up. All attempts where failing to replicate results so at some point I decided to just let it train at 400/100 at the cost of running time. Part of that was due to not having clear intuition about when to ramp up. I currently have a separate model training that started out at 50/50 and then was ramped up to 200/50, and is currently at 160k iterations. As you can see it hasn't seemed to flatten out much:

I saw the update about chunking data earlier but I haven't incorporated it yet. I similarly don't have the intuition of whether I would need to restart training from scratch or continue training from my current point (240k iterations)--and if I can resume then for how long.
Thanks for the link to the output. I'm looking at it and agree your published results look far more comprehensible. I understand that training models can be a delicate process, but I'm struggling to see what could potentially be happening.

@hate5six Thanks for the feedback. We want these results to be reproducible, which is why we released the code. Before release we needed to significantly cleanup the code to make it comprehensible and usable for others, upgrade to TensorFlow 1.0, etc. Hopefully these changes haven't affected the reproducibility of the experiments, which as you say are a delicate process anyway.
When I have some time I'll look into this and try to figure out what might be going on.

@abisee I would appreciate that! Perhaps in the meantime I can fork my 200/50 model and try running with the chunked input and let that run for a while before ramping up to 400/100

@abisee the following results are obtained:
INFO:tensorflow:REFERENCE SUMMARY: the 20th mls season begins this weekend . league has changed dramatically since its inception in 1996 . some question whether rules regarding salary caps and transfers need to change .
INFO:tensorflow:GENERATED SUMMARY: historic occasion was the first ever major league soccer match . the historic occasion was the first ever major league soccer match . it 's first of a new domestic tv and media rights deal with american soccer .
INFO:tensorflow:REFERENCE SUMMARY: bafetimbi gomis collapses within 10 minutes of kickoff at tottenham . but he reportedly left the pitch conscious and wearing an oxygen mask . gomis later said that he was `` feeling well '' . the incident came three years after fabrice muamba collapsed at white hart lane .
INFO:tensorflow:GENERATED SUMMARY: french striker bafetimbi gomis was feeling well '' after collapsing during tottenham . tottenham scored in the seventh minute of treatment . gomis was fine , '' with manager garry monk using the same condition .
While the first one looks off the mark, the generated second summary (about french striker bafetimbi ) is definitely making sense (great work you and team. Thanks a ton.). I am looking at the details.
Unfortunately, run_summarization stopped with "Error: model produced a word ID that isn't in the vocabulary". Any idea why this might occur and how to solve this?

Hi @hate5six @makcbe @abisee
is the generated summary of fixed length? How to make it variable?

Wondering if there is anyone out there who was able to reproduce the results. I'm currently using chunked data and I played with 200/50 and 400/100 for the sequence lengths. I ran several times into NaN issues (after 35K, around 70K and around 145K steps). I restarted each time, last time with a saved checkpoint from around 110K step. I find this rather painful and wondering why this happens.
Any insights? @abisee Thank you in advance for all the work!

Hi @ioana-blue, I too have lost a lot of time trying to get things to run. Make sure you have the latest version of the code and data. There were changes recently to handle empty articles which cause some of the NaN issues. You may also want to look at this: https://github.com/abisee/pointer-generator/issues/4
I found that helpful in getting around the NaN issues at 35k. I was able to train the model to over 200k iterations after making that change.

@hate5six thank you for your input. my code is up-to-date, I think. I added the modification that you suggested from #4 . I had used it before with no success, I added it back, let's see. I'll update here when I have news.

@hate5six Do you get reasonable results when decoding with the trained model for 200K iterations?

@ioana-blue my model reached it last night and I'm just starting to investigate its performance now. I did run with coverage for an additional 3k iterations this morning, as advised by the authors. The generated summaries on the test set are readable, however, I'm also running the model on some of my own data and I'm seeing that the results are largely extractive. If anything the model removes some excess but it doesn't seem to be generating text. I have an issue/example at: https://github.com/abisee/pointer-generator/issues/21

@makcbe: that FLAGS error was fixed in this commit. Make sure you pull it and other fixes.

@ioana-blue:

@abisee - will it be the right next step to increase the vocab count to 200k and start the train, eval process again?

Increasing the vocabulary size from 50k up to 200k will make everything run significantly slower. In our paper we found that (at least for the baseline model) a vocabulary of 150k didn't work better than a vocabulary of 50k. If I had to guess, I don't think increasing the vocabulary would help much. As you found, that "Error: model produced a word ID that isn't in the vocabulary" message was printing because there was a small bug regarding FLAGS. I think nothing was actually going wrong with the vocabulary and word IDs. So increasing vocabulary was a red herring - sorry for confusion.

@abisee I think your previous message was meant for @makcbe if I'm not mistaken :)

@hate5six did you manage to replicate the results?
I trained the model with all default parameter (then, it's the pointer model, right?!) and added some coverage training. The decoding give results far from what is published:
ROUGE-1:
rouge_1_f_score: 0.3142 with confidence interval (0.3122, 0.3161)
rouge_1_recall: 0.3368 with confidence interval (0.3345, 0.3391)
rouge_1_precision: 0.3218 with confidence interval (0.3193, 0.3241)

ROUGE-2:
rouge_2_f_score: 0.1240 with confidence interval (0.1222, 0.1255)
rouge_2_recall: 0.1339 with confidence interval (0.1320, 0.1357)
rouge_2_precision: 0.1267 with confidence interval (0.1249, 0.1283)

ROUGE-l:
rouge_l_f_score: 0.2855 with confidence interval (0.2837, 0.2874)
rouge_l_recall: 0.3060 with confidence interval (0.3036, 0.3082)
rouge_l_precision: 0.2927 with confidence interval (0.2903, 0.2948)

@pltrdy I did, qualitatively at least. The results I was seeing looked similar to what's in the paper/blog post and the final loss with/without coverage agreed, too. I had trouble installing pyrouge so my analysis stopped short of comparing the actual ROUGE scores, unfortunately.

@hate5six could you then provide your decoded files? Would be interested to score it myself.

@pltrdy it looks like I only ran it on 1500 validation articles. I'd be happy to share them but it also wouldn't make for a complete comparison against the published results. Let me know. I could process the whole validation set but I don't have an ETA on that at the moment given my current workload/resource availability.

Or maybe could you just upload your best checkpoint files? I would run decoding & scoring then (and share the result with you if you want)

@pltrdy I trained according the given number of steps in paper and I was able to get
Mine Rouge-1 F1 score = 37.34 | Rouge-2 F1 score = 16.92
Which is 2/1 points less.  I can share the checkpoint files if needed.
I used Java Rouge calculator. I couldn't figure out how to use the pyrouge.
Do you have the files/instructions need to use pyrouge?
Thanks

@abhishekraok interesting, could I know how did you trained your model? (how many training steps? how many coverage training steps?)
You can find PyRouge instructions here: https://pypi.python.org/pypi/pyrouge/0.1.3 you basically need to download ROUGE and to tell pyrouge where it is located using: pyrouge_set_rouge_path /absolute/path/to/ROUGE-1.5.5/directory.
I'm personnally using my own files2rouge tool.

Actually I managed to figure out how to get the Perl Rouge files. I have added a PR for that to be added to the readme. Here are the numbers
ROUGE-1:
rouge_1_f_score: 0.3834 with confidence interval (0.3812, 0.3856)
rouge_1_recall: 0.4362 with confidence interval (0.4335, 0.4390)
rouge_1_precision: 0.3640 with confidence interval (0.3616, 0.3665)
ROUGE-2:
rouge_2_f_score: 0.1668 with confidence interval (0.1647, 0.1689)
rouge_2_recall: 0.1900 with confidence interval (0.1876, 0.1923)
rouge_2_precision: 0.1587 with confidence interval (0.1566, 0.1608)
ROUGE-l:
rouge_l_f_score: 0.3500 with confidence interval (0.3478, 0.3521)
rouge_l_recall: 0.3978 with confidence interval (0.3953, 0.4003)
rouge_l_precision: 0.3325 with confidence interval (0.3302, 0.3348)

@pltrdy  I trained for 230K steps without coverage and 3k steps with coverage as was mentioned in the paper. I have modified the code in my fork to run for exact number of steps. Thanks for the info on Rouge.

How did you guys manage to continuing training with coverage turned on? When I turn on coverage, I am getting Adagrad not found in checkpoint error.

You may be interested in the pretrained model that is now available -- see README.

@abhishekraok Thank you for sharing your modifications, could you share a checkpoint for us to evaluate the model? Thank you very much.

@abhishekraok, @ioana-blue  and @hate5six  Did any of you train the baseline model for this? Due to TF version issue author could not publish the pre-trained model for the baseline  architecture so it would be really great if any of you did it and is ready to share it.
Thank you.

@jasonwbw @LeenaShekhar you can find the trained files for which I got the above results here https://1drv.ms/f/s!AizkA_PooBXSk_5WDscjQfcdqfjF6g The tf version is 1.2.0

@jasonwbw @LeenaShekhar I have uploaded the files from which I got the metrics here https://1drv.ms/f/s!AizkA_PooBXSlIJO82kT0PZ_IboUrA It uses TF 1.2.0

@abhishekraok
May I ask how you generated the files for running Perl scirpt directly? I have tried several pyrouge packages and still cannot get the right result.
Did you self-generate the xml file and use html files of the news for getting the score?

Also, has anyone figured out why pyrouge does not work?
I find many published paper use that package, but I tried many times and still cannot get correct result from it.


 We want to set up a dev-setup.py script that creates and installs a venv, so the dependencies are contained.

Closed in #8


Adding Dependabot for automated dependency upgrades.

Issue #, if available: #309
Description of changes:

Adding dependabot config file to enable automated dependency upgrades

Docs: https://docs.github.com/en/github/administering-a-repository/about-dependabot-version-updates
How to configure: https://docs.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

You can see example automated PRs in my fork: https://github.com/wrijeff/simple-ingest-transformation-utility-pipeline/pulls

Thanks for doing this, this definitely is handy with Insights and other associated features it brings in [in addition to dependency versions]

We can not do major/minor(sometimes) version changes always, I know we have the integration tests that builds and tests data prepper to an extent, so it should catch them but I am worried about OpenTelemetry because they could mess the protobuf between versions. Definitely patch is a good one to do.
Our projiect uses versionMap does it support that?
We should skip the gradle java app inside the examples, we have no integ test to catch it.

We can not do major/minor(sometimes) version changes always, I know we have the integration tests that builds and tests data prepper to an extent, so it should catch them but I am worried about OpenTelemetry because they could mess the protobuf between versions. Definitely patch is a good one to do.
Our projiect uses versionMap does it support that?
We should skip the gradle java app inside the examples, we have no integ test to catch it.

@kowshikn: great points.

Yep, for some of our less-stable dependencies we should probably ignore those for now. I can add otel proto to the config file and we can watch for automated upgrades that look questionable. And yes we will need to continue to invest in our automated testing to catch issues caused by version upgrades.
Version map - I don't think so. But if all dependencies are automatically bumped to the latest then they should remain consistent across all packages (as all will be bumped to the same version).
Good catch. Unfortunately it doesn't look like we can ignore just one directory, so we'll instead have to list out all the directories we do want scanned.

I feel this tool does more good than harm overall (with a bit of effort on our part to integrate it) - I can move this to a new "dependencies" branch where we can slowly bring in each package as we get more comfortable with the tool. Thoughts?

I absolutely agree that this tool is more good. I have no concerns adding it even to the master branch because there is always a PR and I think we can find issues and the our integ test is very solid at least for Trace Analytics.
The real issue is the how we build our dependency in this project, its not something I am very happy about. I am sure we can fix some of them soon when we address the reduce build/docker image size.


Add option that some functions autocompletion adds [] instead of ()

Motivation
Some functions only have an argument of type content as the only mandatory argument. When I do not want to set any of the optional arguments e.g. #footnote[My text $x^2$] is a nicer syntax than #footnote([My text $x^2$]), so I remove the parenthesis and add the square brackets. For some functions (e.g. footnote) it is very common. So it would be nice if there is an option for the user to give a list of functions for which by default the autocompletion should add the [] instead of the ().
Description
Add the option that the user can give a list of functions for which by default the autocompletion should add [] instead of ().
Examples/Questions
This is useful for a variety of commonly used functions, e.g. footnote, smallcaps, strong, ...

Passing too many options to language server might not be ideal, especially considering function with same name.
A better solution might be annotating such information in doc comment for functions, while there is no real doc comment spec for Typst yet.


LodgingBusiness ContextType

I created a new ContextType file called LodgingBusiness. It looks like this:
<?php

namespace JsonLd\ContextTypes;

class LodgingBusiness extends LocalBusiness
{
    /**
     * Property structure
     *
     * @var array
     */
    protected $structure = [
        '@id' => null,
        'starRating' => null,
        'petsAllowed' => null,
    ];
}

I'm trying to use it like this:
$context = \JsonLd\Context::create('lodging_business', [
    'name' => $this->row->title,
]);

but i cant get any output. What am i doing wrong?

It's in the ContextTypes Folder.
Looks like this:
<?php

namespace JsonLd\ContextTypes;

class LodgingBusiness extends AbstractContext
{
    /**
     * Property structure
     *
     * @var array
     */
    protected $structure = [
        'name' => null,
        'description' => null,
        'image' => null,
        'areaServed' => null,
        'priceRange' => null,
        'telephone' => null,
        'faxNumber' => null,
        'email' => null,
        'openingHours' => null,
        'petsAllowed' => null,
        'address' => PostalAddress::class,
        'geo' => GeoCoordinates::class,
        'url' => null,
        'aggregateRating' => AggregateRating::class,
        'review' => Review::class
    ];

    /**
     * Set the opening hours of the business.
     *
     * @param  array $items
     * @return array
     */
    protected function setOpeningHoursAttribute($items)
    {
        return $items;
    }

    /**
     * Set the Reviews
     *
     * @param array $items
     * @return array
     */
    protected function setReviewAttribute($items)
    {
        if ( ! is_array($items))
        {
            return $items;
        }

        return array_map(function ($item) {
            return $this->getNestedContext(Review::class, $item);
        }, $items);
    }

}


[Question] Run viewer on private ip than local host

Hi,
i am having an issue with viewing my rendering of the prompt using the viewer script when I run it on my server. Since I cannot access the localhost link through my server's console, I forward it using ngrok and view it on my local machine (macbook pro). However, the checkpoint doesn't render. As such, I am wondering if there is a way to run the viewer on a private ip than localhost.

I am able to look at the gifs used for eval in the wandb folder so I know the model trained properly but there seems to be some issue with the viewing.
If using the splat viewer is a better option, how and what do I upload to view it in the browser?

Hi coolkite,
Thank you for your interest. In my practical usage, I will use SSH for port tunneling.
ssh -L <YOUR_LOCAL_PORT>:<YOUR_SERVER_IP>:<YOUR_SERVER_PORT> <YOUR_USER>@<YOUR_SERVER>

where the <YOUR_SERVER_PORT> is 8080 by default.
If you wish to use the splat viewer, first convert the .pt checkpoint to .ply file by using the scripts:
python utils/export.py <your_ckpt> --type ply

where the <your_ckpt> can be the path to the .pt checkpoint file or, more conveniently, can be the id for the run (the display name of the run in wandb, e.g. 0|213630|2023-10-11|a_high_quality_photo_of_a_corgi).
After running this command, you can find the exported .ply file in the exports/ply directory.
Then, open the online splat viewer and drag the .ply file into the browser window.
I am really sorry for the inconvenience caused by the design and the incomplete documentation. Feel free to ask any questions about the viewer and our method.
Best regards,
heheyas


Normal Mode Keybindings

I was going to implement a few ones but saw that @gchp already did.
I see a few problemas:

In vim '^' means start of non whitespace text in line, not start of line, which would skip identations, tabulations, etc.
Start of line in vim is actually 0 (zero).
While undo is 'u', redo is actually ctrl+r because 'r' is for replacing characters didnt see it already is implemented as ctrl+r

Yup, you're right about the start of line thing. We don't have a way of finding the first non-whitespace char on a line, however if you want to add those, that would be cool!
I believe I already have Ctrl+r bound to redo?
Modes are still very new, so there's lots to do yet! Thanks for contributing!