feat: mman NonNull

Updates functions in  nix::sys::mman to use NonNull.
The case where the returned pointer is null is the error case, therefore in the Ok case we can return NonNull.

Given that mmap() won't return a null pointer at all, I am actually not sure if this change is needed, thoughts on this?
cc @asomers

n that mmap() won't return a null pointer at all, I am actually not sure if this change is needed, thoughts on this?

If it always returns a non-null pointer this seems even more reason to return NonNull.

Given that mmap() won't return a null pointer at all, I am actually not sure if this change is needed, thoughts on this?

If it always returns a non-null pointer this seems even more reason to return NonNull.

I think both our points are correct here:
From the interface safety perspetive, mmap() won't return a NULL pointer, so wrapping it in a NonNull is useless. However, for pointers that are not NULL, wrapping it in a NonNull will definitely make the interface more Rusty.
So I am not against this now, but still ping @asomers, I would like to hear your thoughts

Just adding my two cents here.
NonNull<T> differs from *mut T in two aspects:

The pointer is guaranteed to be not NULL (obviously). This allows the compiler to perform some optimizations.
NonNull<T> is covariant in T. This is usually desired when dealing with unique pointers (i.e. we own T) or when dealing with shared pointers (e.g. shared ownership Rc, Arc).

mmap allocates new virtual memory and returns a unique pointer to the allocation, which exactly fits the use case of representing ownership with NonNull. This alone makes this change worthwhile for me.
Additionally, this function is fallible and we are only one call away of turning the result into an Option by calling Result::ok, where the aforementioned optimizations apply.
The only reason why NonNull is so underused even in the standard library is that many functions have already been stabilized before NonNull was invented. The first (?) API in std that will use NonNull instead of *mut is the upcoming Allocator API, which coincidentally is also what mmap essentially is.

Just adding my two cents here.
NonNull<T> differs from *mut T in two aspects:

The pointer is guaranteed to be not NULL (obviously). This allows the compiler to perform some optimizations.
NonNull<T> is covariant in T. This is usually desirable when dealing with unique pointers (i.e. we own T, e.g. Vec, Box) or when dealing with shared pointers (e.g. shared ownership Rc, Arc).

mmap allocates new virtual memory and returns a unique pointer to the allocation, which exactly fits the use case of representing ownership with NonNull. This alone makes this change worthwhile for me.
Additionally, this function is fallible and we are only one call away of turning the result into an Option by calling Result::ok, where the aforementioned optimizations apply.
The only reason why NonNull is so underused even in the standard library is that many functions have already been stabilized before NonNull was invented. The first (?) API in std that will use NonNull instead of *mut is the upcoming Allocator API, which coincidentally is also what mmap essentially is.

Makes sense to me, if we are gonna merge  this change, should we also update the munmap(2) interface to something like?
pub unsafe fn munmap(addr: NonNull<c_void>, len: size_t) -> Result<()>

Yes, good point. For symmetry reasons, I believe all other functions in mman should implement this change, too.

Yes, good point. For symmetry reasons, I believe all other functions in mman should implement this change, too.

I've updated the PR with all the other changes.

Yes, good point. For symmetry reasons, I believe all other functions in mman should implement this change, too.

I've updated the PR with all the other changes.
I think a follow-up PR could update the functions to take generic NonNull<[T]> instead of taking NonNull<c_void> and size_t.

I agree, this is a way nicer abstraction, but since this function allocates in pages and doesn't care about the layout of T, an idiomatic mmap should probably just return a page-aligned NonNull<[u8]> and the other functions taking the allocation as an argument should use the page-aligned starting pointer NonNull<u8>, similar to the std allocator trait.
The metadata of the slice pointer would then be the length rounded up to the next multiple of the page size.


HelpText() not working on checkboxes

The helptext-div is not being rendered when you call .HelpText() on a Checkbox.
For example:
@f.FormGroup().CheckBoxFor(m => m.MyProperty).HelpText("Description")
I've tried both overloads.
TwitterBootstrapMVC 3.17.7

Could you tell me if this will be fixed @DmitryEfimenko?

Sorry for the delay. My focus has been moving in other areas lately.
I won't be touching BMVC much going forward.
I'll be fixing this at a later time in bulk with other issues if these come around.

Okay, thanks.


Gradual Dependency Tree Caching

Gradual Dependency Tree Caching (that I will call GDTC since it is a long term) is something that would definitely increase performance, depending on how you use calculator-based objects (Match, Calc).
Consider this example:
Calc(function(useState)
    return useState(one) or useState(two)
end, JustOk)

Scenario 1
Imagine one is a truthy value, and we don't care about two's value for now as well. if one was changed three times, and the three times, the newValue was truthy, it would still trigger a dependencies recapture (detection + tree simplification).
This means that we are spending some time recapturing the same dependencies. While this is a basic example, it gets worse with more advanced usage.
Consider this:
Calc(function(useState)
    local x = useState(x)
    local y = useState(y)
    local z = useState(z)
    local d = useState(d)

    return x + y + z + d
end, JustOk)

This is just a basic demonstration of calc usage which we don't care about lazy evaluation (and we don't need it either). This time, x, y, z, d state objects get recaptured every time the update- despite there are no differences in every case.
Scenario 2
repost:
Calc(function(useState)
    return useState(one) or useState(two)
end, JustOk)

Imagine that is one is always a falsy value, so in every case, we are going to use two's value. An update occurred to one, but its obviously a falsy value, so even though dependencies aren't going to change, we're still recapturing the same dependencies.
Solution: GDTC
A solution to this issue is to implement gradual caching for dependencies trees. It is gradual, because at creation, it is very possible to not be able to predict all the dependencies we are going to use (we could manually pass our dependencies and make useState and useKeyState do some internal job to let us know what dependencies we are currently using), although it can have an instant effect and make calc/match updating blazingly fast after creation (since technically, these objects' creation use the same mechanism as u[dating them)

deferred to V0.3


CLI on Mac doesn't seem to work

Ref https://github.com/microsoft/vscode/issues/102975

Run code-insiders dirname where dirname is the name of a directory.
🐛 Insiders may or may not open dirname.
In a separate terminal, run code-insiders dirname2 where dirname2 is the name of another directory.
🐛 Insiders doesn't open that directory.

Replacing the commands with calls to open doesn't work.

Unable to repro with latest insiders.
Screen Recording.zip

The issue above in my case seems to be caused by having VSCODE_DEV=1 in the env. I'll check the options parameter again and see if I can make it so that during the open command, VSCODE_DEV gets reset. That would help me test using the open command by running the following on the command-line: VSCODE_DEV=1 vscode/.build/electron/Code\ -\ OSS.app/Contents/MacOS/Electron vscode/out/cli.js repo-name

How are you running the open command ? Is it via the cli wrapper, if not can you provide the full command used.

vscode/out/cli.js is the cli wrapper, and that calls the spawn function to run the open command. The open command has the --env arg so I can try using that. spawn also has an options param, though I left it blank for now.

Yes you need to pass the environment set via the spawn options into the open command for things to work correctly https://github.com/microsoft/vscode/blob/9921c378a0006f14b0386ae2e531058c9363ef72/src/vs/code/node/cli.ts#L313-L316


[QUERY] How to get a token to be used when calling endpoints secured behind a API Management Instance

Library name and version
Azure.Identity 1.5.0
Query/Question
I have an API running within an Azure App Service which has a System Assigned Managed Identity, I want to call services exposed via Azure APIM using this identity but I don't know how to get a token. I'm trying to do it as:

        var credentials = new DefaultAzureCredential();
        var token = await credentials.GetTokenAsync(new TokenRequestContext());

But the request fails due to this method requiring scopes, what is the way of getting a token for the app service managed identity?
Environment
.NET 6.0
Azure App Service (Win/Linux)
System Assigned Managed Identity
Azure API Management

Thank you for your feedback.  Tagging and routing to the team members best able to assist.

Hi @evilpilaf - Which service are you trying to use the token with? You'd need to configure the scope that is specific to that service. For example, if it were Azure Storage, the scope would be https://storage.azure.com/.default

Hi @christothes thanks for the reply. I'm trying to call a system behind Api Management, but it's APIM which will be making the validation. Does this mean then that the scope would be <my-apim>.domain.com ? If so could it use my custom domain or do i need the azure name?

I'm not very familiar with the specifics of APIM, but this docs page describes what is required. https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-protect-backend-with-aad

Thanks @christothes, but if I understand correctly this is for registering and using an app in AAD, I was expecting to be able to use the Managed Identity assigned to the App Service directly without having to register anything else in AAD.

As I understand it, in this scenario the client is authenticating only to APIM (represented by the newly created App registration), and APIM makes the request to some back-end resource on its behalf with some identity configured in APIM.
The client would be consumers of the APIM exposed APIs and are unaware of the back-end resources or how auth is happening to them. Managed Identities could come into play here if you have an Azure service (such as an Azure Function) that is calling into your APIM API. In that case the Managed Identity represents the client in the request from <AzureFunction> => <APIM API>


Proposal: struct instead of enum for arrays

Instead of enum:
pub enum Register {
    /// A single register.
    Single(RegisterInfo),
    /// An array of registers.
    Array(RegisterInfo, DimElement),
}

I propose to use struct:
struct Register {
    info: RegisterInfo,
    dim: Option<DimElement>,
}

where dim: None corresponds to Register::Single.
The main goal is to simplify serialize/deserialize.
cc @Emilgardis  @therealprof  @adamgreig

It would also be possible to do this with
#[cfg_attr(
    feature = "serde",
    derive(serde::Serialize, serde::Deserialize),
    serde(untagged)
)]
pub enum Register {
    /// A single register.
    Single {
        /// Register information
        info: RegisterInfo,
    },
    /// An array of registers.
    Array {
        /// Register information
        info: RegisterInfo,
        /// Array dim
        dim: DimElement,
    },
}

I wonder which one is the best

It would also be possible to do this with
#[cfg_attr(
    feature = "serde",
    derive(serde::Serialize, serde::Deserialize),
    serde(untagged)
)]
pub enum Register {
    /// A single register.
    Single {
        /// Register information
        info: RegisterInfo,
    },
    /// An array of registers.
    Array {
        /// Register information
        info: RegisterInfo,
        /// Array dim
        dim: DimElement,
    },
}

I wonder which one is the best

Unfortunately the result is different from I expect.

I can take identical result with:
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(
    feature = "serde",
    derive(serde::Serialize, serde::Deserialize),
    serde(untagged)
)]
pub enum Register {
    /// A single register.
    Single(RegisterInfo),
    /// An array of registers.
    Array(RegisterArray),
}

#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
/// ...
pub struct RegisterArray {
    #[cfg_attr(feature = "serde", serde(flatten))]
    /// ...
    pub dim: DimElement,
    #[cfg_attr(feature = "serde", serde(flatten))]
    /// ...
    pub info: RegisterInfo,
}

But does this make sense?

Let's just merge #173 instead.

The reason for the difference is the ordering, placing dim before info in the enum struct would solve that.
Does

Ah, my bad, I see the issue now. Too bad, I dont think there is a good way to solve this with current serde features


CSUB-553: Sync-disconnect-and-upgrade test

Description of proposed changes

Practical tips for PR review & merge:

[ ] All GitHub Actions report PASS
[ ] Newly added code/functions have unit tests

[ ] Coverage tools report all newly added lines as covered
[ ] The positive scenario is exercised
[ ] Negative scenarios are exercised, e.g. assert on all possible errors
[ ] Assert on events triggered if applicable
[ ] Assert on changes made to storage if applicable

[ ] Modified behavior/functions - try to make sure above test items are covered
[ ] Integration tests are added if applicable/needed

Tests against the disconnected live node, synced from Testnet are passing here: https://github.com/gluwa/creditcoin/actions/runs/5578025040/jobs/10192016515

Rebasing & ready for review


How do i install this plugin using node js

Hello rajesh
How do i install this plugin using node js

As of now you can only install this using bower. You can directly download the files and use them. But installation with node is not yet present, for this component.

Ok Thank you


POT Creation date and Revision Date

I'm most probably overlooking something. If so my sincere apologies.
Is there a way to auto-populate the POT-Creation-Date and the PO-Revision-Date headers automatically? I'm using the gulp wrapper and these headers do not get populated by default.

No, it's not possible right now and I don't have the time to implement it right now. Pull requests are welcome!
If this is implemented it should be via an option to enable it since I don't think there is no way to know if anything is actually changed.

From the docs here it seems PO-Revision-Date should only be set by the PO editor (so not the .pot creater aka wp-pot).
To set the POT-Creation-Date header, just use:
let today = [
  `${d.getUTCFullYear()}`,
  `-${String(d.getUTCMonth()+1).padStart(2, '0')}`,
  `-${String(d.getUTCDate()).padStart(2, '0')}`,
  ` ${String(d.getUTCHours()).padStart(2, '0')}`,
  `:${String(d.getUTCMinutes()).padStart(2, '0')}`,
  `+0000`
].join('');

wpPot( {
  headers: {
    'POT-Creation-Date': today,
  }
} )


bump fauxton to latest & greatest

let's see if we get a green build :)

I pinned Fauxton to 1.0.6 release tag. I think it's good practice to pin only releases, wish to see more them from Fauxton (: You can close this.


Fabio running using Nomad system scheduler breaks Docker.

I realise how unlikely the title to this issue seems but if there is an obvious error in my set up I can't spot it. I want to run Fabio as a Nomad managed service using the Nomad system scheduler (type = "system"). When I do then any subsequent pulls from our private Docker registry fails with the error
failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory
From that point on I am not able to recover Docker.
Some details about our set up:
Ubuntu 14-04
Kernel = 3.13.0-53-generic
Docker = 1.12.2
Nomad = 0.5.0
Fabio = 1.3.4
I have a 3 x servers with 2 x clients. I am trying to run Fabio using the exec driver and the system scheduler. I am running Nomad as the root user on which I believe is required for the exec driver.
I do not see the issue if I run Fabio using the service scheduler.
I do not see the issue if I run a Docker container using the system scheduler .
I do not see the issue if I run another job (sleep binary) using the system scheduler.
I do not see the issue if I run Fabio using the system scheduler but using the raw_exec driver.
Docker is using the LVM storage option but I see the same issue if I drop back to the devicemapper storage option.
Below is a repeatable test case. After that are copies of the job specs used in the test case.

Go to Nomad user
ubuntu@ip-10-75-70-27:~$ sudo su - nomad

Software versions

$ uname -a
Linux ip-10-75-70-27 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 1.12.2, build bb80604
$ nomad version
Nomad v0.5.0

Nomad running as root with no running jobs

$ ps -ef | grep nomad
root     17416     1  0 12:57 ?        00:00:00 /usr/local/bin/nomad agent -config /etc/nomad.d/config.json -rejoin -node=nomad_client_poc1
consul   17540     1  0 12:57 ?        00:00:00 /usr/local/bin/consul agent -config-file /etc/consul.d/config.json -rejoin -node nomad_client_poc1
root     17617  2602  0 12:58 pts/1    00:00:00 sudo su - nomad

Demonstrate Docker pull

$ docker pull dockerregistry.adm.myprivatecloud.net/bti1003:latest
latest: Pulling from bti1003
5a132a7e7af1: Pull complete
fd2731e4c50c: Pull complete
28a2f68d1120: Pull complete
a3ed95caeb02: Pull complete
87f9029820c8: Pull complete
7582f6d126ab: Pull complete
Digest: sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Status: Downloaded newer image for dockerregistry.adm.myprivatecloud.net/bti1003:latest

Remove pulled image

$ docker rmi dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003@sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Deleted: sha256:3fee2600d434e469b6d4ac0e468bd988ebc105536886d6624dc9566577fcafbe
Deleted: sha256:e5fcd939dd4a2a9b9543dea61ca90d2def7c92cd983108916895a39f239799b8
Deleted: sha256:57bd7c9432ae86d63f2342e442eebd0f4dfc340ca61c6a4c7d702b17a315865f
Deleted: sha256:0aaccda2aadfc70ab2248437568fd17f4e8860cf612cc4b7e154b97222dccf91
Deleted: sha256:9dcfe19e941956c63860afee1bec2e2318f6fbd336bc523094ed609a9c437a01
Deleted: sha256:6ff1ee6fc8a0358aeb92f947fb7125cd9e3d68c05be45f5375cb59b98c850b4d
Deleted: sha256:56abdd66ba312859b30b5629268c30d44a6bbef6e2f0ebe923655092855106e8

Run 'sleep' test job

$ ps -ef | grep nomad.*executor
root     17897 17416  6 13:00 ?        00:00:02 /usr/local/bin/nomad executor /var/nomad/alloc/87a48081-5d8e-b1b1-1538-a1e79a3f4152/sleep-task/sleep-task-executor.out
nomad    18299 17619  0 13:07 pts/1    00:00:00 grep nomad.*executor

Pull Docker image

$ docker pull dockerregistry.adm.myprivatecloud.net/bti1003:latest
latest: Pulling from bti1003
5a132a7e7af1: Pull complete
fd2731e4c50c: Pull complete
28a2f68d1120: Pull complete
a3ed95caeb02: Pull complete
87f9029820c8: Pull complete
7582f6d126ab: Pull complete
Digest: sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Status: Downloaded newer image for dockerregistry.adm.myprivatecloud.net/bti1003:latest

Remove pulled image

$ docker rmi dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003@sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Deleted: sha256:3fee2600d434e469b6d4ac0e468bd988ebc105536886d6624dc9566577fcafbe
Deleted: sha256:e5fcd939dd4a2a9b9543dea61ca90d2def7c92cd983108916895a39f239799b8
Deleted: sha256:57bd7c9432ae86d63f2342e442eebd0f4dfc340ca61c6a4c7d702b17a315865f
Deleted: sha256:0aaccda2aadfc70ab2248437568fd17f4e8860cf612cc4b7e154b97222dccf91
Deleted: sha256:9dcfe19e941956c63860afee1bec2e2318f6fbd336bc523094ed609a9c437a01
Deleted: sha256:6ff1ee6fc8a0358aeb92f947fb7125cd9e3d68c05be45f5375cb59b98c850b4d
Deleted: sha256:56abdd66ba312859b30b5629268c30d44a6bbef6e2f0ebe923655092855106e8

Stop 'sleep' job

$ ps -ef | grep nomad.*executor
nomad    18239 17619  0 13:06 pts/1    00:00:00 grep nomad.*executor

Start Fabio job

$ ps -ef | grep nomad.*executor
root     18262 17416 33 13:07 ?        00:00:04 /usr/local/bin/nomad executor /var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task/fabio-task-executor.out
nomad    18299 17619  0 13:07 pts/1    00:00:00 grep nomad.*executor

Pull docker image

$ docker pull dockerregistry.adm.myprivatecloud.net/bti1003:latest
latest: Pulling from bti1003
5a132a7e7af1: Extracting [==================================================>] 65.69 MB/65.69 MB
fd2731e4c50c: Download complete
28a2f68d1120: Download complete
a3ed95caeb02: Download complete
87f9029820c8: Download complete
7582f6d126ab: Download complete
failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory

Fabio job dies (10 minutes later), from syslog

Nov 23 13:17:56 ip-10-75-70-27 nomad[17416]: driver.exec: error destroying executor: 1 error(s) occurred:#012#012* 1 error(s) occurred:#012#012* failed to unmou
nt shared alloc dir "/var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task/alloc": invalid argument
Nov 23 13:17:57 ip-10-75-70-27 nomad[17416]: client: failed to destroy context for alloc '5729f45b-185c-fa7b-7b05-866a774b8c73': 2 error(s) occurred:#012#012* 1 error(s) occurred:#012#012* failed to remove the secret dir "/var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task/secrets": unmount: invalid argument#012* remove /var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task: directory not empty

From Docker log
time="2016-11-23T13:07:59.287783575Z" level=error msg="Error trying v2 registry: failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory"
time="2016-11-23T13:07:59.287830271Z" level=error msg="Attempting next endpoint for pull after error: failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory"

Fabio Job Spec
job "fabio-job" {
  region = "eu"
  datacenters = ["vpc-poc"]
  type = "system"
  update {
    stagger = "5s"
    max_parallel = 1
  }

  group "fabio-group" {
     ephemeral_disk {
      size    = "500"
    }
    task "fabio-task" {
      driver = "exec"
      config {
        command = "fabio-1.3.4-go1.7.3-linux_amd64"
      }

      artifact {
        source = "https://github.com/eBay/fabio/releases/download/v1.3.4/fabio-1.3.4-go1.7.3-linux_amd64"
     }
     logs {
	max_files = 2
	max_file_size = 5
      }
      resources {
        cpu = 500
        memory = 64
        network {
          mbits = 1

          port "http" {
            static = 9999
          }
          port "ui" {
            static = 9998
          }
        }
      }
    }
  }
}

Sleep Job Spec
job "sleep-job" {
  region = "eu"
  datacenters = ["vpc-poc"]
  type = "system"
  update {
    stagger = "5s"
    max_parallel = 1
  }

  group "sleep-group" {
     ephemeral_disk {
      size    = "500"
    }
    task "sleep-task" {
      driver = "exec"
      config {
        command = "/bin/sleep"
        args = ["1000"]
      }

     logs {
	max_files = 2
	max_file_size = 5
      }
      resources {
        cpu = 500
        memory = 64
        network {
          mbits = 1

        }
      }
    }
  }
}

Thanks for the detailed description. My suspicion is that this isn't Fabio related but you've done testing around this. I'm currently on vacation and will start picking up Fabio issues as of next week again.
Frank Schröder

On 24 Nov 2016, at 23:33, michaelmcguinness<EMAIL_ADDRESS>wrote:
I realise how unlikely the title to this issue seems but if there is an obvious error in my set up I can't spot it. I want to run Fabio as a Nomad managed service using the Nomad system scheduler (type = "system"). When I do then any subsequent pulls from our private Docker registry fails with the error
failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory
From that point on I am not able to recover Docker.
Some details about our set up:
Ubuntu 14-04
Kernel = 3.13.0-53-generic
Docker = 1.12.2
Nomad = 0.5.0
Fabio = 1.3.4
I have a 3 x servers with 2 x clients. I am trying to run Fabio using the exec driver and the system scheduler. I am running Nomad as the root user on which I believe is required for the exec driver.
I do not see the issue if I run Fabio using the service scheduler.
I do not see the issue if I run a Docker container using the system scheduler .
I do not see the issue if I run another job (sleep binary) using the system scheduler.
I do not see the issue if I run Fabio using the system scheduler but using the raw_exec driver.
Docker is using the LVM storage option but I see the same issue if I drop back to the devicemapper storage option.
Below is a repeatable test case. After that are copies of the job specs used in the test case.
Go to Nomad user
ubuntu@ip-10-75-70-27:~$ sudo su - nomad
Software versions
$ uname -a
Linux ip-10-75-70-27 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 1.12.2, build bb80604
$ nomad version
Nomad v0.5.0
Nomad running as root with no running jobs
$ ps -ef | grep nomad
root     17416     1  0 12:57 ?        00:00:00 /usr/local/bin/nomad agent -config /etc/nomad.d/config.json -rejoin -node=nomad_client_poc1
consul   17540     1  0 12:57 ?        00:00:00 /usr/local/bin/consul agent -config-file /etc/consul.d/config.json -rejoin -node nomad_client_poc1
root     17617  2602  0 12:58 pts/1    00:00:00 sudo su - nomad
Demonstrate Docker pull
$ docker pull dockerregistry.adm.myprivatecloud.net/bti1003:latest
latest: Pulling from bti1003
5a132a7e7af1: Pull complete
fd2731e4c50c: Pull complete
28a2f68d1120: Pull complete
a3ed95caeb02: Pull complete
87f9029820c8: Pull complete
7582f6d126ab: Pull complete
Digest: sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Status: Downloaded newer image for dockerregistry.adm.myprivatecloud.net/bti1003:latest
Remove pulled image
$ docker rmi dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003@sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Deleted: sha256:3fee2600d434e469b6d4ac0e468bd988ebc105536886d6624dc9566577fcafbe
Deleted: sha256:e5fcd939dd4a2a9b9543dea61ca90d2def7c92cd983108916895a39f239799b8
Deleted: sha256:57bd7c9432ae86d63f2342e442eebd0f4dfc340ca61c6a4c7d702b17a315865f
Deleted: sha256:0aaccda2aadfc70ab2248437568fd17f4e8860cf612cc4b7e154b97222dccf91
Deleted: sha256:9dcfe19e941956c63860afee1bec2e2318f6fbd336bc523094ed609a9c437a01
Deleted: sha256:6ff1ee6fc8a0358aeb92f947fb7125cd9e3d68c05be45f5375cb59b98c850b4d
Deleted: sha256:56abdd66ba312859b30b5629268c30d44a6bbef6e2f0ebe923655092855106e8
Run 'sleep' test job
$ ps -ef | grep nomad.*executor
root     17897 17416  6 13:00 ?        00:00:02 /usr/local/bin/nomad executor /var/nomad/alloc/87a48081-5d8e-b1b1-1538-a1e79a3f4152/sleep-task/sleep-task-executor.out
nomad    18299 17619  0 13:07 pts/1    00:00:00 grep nomad.executor
Pull Docker image
$ docker pull dockerregistry.adm.myprivatecloud.net/bti1003:latest
latest: Pulling from bti1003
5a132a7e7af1: Pull complete
fd2731e4c50c: Pull complete
28a2f68d1120: Pull complete
a3ed95caeb02: Pull complete
87f9029820c8: Pull complete
7582f6d126ab: Pull complete
Digest: sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Status: Downloaded newer image for dockerregistry.adm.myprivatecloud.net/bti1003:latest
Remove pulled image
$ docker rmi dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003:latest
Untagged: dockerregistry.adm.myprivatecloud.net/bti1003@sha256:6d7379af49cc17cc8a0055e06c4cb8374e5be73fe42ce2e8f1abca013c50a62a
Deleted: sha256:3fee2600d434e469b6d4ac0e468bd988ebc105536886d6624dc9566577fcafbe
Deleted: sha256:e5fcd939dd4a2a9b9543dea61ca90d2def7c92cd983108916895a39f239799b8
Deleted: sha256:57bd7c9432ae86d63f2342e442eebd0f4dfc340ca61c6a4c7d702b17a315865f
Deleted: sha256:0aaccda2aadfc70ab2248437568fd17f4e8860cf612cc4b7e154b97222dccf91
Deleted: sha256:9dcfe19e941956c63860afee1bec2e2318f6fbd336bc523094ed609a9c437a01
Deleted: sha256:6ff1ee6fc8a0358aeb92f947fb7125cd9e3d68c05be45f5375cb59b98c850b4d
Deleted: sha256:56abdd66ba312859b30b5629268c30d44a6bbef6e2f0ebe923655092855106e8
Stop 'sleep' job
$ ps -ef | grep nomad.executor
nomad    18239 17619  0 13:06 pts/1    00:00:00 grep nomad.executor
Start Fabio job
$ ps -ef | grep nomad.executor
root     18262 17416 33 13:07 ?        00:00:04 /usr/local/bin/nomad executor /var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task/fabio-task-executor.out
nomad    18299 17619  0 13:07 pts/1    00:00:00 grep nomad.executor
Pull docker image
$ docker pull dockerregistry.adm.myprivatecloud.net/bti1003:latest
latest: Pulling from bti1003
5a132a7e7af1: Extracting [==================================================>] 65.69 MB/65.69 MB
fd2731e4c50c: Download complete
28a2f68d1120: Download complete
a3ed95caeb02: Download complete
87f9029820c8: Download complete
7582f6d126ab: Download complete
failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory
Fabio job dies (10 minutes later), from syslog
Nov 23 13:17:56 ip-10-75-70-27 nomad[17416]: driver.exec: error destroying executor: 1 error(s) occurred:#012#012 1 error(s) occurred:#012#012 failed to unmou
nt shared alloc dir "/var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task/alloc": invalid argument
Nov 23 13:17:57 ip-10-75-70-27 nomad[17416]: client: failed to destroy context for alloc '5729f45b-185c-fa7b-7b05-866a774b8c73': 2 error(s) occurred:#012#012 1 error(s) occurred:#012#012 failed to remove the secret dir "/var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task/secrets": unmount: invalid argument#012 remove /var/nomad/alloc/5729f45b-185c-fa7b-7b05-866a774b8c73/fabio-task: directory not empty
From Docker log
time="2016-11-23T13:07:59.287783575Z" level=error msg="Error trying v2 registry: failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory"
time="2016-11-23T13:07:59.287830271Z" level=error msg="Attempting next endpoint for pull after error: failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory"
Fabio Job Spec
job "fabio-job" {
region = "eu"
datacenters = ["vpc-poc"]
type = "system"
update {
stagger = "5s"
max_parallel = 1
}
group "fabio-group" {
ephemeral_disk {
size    = "500"
}
task "fabio-task" {
driver = "exec"
config {
command = "fabio-1.3.4-go1.7.3-linux_amd64"
}
  artifact {
    source = "https://github.com/eBay/fabio/releases/download/v1.3.4/fabio-1.3.4-go1.7.3-linux_amd64"
 }
 logs {

max_files = 2
max_file_size = 5
}
resources {
cpu = 500
memory = 64
network {
mbits = 1
      port "http" {
        static = 9999
      }
      port "ui" {
        static = 9998
      }
    }
  }
}

}
}
Sleep Job Spec
job "sleep-job" {
region = "eu"
datacenters = ["vpc-poc"]
type = "system"
update {
stagger = "5s"
max_parallel = 1
}
group "sleep-group" {
ephemeral_disk {
size    = "500"
}
task "sleep-task" {
driver = "exec"
config {
command = "/bin/sleep"
args = ["1000"]
}
 logs {

max_files = 2
max_file_size = 5
}
resources {
cpu = 500
memory = 64
network {
mbits = 1
    }
  }
}

}
}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@michaelmcguinness Did you make any progress with this? What do the fabio logs say?

No. Running it as raw_exec to work around the issue. Would I be right in saying that you mean the logs exposed by
$ nomad logs -verbose -stderr <allocid>

That being the case there is nothing interesting there...
2016/12/17 15:52:58 [INFO] Version 1.3.4 starting
2016/12/17 15:52:58 [INFO] Go runtime is go1.7.3
2016/12/17 15:52:58 [INFO] Using routing strategy "rnd"
2016/12/17 15:52:58 [INFO] Using routing matching "prefix"
2016/12/17 15:52:58 [INFO] Setting GOGC=800
2016/12/17 15:52:58 [INFO] Setting GOMAXPROCS=1
2016/12/17 15:52:58 [INFO] Metrics disabled
2016/12/17 15:52:58 [INFO] consul: Connecting to "localhost:8500" in datacenter "vpc-poc"
2016/12/17 15:52:58 [INFO] Admin server listening on ":9998"
2016/12/17 15:52:58 [INFO] HTTP proxy listening on :9999
2016/12/17 15:52:58 [INFO] consul: Using dynamic routes
2016/12/17 15:52:58 [INFO] consul: Using tag prefix "urlprefix-"
2016/12/17 15:52:58 [INFO] consul: Watching KV path "/fabio/config"
2016/12/17 15:52:58 [INFO] consul: Health changed to #2443304
2016/12/17 15:52:58 [INFO] consul: Skipping service "_prometheus-node-exporter-http" since agent on node "dns_slave" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_prometheus-node-exporter-process" since agent on node "dns_slave" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_nomad-server-nomad-serf" since agent on node "nomad_server1" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_nomad-server-nomad-rpc" since agent on node "nomad_server1" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_nomad-server-nomad-serf" since agent on node "nomad_server2" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_nomad-server-nomad-rpc" since agent on node "nomad_server2" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_nomad-server-nomad-http" since agent on node "nomad_server2" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_nomad-server-nomad-serf" since agent on node "nomad_server3" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_prometheus-node-exporter-http" since agent on node "nomad_server3" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Skipping service "_prometheus-node-exporter-process" since agent on node "nomad_server3" is down: Agent not live or unreachable
2016/12/17 15:52:58 [INFO] consul: Manual config changed to #2409046
2016/12/17 15:52:58 [INFO] Updated config to
2016/12/17 15:52:58 [INFO] consul: Registered fabio with id "fabio-ip-10-75-70-74-9998"
2016/12/17 15:52:58 [INFO] consul: Registered fabio with address "<IP_ADDRESS>"
2016/12/17 15:52:58 [INFO] consul: Registered fabio with tags ""
2016/12/17 15:52:58 [INFO] consul: Registered fabio with health check to "http://[<IP_ADDRESS>]:9998/health"
2016/12/17 15:52:58 [INFO] consul: Health changed to #2443305

and then just repeated messages about Consul.

If there is nothing in the fabio logs then that supports my suspicion that this is a docker and/or nomad issue. Maybe the way fabio interacts with docker or the way it shuts down triggers this. However, since you're running this with a single listener you could try to simulate this with a simple go program that runs a web server, then a reverse proxy, and then a reverse proxy that makes long polling http requests.
Below is a simple reverse proxy for testing. Store it in ~/gopath/src/fabiotest/main.go and then build with go build in that directory. Make sure you have set export GOPATH=~/gopath. When running you can test both endpoints with curl localhost:9998 and curl localhost:9999. Except for the long-polling outgoing connection to consul this is in essence the core of fabio. :)
You can use the following Dockerfile:
FROM scratch
ADD / fabiotest
EXPOSE 9998 9999
CMD ["/fabiotest"]

See how far you get with this.
package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
)

func main() {
	var proxyAddr, uiAddr, proxyURL string
	flag.StringVar(&proxyAddr, "proxy", ":9999", "host:port of the proxy")
	flag.StringVar(&uiAddr, "ui", ":9998", "host:port of the ui")
	flag.StringVar(&proxyURL, "proxyURL", "https://www.google.com/", "proxy url")
	flag.Parse()

	log.Println("fabiotest starting")

	go func() {
		u, err := url.Parse(proxyURL)
		if err != nil {
			log.Fatal("proxyURL:", err)
		}
		log.Println("proxy listening on", proxyAddr, "proxying", u)
		rp := httputil.NewSingleHostReverseProxy(u)
		if err := http.ListenAndServe(proxyAddr, rp); err != nil {
			log.Fatal("proxy:", err)
		}
	}()

	go func() {
		log.Println("UI listening on", uiAddr)
		http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
			fmt.Fprintln(w, "UI is OK")
		})
		if err := http.ListenAndServe(uiAddr, nil); err != nil {
			log.Fatal("ui:", err)
		}
	}()

	log.Println("Press CTRL-C to stop")
	select {}
}

Thanks I'll give this a go and let you know the result. Actually I upgraded to Nomad 0.5.1 just in case but no change.

This issue remains with this test code so as you suspected it is a more generic issue. I actually also tested it with the Consul binary and got the same thing. Thank you for your time on this. I'll head over to the Nomad issues page with it.
As a side note, what should have I been expecting from 'curl localhost:9999'. I am getting a 404.
Also I noticed that you posted a Dockerfile. As the issue was specifically for the exec driver I was not sure what you wanted me to do with it.

You may (or may not :)) be interested to know that this issue is something to do with the kernel version and the LVM storage driver implementation. Haven't quite figured it out but switching to AUFS makes the issue go away.
Thanks again for your attention and a great utility.

I am interested and I'm glad you've figured it out. If you have a reference issue for nomad feel free to link it.
Thanks and merry christmas. Enjoy the holidays.

I raised this up with Kelsey Hightower as it was his demo that made me look at it. Not for a fix but just for some info about his env. I'm not sure I am going to burn time raising it with Nomad as we are way behind here with our kernel revision (one of the many things on my to-do list). It seems to me that LVM may not be the strategic storage option for Docker so I think fixing forward by upgrading is the way to go.


Video.js version 5 support

Hi! yep, a terrifically useful plugin - thanks!!
I've been stumbling around looking for a way to have it work with the latest version of wavesurfer and the just released version 5 master branch of videojs.
If anyone could point out how to get running with the new video.js (version 50 which has lots of responsive features now I believe.

Thanks @tweed! At the time of this writing video.js 5.0 has not been released yet. I'm holding off support for that version until it's actually released and called somewhat stable. I'm planning to keep the current version of this plugin compatible with video.js 4.x in a stable branch (1.x.x) and start a 2.0 version on the trunk that is compatible with video.js 5.
Let's keep this ticket open to track video.js 5 progress.

Sensational - makes great sense - thanks!
So, I went back to version 4 and understood better that the development branch is now version 5 though stable is of course version 4.
had hoped to see the responsive magic in version 4 though appears, at least for my limited skills, only possible in version 5.
All good !!! looking forward, when you have time of course, to version 2 trunk - woohoo! great to know.

I tried to use this with videos 5, and got this exception:
TypeError: undefined is not an object (evaluating 'videos.Component.extend')
On line 16:
videojs.Waveform = videojs.Component.extend({
It looks like videojs changed their component system in videojs 5, unfortunately for plugin developers.

Thanks for testing that @paulhovey. I released a new version where npm and bower will limit video.js to v4.x releases only for now.

Here's a list with details about the changes in v5: https://github.com/videojs/video.js/wiki/5.0-Change-Details

+1
This excellent plugin should work with v5

progress...

Well, looks like it wasn't too much work. If you want to help test (@doublex, @tweed, @paulhovey?), check the videojs5 branch or get the zip: https://github.com/collab-project/videojs-wavesurfer/archive/videojs5.zip. Cheers.

I tried it and it works!  Awesome!  Thanks for this great plugin!

thanks for testing @paulhovey. About to release 1.0.0! Any issues, please open a new ticket.


Debian binary

Hello,
While installing redqueen on an Ubuntu 16 host, is it possible to target a binay that was compiled on a Debian os ?
I am getting some issues with the dependancies, not foung locally when generating the packed binaries (binay_info + binary_fuzz).
Any idea about this ? thank you.

The examples for fuzzing userspace targets compile the target on the host and then load it into the target VM (via initrd or program/agent). But this is only a shortcut for quick install/testing. Normally your target binary is part of the VM image (or kernel or ramdisk) and you only load the agent (loader) to communicate and launch your actual target.
Look at the VM image fuzzing flow to see how to install a custom loader/agent inside a VM image:
https://github.com/IntelLabs/kAFL/blob/master/doc/README.kAFL.md
Note that this guide is a little outdated, you may have to check with the Windows fuzzing tutorial for most recent flow of creating and loading the VM image with kAFL.

The examples for fuzzing userspace targets compile the target on the host and then load it into the target VM (via initrd or program/agent). But this is only a shortcut for quick install/testing. Normally your target binary is part of the VM image (or kernel or ramdisk) and you only load the agent (loader) to communicate and launch your actual target.
Look at the VM image fuzzing flow to see how to install a custom loader/agent inside a VM image:
https://github.com/IntelLabs/kAFL/blob/master/doc/README.kAFL.md
Note that this guide is a little outdated, you may have to check with the Windows fuzzing tutorial for most recent flow of creating and loading the VM image with kAFL.

Thanks for the reply. Actually I see the logic when it comes to drivers but not a given binary. Could you say how to point the target binary from the host to guest while running command "kafl_user_prepare.py" in redqueen to generate the packed binaries ? How to use the loader for this? I do not find anything in the given docs.

Take a look at tests/user_bench, kafl_user_prepare.py is used there as a part of the process in run.sh.

Take a look at tests/user_bench, kafl_user_prepare.py is used there as a part of the process in run.sh.

Yes but this is still does recompile the target on host. I managed to do this with redqueen without problem on lava and other targets. My question is on fuzzing a binary present in the guest VM only (A Debian). I suppose that packing it (to generate binary_info and binary_fuzz) must be done on the guest since the dependencies are not present on the Ubuntu 16 host. May be the solution is not redqueen but kAFL while building a custom loader that lauches the binary in the VM before freezing it.  Will the range address ip0 of the binary appear after runing kafl_info.py in such a scenario ? What do you think?

You can try to download the repo inside the guest and build the info+fuzz initramfs there, then copy them back to the host.
Note that the build scripts in tests/user_bench/ are just a way to get started and document/test the flow. If your binary is similar to the examples they might work. Otherwise you will have to customize the ld_preload forkserver  and/or how your application is loaded inside the initrd.
That forkserver is the only fundamental difference in the two approaches. You could implement your own agent that launches the target application inside the VM, but then it may be difficult to tell the CR3 and -ip0 range of that target process. The forkserver does that for you, and it can also restart the target directly instead of asking kAFL to reload the VM snapshot.

Thanks a lot. I will give it a try following your advice.

I packed the binaries within the VM (needed to deactivated Qemu self_chek by the way) but QEMU is required to run kafl_info.py. So, I moved the resulting binary_info outside and run the same script kafl_info.py on host but the process crashes (C return in Qemu protocol). It is stuck in the __debug_recv_expect() while loop. Do you have an idea about the cause?

This is the guest crashing while trying to execute info, so the issue is still with the agent. You could try to execute the info binary manually within the guest. It is expected to crash at the very end because the hypercall at the very end of ld_preload_info.c will fail outside kAFL. But it seems you are getting a crash before that. The 'C'rash you are seeing is likely one of the panic() calls in userspace_loader.c which packages your target binary together with ld_preload_info.c.


Add support for meta tag helper

Summary
Add handler for meta_tag that understands the intricacies of this HTML markup.
Motivation
It's confusing to write ep using tag meta.
References
Me being all confused by the tag examples, just didn't understand HTML.
https://irclog.perlgeek.de/mojo/2017-04-23#i_14474559

Even if the examples makes sense, I must say I think there are too many special cases to make this tag helper work properly.

:-1: from me as well for same reason.

Guess this also did not attract the required votes from mojo-core.


SQS messages don't contain MessageGroupId

My use-case involves a simple FIFO queue, where the MessageGroupId isn't only an instruction to SQS, but also contains some data that the consumer is also interested in accessing later. (Note that this is not the same kind of "attribute" as the user-land ones using MessageAttributes)
Reproduction script
from moto import mock_sqs
import boto3

with mock_sqs():
    queue = boto3.resource('sqs').create_queue(
        QueueName="test_queue",
        Attributes={
            'FifoQueue ': 'true',
            'ContentBasedDeduplication': 'false'
        }
    )

    queue.send_message(
        MessageBody="{}",
        MessageDeduplicationId="DDID-1",
        MessageGroupId="GID-1",
    )

    messages = queue.receive_messages(
        AttributeNames=['MessageGroupId'],
        MaxNumberOfMessages=10,
        VisibilityTimeout=10,
        WaitTimeSeconds=0,
    )

    msg = messages[0]
    mgid = msg.attributes['MessageGroupId']
    assert "GID-1" == mgid

Fails with:
Traceback (most recent call last):
  File "test.py", line 27, in <module>
    mgid = msg.attributes['MessageGroupId']
KeyError: 'MessageGroupId'

Next steps
I plan to start a PR later which will at least allow the data to flow from producer to consumer.
I am not familiar with the moto codebase, but my initial scan of the code suggests changes would be necessary to:

Inside moto.sqs.models.SQSBackend#send_message pull the data out with self._get_param('MessageGroupId')
Store the value as a property of moto.sqs.models.Message
Put in an XML <Attribute> block in moto.sqs.responses.RECEIVE_MESSAGE_RESPONSE that emits the value from the mode.

I expect the main issues will involve the extent to which the attribute should only be allowed or appear on certain queue-types.
Out of scope
Will not implement actually using the MessageGroupId for selectively ordering messages -- that's a much bigger problem and may be related to work needed for #1438.

Closed with #1580

I noticed in "out of scope":

Will not implement actually using the MessageGroupId for selectively ordering messages

Do you know if this is being tracked? I didn't see it but just checking.

@azhadsyed Moto does have logic to ensure only one message per group is processed simultaneously - is that what you're referring to? https://github.com/getmoto/moto/blob/4ed8b93e182891763d23aa6fe6cacf7e0205a509/moto/sqs/models.py#L916
If there is any other logic still missing, you can just create an new issue for this - that is the only way we track feature requests.

I'm seeing some behavior where Moto isn't exposing messages within the same group in a FIFO manner (e.g. given messages 1, 2, 3, published in that order, message 3 is visible before message 2) ...but either way, I will follow the issue process, thanks for the help!


Invalid JSON primitive during scoop search

I found today whenever I tried to search for a package on Scoop the following error would be produced.
convertfrom-json : Invalid JSON primitive: .
At C:\Users\VictorSong\scoop\apps\scoop\current\lib\manifest.ps1:10 char:45
+     Get-Content $path -raw -Encoding UTF8 | convertfrom-json -ea stop
+                                             ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
    + FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

Is there any way I can diagnose this issue?

Same here

Hi Luke.  First poster here, so bear with me.   I had the same issue and found that the file 'C:\Users<username>\scoop\buckets\extras\bucket\deepgit.json'  had an extra comma at the end of line 26.  Opening the file in VSCode showed it as improperly coded JSON.   Not sure if that's the same issue people are having.  I only started noticing it yesterday.
See https://github.com/lukesampson/scoop-extras/blob/master/bucket/deepgit.json
BTW, thanks for all the work on Scoop!

This is the same as #5201/#5202.  If you retry, it should be OK now.

@pcrama ,  Thank you.   Confirmed the fix made by the deepgit team is working.  The error is no longer showing when doing scoop operations.


Auto login

I like the way that you build/write the playbooks.
But I have a question, why when I login to the jenkins UI it does not ask me for user/password
Regards

Jenkins, sadly, by default does not require a user to log in to see the dashboard and some parts of the Jenkins UI. I usually add plugins like role-strategy and configure Jenkins with different user roles and don't allow the anonymous user to do anything (only logged in users get any kind of access.
Please see https://wiki.jenkins.io/display/JENKINS/Securing+Jenkins for more info.

Had the same issue and fixed it by disabling anonymous access through an Ansible task:
- name: Deny anonymous read access
  lineinfile:
    path: ~/config.xml
    line: "<denyAnonymousReadAccess>true</denyAnonymousReadAccess>"
    regexp: "<denyAnonymousReadAccess>"
  become: true
  become_user: jenkins
  become_method: sudo
  notify: RESTART_JENKINS


Another bugsquish round

Fixes:

Another outstanding bug with the refresh OAuth script, as we've got a user who predates MediaWiki logging user registration dates (#564 again), and it turns out that the reported error is actually a problem.
Rework the request comment storage into the comment table, allowing existing comment editing to take place (Fixes #490 - prerequisite for OAuth fix because database patches...)
Add autocomplete hints to credential fields for better browser interop (fixes #569)
Split of visibility of related requests into a new right, granted to users (fixes #568 - regression from legacy)

Codecov Report

Merging #570 into master will decrease coverage by 0.00%.
The diff coverage is 0.00%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update afc32f2...64e9a06. Read the comment docs.

I specifically would like to test #568 in sandbox for my usability report first before code review. @stwalkerster Could you verify that the sandbox is on this version?

https://accounts-dev.wmflabs.org/rc/internal.php is now running this PR, https://accounts-dev.wmflabs.org/waca/acc.php is still running the legacy tool for comparison.

Another bug squished, which should take us to zero user-reported bugs that I'm aware of in newinternal.
I've updated the RC environment to the tip of this branch for you.

RC environment updated

I still can't test these changes due to the id noticeboard triggering.


Provide proto files for sensor protocol buffers.

Ref: https://github.com/google/science-journal-arduino/blob/master/lib/goosci/sensor.pb.h

Should be available @ https://github.com/google/science-journal/tree/master/OpenScienceJournal/whistlepunk_library/src/main/proto


IO error during test, commit b909d6f73, file modred/tests/testera.py

line 43:
self.data_dir = 'files_ERA'
test with
python -c 'import modred.tests; modred.tests.run()'
reports IOerror:
IOError: files_ERA/A_in1_out1.txt not found.
changed to absolute path solves the problem.
os is ubuntu18.04(python is actually python2.7)

I just pushed a small bug fix.  Can you check for me if this works?  It was not a bug on my Mac or Windows machines, so I cannot verify this myself.  However, I changed the path specification to be more robust, similar to the way it is in the OKID tests, so I suspect this will work.

I just pushed a small bug fix. Can you check for me if this works? It was not a bug on my Mac or Windows machines, so I cannot verify this myself. However, I changed the path specification to be more robust, similar to the way it is in the OKID tests, so I suspect this will work.

Problem solved, nice work and thanks a lot!


Everything is in Japanese?

When I opened my app today after the latest patch, all of my bugs and fish are named in what I believe is Japanese. My phone language is Canadian English / Canada for location.

Edit: Now looking at the writing closer as opposed to just the update it seems like Japanese. Have attached a photo, not sure if it matters.

Yep, we know :(. Fix is in review, see: https://github.com/Dimillian/ACHNBrowserUI/issues/141

Thank you, sorry to bother you with another issue, I didn’t see that!

@reflectviolet no worries :), hope Apple will accept the fix soon 🤞


[Bug Report] Error document el-upload  attribute show-upload-list not found.

Element UI version
1.4.2
OS/Browsers version
chrome
Vue version
2.4.2
Reproduction Link
https://jsfiddle.net/hoanghiep/cwucaq61/
Steps to reproduce
document write show-upload-list set value true and false.
in when using is :show-file-list="false"
What is Expected?
fix or write document
What is actually happening?
fix or write document Attributes show show-upload-list

Fixed in #6567


Investigate layer rasterization performance improvements

Curious to know whether setting
layer.shouldRasterize = true
layer.rasterizationScale = UIScreen.main.scale

on the swipe cards provides any performance improvements. Since the overlay views animate their alpha value, the answer is not cut-and-dry to me. Maybe set it only on certain subviews?Should profile with Core Animation instrument.

Decided to table this issue. The dropped FPS is only visible on simulators. On multiple real iOS devices, the Core Animation profiler reached 60 FPS under constant swiping. Can reopen this if necessary

Some great resources:

https://www.toptal.com/ios/ios-animation-and-tuning-for-efficiency
https://yalantis.com/blog/mastering-uikit-performance/

Reopening since rasterization should be enabled by default on the footer and content to avoid jagged edges during dragging


Jdenticon is shown only after refreshing the page.

So, when at first launch the jdenticon area is just white< but if I refresh the page image appears. I have even tried to use jdenticon.update(), but it doesn't help.
I attached file showing debugging info.
What information can I provide to clarify the problem?
Thank you

And thank you for such a gret lib!:)

You're welcome :) Hard to tell the cause without knowing the context in which Jdenticon is used. I don't see any problems in the screenshot, so I have some questions.

Do you use the canvas or svg approach?
Are the svg/canvas elements delivered as static html or generated dynamically by using plain Javascript or a framework like Angular?
If using a framework, which one do you use?
Is the issue isolated to a single browser?

Regarding question 2 icons won't be automatically generated for DOM elements inserted after the initial page load.

Thanks for your reply!

I use canvas.
2-3. Canvas elements are in templates which are used in different states(I am using Angular)
The issue is isolated to Google Chrome. In Edge icons are never generetated even after refreshing the page.

So is there any way to use Jdenticons with Angular?

The problem is that jdenticon.update is never called for canvases created dynamically as by Angular. A solution could be to create a directive that is responsible for calling jdenticon.update when the canvas element is constructed.
See this fiddle for an example:
https://jsfiddle.net/w5h6msvd/
Seems to work in Edge, IE, Chrome, FF and Opera.

Hey, I have used this:
$('canvas').initialize( function(){
  jdenticon.update('.settings-avatar');
  jdenticon.update('.userpic');
});

and it worked:) Thank you

I am facing a similar problem while using Jidenticon with ReactJS. On state changes the icon just does not come up and update() does not work properly


Reference or incorporate PyRTM

PyRTM contains functional python wrappers for SMARTS and SBdart, and is capable of modelling spectra in clear-sky and cloudy conditions. This would be a great set of functionality to incorporate into PVLIB.  It could be a good project to either integrate the whole code base into the pvlib project (I was involved in the PyRTM project, and I think this would be great). Or, it might we worthwhile to pull in  a set of functions to reference it in PVLIB, as PyRTM is already up on pip.
@uniphil any thoughts?

Looks interesting. Some thoughts on PyRTM and the broader topic:
I would like to understand some of the pluses and minuses of the many models on https://en.wikipedia.org/wiki/Atmospheric_radiative_transfer_codes for our application. Can anybody point me to some good literature? @Calama-Consulting's referenced paper says that SBDART is simple. @NelisW uses MODTRAN, which does not appear to be open source. RRTMG is now fairly standard in my field, though I don't personally have experience with it. SMARTS seems popular.
 On the other hand, we have an opportunity to help lead the PV community, or at least set some good examples and make some good default choices for people like me that don't already know the literature.
Many of these tools are very general-purposed, and we should keep our focus on the application of these tools to PV. So, I think wrappers are generally the way to go, but I also don't want to introduce dependencies to projects that are no longer maintained. We also need to consider ease of installation. For example, PyRTM itself may be trivial to install, but what about its dependencies? Can we use conda to bundle the whole thing?
Might be wrong about this, but I think that since PyRTM is GNU-licensed you'd have to get permission from all contributors to pull any part of it into pvlib-python.

Here is the selection process that we went through, as I remember it. I'll say up front that this wasn't an exhaustive search, but there was some rationale behind it:

SMARTS widely used in the solar industry, and well validated. Downside is that it does not model clouds

MODTRAN very heavily validated, but not open source. This is the tool against which all other tools are measured. The libraries that @NelisW uses looks like they do a great job of reading in the outputs of MODTRANS, but it isn't likely to get integrated fully into a workflow

SBDART this tool has the added advantage of being able to model various forms of clouds, and is validated against MODTRAN http://www.arm.gov/publications/tech_reports/handbooks/rss/publications/ASES2006_Gueymard.pdf

RRTM we evaluated this, it seemed to be a very versatile model, however it seemed like it might be too skewed towards atmospheric science and so might be a bit bulky/hard to use. That being said, that's not something that a nice clean wrapper can't fix.

The license for SBDART is a bit ambiguous, the code  is available at a few locations online, but I'm not sure what the actual license is for the code. The choice of SMARTS and SBDART was made because they seemed to be a good balance between utility, size, accuracy and liscencing.
As for the liscence of PYRTM, the creators are myself and @uniphil, and I believe we would be open to modifying the terms to whatever makes the code most useful.

I agree with @Calama-Consulting on Modtran. It is somewhat expensive, hard to use, HUGE, and the licence prohibits secondary distribution in any form.  My idea would be to do what SMARTS already seems to do: package reference atmospheres (which I could pre-calculate using Modtran). But since SMARTS already do this, there is little point in doing this.  It would make much more sense to integrate SBDART, SMARTS or PYRTM with pvlib.  At this point I know too little about any of these tools talk any sense (clearly @Calama-Consulting knows a lot more).  Give me a while to digest and then I will come back to you.

I stumbled across this PyRRTM package but I haven't looked at it closely: https://github.com/tomflannaghan/pyrrtm

This Gueymard and Ruiz-Arias article might be of interest to people that come across this issue or #101.

@wholmgren, it's actually this one from Queens-Applied-Sustainability PyRTM and this wiki. Requires Chris Gueyrmard's SMARTS model which is easy to compile. I used gfortran from msys-2 on windows, so I'm sure you can do it in linux or mac as well.
RRTM from Atmospheric & Environmental Research (AER)
Radiative Transfer Working Group is rapid radiative xfer model, but not a spectral model, probably broadband, used in forecasting models? Not sure, shoulkd shoot @tomflannaghan a message.
Watch out lots of irrelevant pyrtms (remember the milk) on the cheeseboard.

LibRadtran is another RTM that is an all weather (clouds) model. @aoumbe was involved so maybe he can say more about. it.

Also Gueymard has a new broadband all weather model called REST2.

As far as references go, Josh Stein PVPMC site at Sandia has a section on irradiance modeling and NREL RREDC has the BIRD clear sky model, SPECTRL2 and the DISC all weather RTM model on which Perez's DIRINT and DIRINDEX models are based.

RRTM is used widely in forecasting/climate models. It's a correlated-k model (see this paper for example), which is calibrated using a line-by-line model that has been validated with observations. As far as I'm aware it is very accurate. I used the model for some fairly theoretical work with no clouds or aerosols, but the model does support these and is very comprehensive.
The python wrapper I wrote (which is pyrrtm as mentioned above) might be a useful starting point for someone who wanted to call the model from python. However, I did write it with my more theoretical use in mind so doesn't currently support clouds or aerosols. Adding these would probably be quite straightforward.


Fix flaky test - TestSpanProcessorWithOnDroppedSpanOption

Which problem is this PR solving?
Resolves #4450
Short description of the changes
Due to concurrency, it is not guaranteed that which one of the second span and third span will be dropped in the test or both could be dropped. But it is guaranteed that at least one of them will be dropped.
Update the test to set a boolean flag when any span is dropped.
Welcome to suggestion!

closing to minimize noise, feel free to reopen if comments are addressed


Wrong gradients in combination with placeholders.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 14.04.5 LTS, Debian 4.18.10-2, Archlinux
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v1.4.0-19-ga52c8d9 1.4.1, v1.10.1-0-g4dcfddc5d1 1.10.1, 1.10.0,
Python version: 3.6.0, 3.4.3, 3.7.0
CUDA/cuDNN version: 9.0/7.1.2, 8.0.61/6.0.21, CPU only
GPU model and memory: Titan 12GB

Describe the current behavior
Incorrect gradient computation when placeholder weights are involved. In a situation where two variables are involved, optimization of one of the variables leads to wrong gradients for the other variable when placeholders weights are used.
Describe the expected behavior
Gradients should be correct and there should be no difference in gradients whether a placeholder or another variable is used.
Code to reproduce the issue
Here is a minimal example to reproduce the issue (the case of indirect = True can be used as a workaround and shows the expected output in contrast of indirect = False):
import tensorflow as tf

def test(indirect):
    print("indirect:", indirect)
    a = tf.Variable(1.0)
    b = tf.Variable(2.0)
    weight_var = tf.Variable(3.0, trainable = False)

    weight_placeholder = tf.placeholder(tf.float32)
    if indirect:
        weight = tf.assign(weight_var, weight_placeholder)
    else:
        weight = weight_placeholder

    common_loss = tf.square(a*b)
    b_only_loss = tf.square(b)

    common_grads = tf.gradients(
            common_loss,
            [b])
    b_only_grads = tf.gradients(
            weight*b_only_loss,
            [b])
    summed_grads = tf.gradients(
            common_loss + weight*b_only_loss,
            [b])
    grad_diff = tf.abs(common_grads[0] + b_only_grads[0] - summed_grads[0])

    optimizer = tf.train.GradientDescentOptimizer(learning_rate=1.0)
    opt_a = optimizer.minimize(common_loss, var_list=[a])

    feed={weight_placeholder: 0.0}
    fetch={"common_grads": common_grads, "b_only_grads": b_only_grads,
           "summed_grads": summed_grads, "grad_diff": grad_diff,
           "opt_a": opt_a}
    s = tf.Session()
    s.run(tf.global_variables_initializer())
    r = s.run(fetch, feed)
    for k in sorted(r):
        print("{:16}".format(k), r[k])
    assert r["grad_diff"] < 1e-11, r["grad_diff"]

if __name__ == "__main__":
    print("tensorflow", tf.GIT_VERSION, tf.VERSION)
    test(True)
    test(False)

Example output:
tensorflow b'unknown' 1.10.0
indirect: True
b_only_grads     [0.0]
common_grads     [4.0]
grad_diff        0.0
opt_a            None
summed_grads     [4.0]
indirect: False
b_only_grads     [0.0]
common_grads     [4.0]
grad_diff        32.0
opt_a            None
summed_grads     [-28.0]
Traceback (most recent call last):
  File "minimal_example.py", line 47, in <module>
    test(False)
  File "minimal_example.py", line 41, in test
    assert r["grad_diff"] < 1e-11, r["grad_diff"]
AssertionError: 32.0

Other info / logs
The output is not deterministic and sometimes produces correct results so the code might have to be run multiple times. The code also works fine without the opt_a call. Constant values for the weight also work fine, just in combination with a placeholder it seems to be problematic. The problem remains the same for other values of the weight and thus in general violates commutativity of the gradient but the case of a weight of zero illustrates the problem best because the zero weighted summand really should not have any influence on the gradients. Finally, the gradients are not just reported falsely but optimization of b, e.g. optimizer.minimize(common_loss + weight*b_only_loss, var_list=[b]), produces wrong results (i.e. gradient descent uses wrong gradients).

Nagging Assignee @skye: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @skye: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@pesser,
Can you please let us know if we can close this issue, as the Placeholders are not relevant in Tensorflow Versions, 2.x? Thanks!

The issue persists with recent versions and tf.compat.v1 but it can be closed from my side since I'm not relying on it.

@pesser,
Your code could be executed successfully without any Assertion Error. Please find the Gist of the working code. Thanks!


Update docker version

Recently CVE of docker engine is reported.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5736
Since it has been a long time for OpenWhisk to use docker-1.12, I think it would be a good time to update docker version to the latest one not only to deal with the CVE but also to take advantage of the latest docker.
The current latest docker version is 18.09.3.
For easier maintenance and better interoperability, it would be great to update both the server and the client.
(ex: https://github.com/apache/incubator-openwhisk/blob/master/core/invoker/Dockerfile#L8)
Since it might induce some fundamental changes in docker API, it can be a breaking change.

closing as #4430 is merged


Out of memory while downloading large files

While trying to download a large .zip file (1.3GB) in a test VM with 512MB of RAM the only output I got from the puppet run was Killed. I increased the VM RAM incrementally and got Error: Could not run: failed to allocate memory until I had increased the RAM to 2GB, i.e. the file could be stored in RAM as a whole until flushed to disk.
Is this a known unfixable issue? Or an issue with the provider implementation? Or is there some workaround that can be applied?

Are you pulling the file over http or ftp?

Ah, sorry I left that out. This is a HTTP Basic Auth pull.

Looking a bit at the code, this looks like an issue in the faraday library. However I dont see an issue on the faraday bug tracker for it. I'll try to find some time to bang on it today.

Thanks for looking into this. Other potentially important information.
Puppet 3.7.5
CentOS 6
and therefore Ruby 1.8.7

I am able to reproduce the problem at least.

In this case might as well swap out faraday with net/http and eliminate a dependency. I'll take a crack at it over next few days.

@nanliu do you think its in faraday? Is the typical pattern in puppet to hold in ram before file flushing? I'm just not experienced enough to know.

I thought this was one of things I verified early on, but clearly missed it when evaluating faraday. Per their roadmap: https://github.com/lostisland/faraday/wiki/Roadmap and open PR: https://github.com/lostisland/faraday/pull/156, I don't think we are able to do streaming with faraday. net/http will require some code to handle http errorcodes and redirect, but will support streaming.

Since it looks like the Faraday PR is stalled, should we move to net/http or similar?

Yes, we should switch to net/http as default provider.


Allow cash out to be greater than total buy-in

Allow cash out to be greater than total buy-in since, which can occur when

Game winners cash out before everyone has bought in
There has been some user error

Hello @MatthewUng! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file src/game/route.py:

Line 217:80: E501 line too long (84 > 79 characters)
Line 221:80: E501 line too long (113 > 79 characters)


App packaged with wap project builds for the wrong architecture

Describe the bug
A C# app (project type "Packaged with WAP (WinUI3 in Desktop)")) which is build on a x64 machine does not build arm64 binaries even if the arm64 architecture is selected in the dropdown menu.
Steps to reproduce the bug

Create a new blank c# app packaged with a WAP project
Select arm64 as the target architecture
Build the project itself or the WAP package
The generated binaries are x64 instead of arm64

Adding the PublishProfile property to the app project seems to fix this issue
Expected behavior
App should be built according to the selected architecture
Screenshots
No response
NuGet package version
1.0.0
Packaging type
Packaged (MSIX), Unpackaged
Windows version
Windows 11 (22000)
IDE
Visual Studio 2022
Additional context
No response

Close this issue as not repro.


Conditional Graph Generation Issue

Hi, I met a question about the regressor model. I think the model does not correctly performed the conditional generation.
The error for this command python3 src/guidance/train_qm9_regressor.py +experiment=regressor_model.yaml happens at the dimension of graph labels.
Here is the error.
target: tensor([], size=(512, 0))
Shape of pred.y: torch.Size([512, 1])
Shape of target: torch.Size([512, 0])
Can you please debug it?

Hello, can you make sure that you reprocessed the dataset, as specified in the README of the branch?
Thank you

On 27 Sep 2023, at 04:07, yikang613 @.***> wrote:
Hi, I met a question about the regressor model. I think the model does not correctly performed the conditional generation.
The error for this command python3 src/guidance/train_qm9_regressor.py +experiment=regressor_model.yaml happens at the dimension of graph labels.
Here is the error.
target: tensor([], size=(512, 0))
Shape of pred.y: torch.Size([512, 1])
Shape of target: torch.Size([512, 0])
Can you please debug it?
—
Reply to this email directly, view it on GitHub https://github.com/cvignac/DiGress/issues/68, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEJOOTW5T6YDNV2JAWZ72ZLX4OC47ANCNFSM6AAAAAA5ITIX3Y.
You are receiving this because you are subscribed to this thread.

I have the same problem, can you describe the solution in more detail?

Hello, can you make sure that you reprocessed the dataset, as specified in the README of the branch? Thank you
…
On 27 Sep 2023, at 04:07, yikang613 @.***> wrote: Hi, I met a question about the regressor model. I think the model does not correctly performed the conditional generation. The error for this command python3 src/guidance/train_qm9_regressor.py +experiment=regressor_model.yaml happens at the dimension of graph labels. Here is the error. target: tensor([], size=(512, 0)) Shape of pred.y: torch.Size([512, 1]) Shape of target: torch.Size([512, 0]) Can you please debug it? — Reply to this email directly, view it on GitHub <#68>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEJOOTW5T6YDNV2JAWZ72ZLX4OC47ANCNFSM6AAAAAA5ITIX3Y. You are receiving this because you are subscribed to this thread.
I

I met a similar problem for the guided generation. My solution is to denote the $target$ as $guidance_target$ in the file  "regressor_model.yaml". I think Clement make a mistake here.
In fact the problem is that the config file "regressor_model.yaml" denote the $target$ while the "qm9_dataset.py" denote it as $cfg.general.guidance_target$

Hi, I met a question about the regressor model. I think the model does not correctly performed the conditional generation. The error for this command python3 src/guidance/train_qm9_regressor.py +experiment=regressor_model.yaml happens at the dimension of graph labels. Here is the error. target: tensor([], size=(512, 0)) Shape of pred.y: torch.Size([512, 1]) Shape of target: torch.Size([512, 0]) Can you please debug it?

Hi, I can't find the file:src/guidance/train_qm9_regressor.py

I met the same problem when I run "python3 src/guidance/train_qm9_regressor.py +experiment=regressor_model.yaml".
RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([512, 1]) and torch.Size([512, 0]).


⚠ Bofrost.de has degraded performance

In 4166233, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 962 ms

Resolved: Bofrost.de performance has improved in f54d4a7 after 29 minutes.


Rails project should run bundle before build image

File: compose/rails.md, CC @londoncalling
In section "Build the project", if use "--skip-bundle" when create project, Gemfile.lock will be empty. Event if build image in the next step, because volume do not mount in build process, changes in Gemfile.lock will not sync to host. However Gemfile.lock should be check into version control (http://bundler.io/v1.14/man/bundle-install.1.html#THE-GEMFILE-LOCK).
My opinion is do not use "--skip-bundle" when create project:
docker-compose run web rails new . --force --database=postgresql

If edit Gemfile later, we should run bundle first in order to update Gemfile.lock:
docker-compose run web bundle

Then rebuild image:
docker-compose build

@chloerei I've run through this quickstart again both ways, from a clean project directory with no other containers or images present at the start of the test:

first with commands as shown (using --skip-bundle in the docker-compose run command)
then in a new project, leaving off --skip-bundle per your suggestion.

Both work, and in both cases there is content in Gemfile.lock after the first docker-compose run .... The only difference I see is that if I leave off --skip-bundle, the first build goes a little faster.
I did see some clarifications I should add about taking the app down, and finding the web page on Docker for Mac and Windows, so I'll make those changes, but I don't think the we need to remove --skip-bundle from the command
cc: @shin- @dnephin @nathanleclaire
For reference, here are results of the first most relevant steps, following the procedure as written, which includes --skip-bundle:
Create files
Create the files per the steps at define the project in the Quickstart: Compose and Rails.

Building dependency tree...
Reading state information...
The following extra packages will be installed:
  dpkg-dev fakeroot libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libc-ares2 libdpkg-perl libfakeroot
  libfile-fcntllock-perl libpq5 libtimedate-perl libv8-3.14.5
Suggested packages:
  debian-keyring postgresql-doc-9.4
The following NEW packages will be installed:
  build-essential dpkg-dev fakeroot libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libc-ares2 libdpkg-perl
  libfakeroot libfile-fcntllock-perl libtimedate-perl libv8-3.14.5 nodejs
The following packages will be upgraded:
  libpq-dev libpq5
2 upgraded, 13 newly installed, 0 to remove and 35 not upgraded.
Need to get 5197 kB of archives.
After this operation, 12.5 MB of additional disk space will be used.

Preparing to unpack .../libpq-dev_9.4.12-0+deb8u1_amd64.deb ...
Unpacking libpq-dev (9.4.12-0+deb8u1) over (9.4.10-0+deb8u1) ...
Preparing to unpack .../libpq5_9.4.12-0+deb8u1_amd64.deb ...
Unpacking libpq5:amd64 (9.4.12-0+deb8u1) over (9.4.10-0+deb8u1) ...
Selecting previously unselected package libc-ares2:amd64.
Preparing to unpack .../libc-ares2_1.10.0-2+deb8u1_amd64.deb ...
Unpacking libc-ares2:amd64 (1.10.0-2+deb8u1) ...
Selecting previously unselected package libtimedate-perl.
Preparing to unpack .../libtimedate-perl_2.3000-2_all.deb ...
Unpacking libtimedate-perl (2.3000-2) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../libdpkg-perl_1.17.27_all.deb ...
Unpacking libdpkg-perl (1.17.27) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../dpkg-dev_1.17.27_all.deb ...
Unpacking dpkg-dev (1.17.27) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../build-essential_11.7_amd64.deb ...
Unpacking build-essential (11.7) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../libfakeroot_1.20.2-1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.20.2-1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../fakeroot_1.20.2-1_amd64.deb ...
Unpacking fakeroot (1.20.2-1) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../libalgorithm-diff-perl_1.19.02-3_all.deb ...
Unpacking libalgorithm-diff-perl (1.19.02-3) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../libalgorithm-diff-xs-perl_0.04-3+b1_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-3+b1) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../libalgorithm-merge-perl_0.08-2_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-2) ...
Selecting previously unselected package libfile-fcntllock-perl.
Preparing to unpack .../libfile-fcntllock-perl_0.22-1+b1_amd64.deb ...
Unpacking libfile-fcntllock-perl (0.22-1+b1) ...
Selecting previously unselected package libv8-3.14.5.
Preparing to unpack .../libv8-3.14.5_<IP_ADDRESS>-8.1_amd64.deb ...
Unpacking libv8-3.14.5 (<IP_ADDRESS>-8.1) ...
Selecting previously unselected package nodejs.
Preparing to unpack .../nodejs_0.10.29~dfsg-2_amd64.deb ...
Unpacking nodejs (0.10.29~dfsg-2) ...
Setting up libpq5:amd64 (9.4.12-0+deb8u1) ...
Setting up libpq-dev (9.4.12-0+deb8u1) ...
Setting up libc-ares2:amd64 (1.10.0-2+deb8u1) ...
Setting up libtimedate-perl (2.3000-2) ...
Setting up libdpkg-perl (1.17.27) ...
Setting up dpkg-dev (1.17.27) ...
Setting up build-essential (11.7) ...
Setting up libfakeroot:amd64 (1.20.2-1) ...
Setting up fakeroot (1.20.2-1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
Setting up libalgorithm-diff-perl (1.19.02-3) ...
Setting up libalgorithm-diff-xs-perl (0.04-3+b1) ...
Setting up libalgorithm-merge-perl (0.08-2) ...
Setting up libfile-fcntllock-perl (0.22-1+b1) ...
Setting up libv8-3.14.5 (<IP_ADDRESS>-8.1) ...
Setting up nodejs (0.10.29~dfsg-2) ...
update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode
Processing triggers for libc-bin (2.19-18+deb8u7) ...
 ---> cd9e26202411
Removing intermediate container 9378c9947b82
Step 3/8 : RUN mkdir /myapp
 ---> Running in 17e38dbff8e6
 ---> f1568ab29bba
Removing intermediate container 17e38dbff8e6
Step 4/8 : WORKDIR /myapp
 ---> f9e89880199f
Removing intermediate container 50fad74217ad
Step 5/8 : ADD Gemfile /myapp/Gemfile
 ---> a3d24c2ecb2b
Removing intermediate container 20b7df6ee1fd
Step 6/8 : ADD Gemfile.lock /myapp/Gemfile.lock
 ---> d3b1c82537a7
Removing intermediate container 2201ed8f9882
Step 7/8 : RUN bundle install
 ---> Running in 0a84cd6f3c19
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies...
Installing rake 12.0.0
Installing concurrent-ruby 1.0.5
Installing i18n 0.8.1
Installing minitest 5.10.2
Installing thread_safe 0.3.6
Installing builder 3.2.3
Installing erubis 2.7.0
Installing mini_portile2 2.1.0
Installing rack 2.0.3
Installing nio4r 1.2.1 with native extensions
Installing websocket-extensions 0.1.2
Installing mime-types-data 3.2016.0521
Installing arel 7.1.4
Using bundler 1.14.6
Installing method_source 0.8.2
Installing thor 0.19.4
Installing tzinfo 1.2.3
Installing nokogiri 1.7.2 with native extensions
Installing rack-test 0.6.3
Installing sprockets 3.7.1
Installing websocket-driver 0.6.5 with native extensions
Installing mime-types 3.1
Installing activesupport <IP_ADDRESS>
Installing loofah 2.0.3
Installing mail 2.6.5
Installing rails-dom-testing 2.0.3
Installing globalid 0.4.0
Installing activemodel <IP_ADDRESS>
Installing rails-html-sanitizer 1.0.3
Installing activejob <IP_ADDRESS>
Installing activerecord <IP_ADDRESS>
Installing actionview <IP_ADDRESS>
Installing actionpack <IP_ADDRESS>
Installing actioncable <IP_ADDRESS>
Installing actionmailer <IP_ADDRESS>
Installing railties <IP_ADDRESS>
Installing sprockets-rails 3.2.0
Installing rails <IP_ADDRESS>
Bundle complete! 1 Gemfile dependency, 38 gems now installed.
Bundled gems are installed into /usr/local/bundle.
 ---> c3fd68814880
Removing intermediate container 0a84cd6f3c19
Step 8/8 : ADD . /myapp
 ---> dc7d905c461e
Removing intermediate container 32d12a87edc7
Successfully built dc7d905c461e
Successfully tagged railscompose_web:latest
WARNING: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
       exist
      create  README.md
      create  Rakefile
      create  config.ru
      create  .gitignore
       force  Gemfile
      create  app
      create  app/assets/config/manifest.js
      create  app/assets/javascripts/application.js
      create  app/assets/javascripts/cable.js
      create  app/assets/stylesheets/application.css
      create  app/channels/application_cable/channel.rb
      create  app/channels/application_cable/connection.rb
      create  app/controllers/application_controller.rb
      create  app/helpers/application_helper.rb
      create  app/jobs/application_job.rb
      create  app/mailers/application_mailer.rb
      create  app/models/application_record.rb
      create  app/views/layouts/application.html.erb
      create  app/views/layouts/mailer.html.erb
      create  app/views/layouts/mailer.text.erb
      create  app/assets/images/.keep
      create  app/assets/javascripts/channels
      create  app/assets/javascripts/channels/.keep
      create  app/controllers/concerns/.keep
      create  app/models/concerns/.keep
      create  bin
      create  bin/bundle
      create  bin/rails
      create  bin/rake
      create  bin/setup
      create  bin/update
      create  config
      create  config/routes.rb
      create  config/application.rb
      create  config/environment.rb
      create  config/secrets.yml
      create  config/cable.yml
      create  config/puma.rb
      create  config/spring.rb
      create  config/environments
      create  config/environments/development.rb
      create  config/environments/production.rb
      create  config/environments/test.rb
      create  config/initializers
      create  config/initializers/application_controller_renderer.rb
      create  config/initializers/assets.rb
      create  config/initializers/backtrace_silencers.rb
      create  config/initializers/cookies_serializer.rb
      create  config/initializers/cors.rb
      create  config/initializers/filter_parameter_logging.rb
      create  config/initializers/inflections.rb
      create  config/initializers/mime_types.rb
      create  config/initializers/new_framework_defaults.rb
      create  config/initializers/session_store.rb
      create  config/initializers/wrap_parameters.rb
      create  config/locales
      create  config/locales/en.yml
      create  config/boot.rb
      create  config/database.yml
      create  db
      create  db/seeds.rb
      create  lib
      create  lib/tasks
      create  lib/tasks/.keep
      create  lib/assets
      create  lib/assets/.keep
      create  log
      create  log/.keep
      create  public
      create  public/404.html
      create  public/422.html
      create  public/500.html
      create  public/apple-touch-icon-precomposed.png
      create  public/apple-touch-icon.png
      create  public/favicon.ico
      create  public/robots.txt
      create  test/fixtures
      create  test/fixtures/.keep
      create  test/fixtures/files
      create  test/fixtures/files/.keep
      create  test/controllers
      create  test/controllers/.keep
      create  test/mailers
      create  test/mailers/.keep
      create  test/models
      create  test/models/.keep
      create  test/helpers
      create  test/helpers/.keep
      create  test/integration
      create  test/integration/.keep
      create  test/test_helper.rb
      create  tmp
      create  tmp/.keep
      create  tmp/cache
      create  tmp/cache/assets
      create  vendor/assets/javascripts
      create  vendor/assets/javascripts/.keep
      create  vendor/assets/stylesheets
      create  vendor/assets/stylesheets/.keep
      remove  config/initializers/cors.rb

List files
$ ls -l
total 56
-rw-r--r--   1 vmb  staff   223 May 18 14:26 Dockerfile
-rw-r--r--   1 vmb  staff  1738 May 18 14:29 Gemfile
-rw-r--r--   1 vmb  staff  2764 May 18 14:29 Gemfile.lock
-rw-r--r--   1 vmb  staff   374 May 18 14:29 README.md
-rw-r--r--   1 vmb  staff   227 May 18 14:29 Rakefile
drwxr-xr-x  10 vmb  staff   340 May 18 14:29 app
drwxr-xr-x   7 vmb  staff   238 May 18 14:29 bin
drwxr-xr-x  14 vmb  staff   476 May 18 14:29 config
-rw-r--r--   1 vmb  staff   130 May 18 14:29 config.ru
drwxr-xr-x   3 vmb  staff   102 May 18 14:29 db
-rw-r--r--   1 vmb  staff   211 May 18 14:27 docker-compose.yml
drwxr-xr-x   4 vmb  staff   136 May 18 14:29 lib
drwxr-xr-x   3 vmb  staff   102 May 18 14:29 log
drwxr-xr-x   9 vmb  staff   306 May 18 14:29 public
drwxr-xr-x   9 vmb  staff   306 May 18 14:29 test
drwxr-xr-x   4 vmb  staff   136 May 18 14:29 tmp
drwxr-xr-x   3 vmb  staff   102 May 18 14:29 vendor

cat Gemfile.lock
vmb at snap in ~/sandbox/rails-compose
$ cat Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    actioncable (<IP_ADDRESS>)
      actionpack (= <IP_ADDRESS>)
      nio4r (~> 1.2)
      websocket-driver (~> 0.6.1)
    actionmailer (<IP_ADDRESS>)
      actionpack (= <IP_ADDRESS>)
      actionview (= <IP_ADDRESS>)
      activejob (= <IP_ADDRESS>)
      mail (~> 2.5, >= 2.5.4)
      rails-dom-testing (~> 2.0)
    actionpack (<IP_ADDRESS>)
      actionview (= <IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
      rack (~> 2.0)
      rack-test (~> 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.0.2)
    actionview (<IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
      builder (~> 3.1)
      erubis (~> 2.7.0)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.0.2)
    activejob (<IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
      globalid (>= 0.3.6)
    activemodel (<IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
    activerecord (<IP_ADDRESS>)
      activemodel (= <IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
      arel (~> 7.0)
    activesupport (<IP_ADDRESS>)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (~> 0.7)
      minitest (~> 5.1)
      tzinfo (~> 1.1)
    arel (7.1.4)
    builder (3.2.3)
    concurrent-ruby (1.0.5)
    erubis (2.7.0)
    globalid (0.4.0)
      activesupport (>= 4.2.0)
    i18n (0.8.1)
    loofah (2.0.3)
      nokogiri (>= 1.5.9)
    mail (2.6.5)
      mime-types (>= 1.16, < 4)
    method_source (0.8.2)
    mime-types (3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2016.0521)
    mini_portile2 (2.1.0)
    minitest (5.10.2)
    nio4r (1.2.1)
    nokogiri (1.7.2)
      mini_portile2 (~> 2.1.0)
    rack (2.0.3)
    rack-test (0.6.3)
      rack (>= 1.0)
    rails (<IP_ADDRESS>)
      actioncable (= <IP_ADDRESS>)
      actionmailer (= <IP_ADDRESS>)
      actionpack (= <IP_ADDRESS>)
      actionview (= <IP_ADDRESS>)
      activejob (= <IP_ADDRESS>)
      activemodel (= <IP_ADDRESS>)
      activerecord (= <IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
      bundler (>= 1.3.0, < 2.0)
      railties (= <IP_ADDRESS>)
      sprockets-rails (>= 2.0.0)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.0.3)
      loofah (~> 2.0)
    railties (<IP_ADDRESS>)
      actionpack (= <IP_ADDRESS>)
      activesupport (= <IP_ADDRESS>)
      method_source
      rake (>= 0.8.7)
      thor (>= 0.18.1, < 2.0)
    rake (12.0.0)
    sprockets (3.7.1)
      concurrent-ruby (~> 1.0)
      rack (> 1, < 3)
    sprockets-rails (3.2.0)
      actionpack (>= 4.0)
      activesupport (>= 4.0)
      sprockets (>= 3.0.0)
    thor (0.19.4)
    thread_safe (0.3.6)
    tzinfo (1.2.3)
      thread_safe (~> 0.1)
    websocket-driver (0.6.5)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.2)

PLATFORMS
  ruby

DEPENDENCIES
  rails (= <IP_ADDRESS>)

BUNDLED WITH
   1.14.6

From here continue with docker-compose build and rest of steps
vicky at snap in ~/sandbox/rails-compose
$ docker-compose build
db uses an image, skipping
Building web
Step 1/8 : FROM ruby:2.3.3
 ---> 0e1db669d557
Step 2/8 : RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs
 ---> Using cache
 ---> cd9e26202411
. . .
Installing web-console 3.5.1
Installing rails <IP_ADDRESS>
Installing sass-rails 5.0.6
Bundle complete! 15 Gemfile dependencies, 62 gems now installed.
Bundled gems are installed into /usr/local/bundle.
 ---> 27a7861f7101
Removing intermediate container 994aca60315c
Step 8/8 : ADD . /myapp
 ---> 0e8190956241
Removing intermediate container cad2fe295c07
Successfully built 0e8190956241
Successfully tagged railscompose_web:latest
. . .

I found the Gemfile.lock content in your post only have one denpendencies:
DEPENDENCIES
  rails (= <IP_ADDRESS>)

That is base on the initial Gemfile, not create by rails new. In detail, the Gemfile.lock is update when rails command setup, before rails new generate files. So although set --skip-bundle, it still write Gemfile.lock.
But the content in Gemfile.lock is not match the Gemfile generate by rails new, and it will not update by docker-compose build, you can see at the last of your test, Gemfile.lock is same with previous.
Until we run any rails command in container, like docker-compose up, then rails will update and sync Gemfile.lock to host, then it is match current Gemfile.
This behavior is Implicit, and will cause problems in some situations. For example, edit Gemfile, build image, but not run any rails command in container, the Gemfile.lock will not update. Build image on other machine with same source code, it maybe not install same version gems because Gemfile.lock is outdate, and bundle resolve dependencies again.
So I recommend rails new with bundle and run docker-compose run web bundle install after edit Gemfile, explicit update Gemfile.lock.

Okay @chloerei that makes sense.
I'm also making changes to this to run it with docker stack deploy per suggestion from @ManoMarks. I'm working on a PR for it that I haven't submitted yet, so it will contain all these changes.
If I understand correctly, then, we'll start with:
docker-compose run web rails new . --force --database=postgresql

At the end, when we are explaining steps for edit and rebuild, run:
docker-compose run web bundle install

Then, after that, docker-compose up. Does that sound right?
@ManoMarks @shin- @dnephin feel free to jump in :-)

After (1)
docker-compose run web bundle install

Should run (2)
docker-compose build

Then (3)
docker-compose up

Because (1) update Gemfile.lock to host, but do not install gems in image. If skip to (3), it will raise Bundler::GemNotFound.

And (3) is not necessary step for edit and rebuild, but it should run after rebuild.

I believe the example uses mounts, so you don't actually need to rebuild, unless the mounts aren't working for some reason.
You can use docker-compose --build up instead of a separate build then up.
I don't think it's strictly necessary to rebuild, but I guess we could add it, if we explain what it's done.

@dnephin thanks, yes the example totally works. I was just thinking that if for some reason they wanted to make a change to base code, then they would need to re-build, and it would be nice to show how to do that.

To clarify, I say:

Until we run any rails command in container, like docker-compose up, then rails will update and sync Gemfile.lock to host, then it is match current Gemfile.

Do not mean docker-compose up is a necessary step for rebuild, it means who finished the tutorial will accident get a correct Gemfile.lock, but will found problem after edit Gemfile.
docker-compose up --build looks good, I didn‘t know before. 👍

@chloerei thanks! I'll ping you on Monday or so with the PR. Great feedback.


Change the default error message with simple_form

I don't manage to change the default error message from simple form, i tried to edit the simple form locale file but it seems to be ignored
Here is my locale file:
#config/locales/simple_form.en.yml
en:
  simple_form:
    error_notification:
      default_message: "A custom message:"

But i still get "Please review the problems below:"
Do you have any idea what i could be doing wrong ?
P.S: Using simple_form 2.0.4
Thanks for your help, Vincent.

Same issue here, and I can't wrap my head around why this doesnt work.
Using simple_form (4.1.0)


⚠ Bofrost.de has degraded performance

In a7d62a5, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1417 ms

Resolved: Bofrost.de performance has improved in fc99ce4.


re2: fix static, always build tests

Description of changes
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Sorry about the rebuild confusion, should be fixed now.

Please target staging. Do not just change the target branch in this PR (it will ping a lot of people).

@azahi this change does not cause a mass rebuild, no need to target staging.

I see, let's merge it then.


Restrict editing redirects per role/website

Hi,
I'd like to restrict access to redirects based on website and roles.
Our solution has multiple websites with separate roles for access. For example if account has a roleX with access only to websiteX, then only redirects of websiteX should be visible and editable.
I thought about using services.Intercept to intercept IRedirectsService (or even more interfaces if required) and changing returned results based on current url, matched website and current user roles.
Is there a better way to do this than intercepting service? Or could be that there are plans for implementing restrictions?
Example code which I though of can be find here on my post in Optimizely forum.
Hopefully @marisks you're best person to help
Thank you

Hi! No there are no plans to make more granular access rights. That is not a common requirement.
And sure you can decorate IRedirectsService and filter results based on your needs.