MPI distributed renderer's renderFrame hangs when more than one instance is ran on the same node.

Everything is ok with -n 1 -hosts localhost and -ppn 1 -hosts .... Hangs with -n 2 -hosts localhost.
In the test case, MPI is already initialized before the first call to Ospray.
Tracking it further. I'll update.

We are keen to know what you find, as multiple ranks on a single node should work (though not optimal for rendering performance).
3 quick questions for my own clarity:

What hardware is this running on (Xeon/KNL/laptop)?
What device are you using (mpi_offload or mpi_distributed)?
Is this a custom app, a VTK app (i.e. ParaView or VisIt), or one of our viewers?

The answer to (2) can have very different issues associated with them. Even if you end up figuring it out on your own, it would be helpful to know a run time environment that resulted in a problem. :)
Cheers,
Jeff

Dear Jeff,
1- Laptop, Intel Core i5-8350U<EMAIL_ADDRESS>2- The mpi_distributed device. I partition the problem in 3 dimensions, and each rank adds its own block of geometry (with appropriate offsets according to rank) to a model. I assume things are just as in non-mpi mode except this, and call renderFrame on all ranks, however only store the output image of rank 0.
3- It is a custom app that does particle tracing and then renders the streamlines in-situ with Ospray.
I could trace that it does not get past the condition wait in DistributedFrameBuffer.cpp:345:
auto startWaitFrame = high_resolution_clock::now();
std::unique_lock<std::mutex> lock(mutex);
frameDoneCond.wait(lock, [&]{
  return frameIsDone;
});

Any ideas? The Ospray part of my code is only about 150 lines. Do you see something stupidly off below? :
#include <prs/stages/raytracer.hpp>

namespace prs
{
raytracer::raytracer (const std::size_t thread_count, boost::mpi::communicator* communicator) : communicator_(communicator)
{
  if (ospLoadModule("ispc") != OSP_NO_ERROR)
    throw std::runtime_error("Failed to load OSPRay ISPC module");
  if (ospLoadModule("mpi") != OSP_NO_ERROR)
    throw std::runtime_error("Failed to load OSPRay MPI module");

  const auto device = ospray::cpp::Device("mpi_distributed");
  ospDeviceSetStatusFunc(device.handle(), [ ] (const char* message) { std::cout << message << "\n"; });
  device.set       ("numThreads" , thread_count);
  device.set       ("setAffinity", 1           );
  device.set       ("masterRank" , 0           );
#if _DEBUG
  device.set       ("logLevel"   , 2           );
  device.set       ("logOutput"  , "cout"      );
  device.set       ("errorOutput", "cerr"      );
#endif
  device.commit    ();
  device.setCurrent();

  auto material = ospray::cpp::Material("raycast", "OBJMaterial");
  material.set   ("Ks", 0.7F, 0.7F, 0.7F);
  material.set   ("Ns", 10.0F);
  material.commit();

  geometry_   .emplace_back(std::make_unique<ospray::cpp::Geometry>("streamlines"));
  vertex_data_.emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_FLOAT3A, nullptr)); vertex_data_.back()->commit();
  color_data_ .emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_FLOAT4 , nullptr)); color_data_ .back()->commit();
  index_data_ .emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_INT    , nullptr)); index_data_ .back()->commit();

  geometry_[0]->set        ("vertex"      , *vertex_data_.back());
  geometry_[0]->set        ("vertex.color", *color_data_ .back());
  geometry_[0]->set        ("index"       , *index_data_ .back());
  geometry_[0]->set        ("radius"      , 0.1F);
  geometry_[0]->setMaterial(material);
  geometry_[0]->commit     ();

  model_ = std::make_unique<ospray::cpp::Model>();
  model_->addGeometry(*geometry_[0]);
  model_->set        ("id", communicator_->rank());
  model_->commit     ();

  camera_ = std::make_unique<ospray::cpp::Camera>("perspective");
  camera_->set   ("fovy"      , 68.0F);
  camera_->set   ("imageStart", ospcommon::vec2f{0.0F, 1.0F});
  camera_->set   ("imageEnd"  , ospcommon::vec2f{1.0F, 0.0F});
  camera_->commit();

  auto ambient_light = ospray::cpp::Light("ambient");
  ambient_light.set   ("intensity", 0.2F);
  ambient_light.commit();
  const auto ambient_handle = ambient_light.handle();

  auto distant_light = ospray::cpp::Light("distant");
  distant_light.set   ("direction"      , 1.0F, 1.0F, -0.5F);
  distant_light.set   ("color"          , 1.0F, 1.0F,  0.8F);
  distant_light.set   ("intensity"      , 0.8F);
  distant_light.set   ("angularDiameter", 1.0F);
  distant_light.commit();
  const auto distant_handle = distant_light.handle();

  std::vector<OSPLight> lights_list = {ambient_handle, distant_handle};
  lights_ = std::make_unique<ospray::cpp::Data>(lights_list.size(), OSP_LIGHT, lights_list.data());
  lights_->commit();

  renderer_ = std::make_unique<ospray::cpp::Renderer>("mpi_raycast");
  renderer_->set   ("aoSamples"            , 1       );
  renderer_->set   ("aoTransparencyEnabled", false   );
  renderer_->set   ("oneSidedLighting"     , true    );
  renderer_->set   ("spp"                  , 1       );
  renderer_->set   ("bgColor"              , 1.0F, 1.0F, 1.0F, 1.0F);
  renderer_->set   ("camera"               , *camera_);
  renderer_->set   ("lights"               , *lights_);
  renderer_->set   ("model"                , *model_ );
  renderer_->commit();

  framebuffer_ = std::make_unique<ospray::cpp::FrameBuffer>(ospcommon::vec2i(32, 32), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
raytracer::~raytracer()
{
  framebuffer_.reset();
  lights_     .reset();
  index_data_ .clear();
  color_data_ .clear();
  vertex_data_.clear();
  geometry_   .clear();
  model_      .reset();
  camera_     .reset();
  renderer_   .reset();
  ospShutdown();
}

void  raytracer::set_streamlines(streamlines<>*                      streamlines)
{
  auto material = ospray::cpp::Material("scivis", "OBJMaterial");
  material.set   ("Ks", 0.7F, 0.7F, 0.7F);
  material.set   ("Ns", 10.0F);
  material.commit();

  model_ = std::make_unique<ospray::cpp::Model>();
  geometry_   .clear();
  vertex_data_.clear();
  color_data_ .clear();
  index_data_ .clear();
  geometry_   .reserve(streamlines->chunks.size());
  vertex_data_.reserve(streamlines->chunks.size());
  color_data_ .reserve(streamlines->chunks.size());
  index_data_ .reserve(streamlines->chunks.size());

  if(communicator_->rank() == 0)
  {
  for(auto& chunk : streamlines->chunks)
  {
    if (chunk.indices.empty())
      continue;

    const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
    const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
    const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
    const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
    geometry->setMaterial(material);
    geometry->set        ("vertex"      , *vertex);
    geometry->set        ("vertex.color", *color );
    geometry->set        ("index"       , *index );
    geometry->set        ("radius"      , streamlines->radius);
    geometry->commit     ();
    model_  ->addGeometry(*geometry);
  }
  model_->set   ("id", communicator_->rank());
  model_->commit();
  }

  renderer_->set   ("model", *model_);
  renderer_->commit();

  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::set_camera     (const camera&                       camera     )
{
  camera_->set   ("pos", ospcommon::vec3f{camera.position[0], camera.position[1], camera.position[2]});
  camera_->set   ("dir", ospcommon::vec3f{camera.forward [0], camera.forward [1], camera.forward [2]});
  camera_->set   ("up" , ospcommon::vec3f{camera.up      [0], camera.up      [1], camera.up      [2]});
  camera_->commit();

  framebuffer_->clear (OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::set_image_size (const std::array<std::uint64_t, 2>& image_size )
{
  framebuffer_size_ = image_size;

  camera_->set   ("aspect", framebuffer_size_[0] / static_cast<float>(framebuffer_size_[1]));
  camera_->commit();

  framebuffer_ = std::make_unique<ospray::cpp::FrameBuffer>(
    ospcommon::vec2i(static_cast<std::int32_t>(framebuffer_size_[0]), static_cast<std::int32_t>(framebuffer_size_[1])),
    OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}

void  raytracer::trace          (const std::size_t                   iterations )
{
  std::cout << "I will get stuck soon." << std::endl;
  for (std::size_t i = 0; i < iterations; ++i)
    renderer_->renderFrame(*framebuffer_, OSP_FB_COLOR | OSP_FB_ACCUM);
  std::cout << "No?" << std::endl;
}
image raytracer::serialize      ()
{
  image image;
  const auto bytes = static_cast<std::uint8_t*>(framebuffer_->map(OSP_FB_COLOR));
  image.set_data    (bytes, framebuffer_size_[0] * framebuffer_size_[1] * sizeof(std::array<std::uint8_t, 4>));
  image.mutable_size()->Resize(2, 0);
  image.mutable_size()->Set   (0, framebuffer_size_[0]);
  image.mutable_size()->Set   (1, framebuffer_size_[1]);
  framebuffer_->unmap(bytes);
  return image;
}
}

Hey @acdemiralp ,
From looking through the code I think the issue your model and geometry setup in set_streamlines. The problem with the code below is that only rank 0 specifies the streamline geometry, and the other ranks have empty models with no data. When running with the distributed device each rank needs to specify its own data independently, we don't broadcast out data or API calls like in the offload device. So instead each rank should run the loop to setup the geometry in that function, but will specify different geometry from its local chunk. A possible implementation would be as follows, where now chunk actually refers to a distinct streamline piece which is computed by and owned by the rank. You may also want to look at how the scene is setup in the MPIDistributed Spheres Tutorial.
// chunk is a unique streamlines<> owned by this rank only
if (!chunk.indices.empty()) {
	const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
	const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
	const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
	const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
	geometry->setMaterial(material);
	geometry->set        ("vertex"      , *vertex);
	geometry->set        ("vertex.color", *color );
	geometry->set        ("index"       , *index );
	geometry->set        ("radius"      , streamlines->radius);
	geometry->commit     ();
	model_  ->addGeometry(*geometry);
}
model_->set   ("id", communicator_->rank());
// If the model bounds are not disjoint, also specify region.lower
// and region.upper here
model_->commit();
renderer_->set   ("model", *model_);
renderer_->commit();

I think that's likely the reason it hangs, but some other possible issues may also be worth looking at if that doesn't fix it.
Another possible issue is that there might be some overlap between the bounding boxes of the geometry specified on each rank, if you have some ghost zones or something. It might be useful to check the list of bounding boxes and model information that the renderer is getting. If you increase the log level to 3 you should see a list of bounding boxes and list of ranks that own them (printed here). If there's any overlap this could be an issue, or if some ranks bounds are empty (a box3f(inf, -inf)) the compositor may not figure out what pixels they touch properly. To fix this you can override the model bounds on each rank by setting the region.lower and region.upper vec3f parameters on the models, to clip the bounds to just the grid cell they own.
Another possible issue is if you're creating the framebuffer on one rank but not the others, or are out of order in how you call it. If the ranks get out of sync here they won't be communicating with the right framebuffer object on the other ranks, so making the 32x32 pixel framebuffer in the constructor may be an issue.
Another thing that may not be a bug but is probably good to remove is that in the constructor you setup a scene and geometry with empty vertex values/etc, I'm not sure if that may cause some issue as well with figuring out the object's bounds locally. Same with making the 32x32 empty framebuffer.
Since you're using AO on distributed data you might also see some artifacts if you don't replicate the data around to build ghost geometry as well for computing the AO/shadows.

Dear Will,
Thank you very much for the detailed response.
I apologize, the first error you noticed was for debugging purposes after I started getting this problem. It exists even when if (rank == 0) is removed.
I am looking into the other issues you mentioned and going to update soon.

Getting there. Based on Will's suggestions, I have placed some communicator barriers around geometry construction and renderFrame. Now rank 0 yields me the following image:

The expected (single process) image:

The framebuffer compositor seems to be confused.

Interesting, not hanging is good! It sounds like the framebuffer handles are sync'd up properly now, so the renderer can at least communicate properly between processes.
My guess as to why you're seeing roughly half of the data missing is due to only rank 0 setting up the geometry in your set_streamlines function. Since only rank 0 has any data or volume data the others will just render some background. Though in this case it looks like a volume dataset, which I don't see setup in your code? Could you share that setup as well? For the volume I would also recommend checking out the MPI Distributed Structured Volume Tutorial which will show how to setup a distributed volume dataset, offset the bricks and clip ghost voxels by overriding the model's region bounds. The background color differences may come from setting different bgColor on each process, or not setting it on some of them.

I decided to try out Ospray 1.7.3 just as a last hour effort at work, without any changes to code and it worked, showing me the image I expected to see:

Due to time constraints (EGPGV), I cannot spend more time on fixing what's wrong with the latest distributed renderer at the moment, since I still have to make that mid-line disappear and write a half-paper on top of that in 11 days.
I will get back onto it after the deadline, because your library is awesome in many ways and I wanna make it better, although it really makes me sad when behavior changes randomly version to version, resulting in 2 research paper ragequits until today.
@Will:

The rank==0 was due to debugging as I mentioned earlier. It nevertheless gave the earlier results you mentioned without that if statement.
Those are not volumes but very tiny streamlines seen from afar.
This code works in 1.7.3 but not in 1.8.0:

#include <prs/stages/raytracer.hpp>

namespace prs
{
raytracer::raytracer (boost::mpi::communicator* communicator, const std::size_t thread_count) : communicator_(communicator), raw_communicator_(*communicator)
{
  if (ospLoadModule("ispc") != OSP_NO_ERROR)
    throw std::runtime_error("Failed to load OSPRay ISPC module");
  if (ospLoadModule("mpi") != OSP_NO_ERROR)
    throw std::runtime_error("Failed to load OSPRay MPI module");

  const auto device = ospray::cpp::Device("mpi_distributed");
  ospDeviceSetStatusFunc(device.handle(), [ ] (                const char* message) { std::cout << message << "\n"; });
  ospDeviceSetErrorFunc (device.handle(), [ ] (OSPError error, const char* message) { std::cout << message << "\n"; });
  device.set       ("numThreads"       , thread_count      );
  device.set       ("setAffinity"      , 1                 );
  device.set       ("masterRank"       , 0                 );
  device.set       ("worldCommunicator", &raw_communicator_);
//#if _DEBUG
  device.set       ("logLevel"         , 3                 );
  device.set       ("logOutput"        , "cout"            );
  device.set       ("errorOutput"      , "cerr"            );
//#endif
  device.commit    ();
  device.setCurrent();

  auto material = ospray::cpp::Material("scivis", "OBJMaterial");
  material.set   ("Ks", 0.7F, 0.7F, 0.7F);
  material.set   ("Ns", 10.0F);
  material.commit();

  geometry_   .emplace_back(std::make_unique<ospray::cpp::Geometry>("streamlines"));
  vertex_data_.emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_FLOAT3A, nullptr)); vertex_data_.back()->commit();
  color_data_ .emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_FLOAT4 , nullptr)); color_data_ .back()->commit();
  index_data_ .emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_INT    , nullptr)); index_data_ .back()->commit();

  geometry_[0]->set        ("vertex"      , *vertex_data_.back());
  geometry_[0]->set        ("vertex.color", *color_data_ .back());
  geometry_[0]->set        ("index"       , *index_data_ .back());
  geometry_[0]->set        ("radius"      , 0.1F);
  geometry_[0]->setMaterial(material);
  geometry_[0]->commit     ();

  model_ = std::make_unique<ospray::cpp::Model>();
  model_->addGeometry(*geometry_[0]);
  model_->set        ("id", communicator_->rank());
  model_->commit     ();

  camera_ = std::make_unique<ospray::cpp::Camera>("perspective");
  camera_->set   ("fovy"      , 68.0F);
  camera_->set   ("imageStart", ospcommon::vec2f{0.0F, 1.0F});
  camera_->set   ("imageEnd"  , ospcommon::vec2f{1.0F, 0.0F});
  camera_->commit();

  auto ambient_light = ospray::cpp::Light("ambient");
  ambient_light.set   ("intensity", 0.2F);
  ambient_light.commit();
  const auto ambient_handle = ambient_light.handle();

  auto distant_light = ospray::cpp::Light("distant");
  distant_light.set   ("direction"      , 1.0F, 1.0F, -0.5F);
  distant_light.set   ("color"          , 1.0F, 1.0F,  0.8F);
  distant_light.set   ("intensity"      , 0.8F);
  distant_light.set   ("angularDiameter", 1.0F);
  distant_light.commit();
  const auto distant_handle = distant_light.handle();

  std::vector<OSPLight> lights_list = {ambient_handle, distant_handle};
  lights_ = std::make_unique<ospray::cpp::Data>(lights_list.size(), OSP_LIGHT, lights_list.data());
  lights_->commit();

  renderer_ = std::make_unique<ospray::cpp::Renderer>("mpi_raycast");
  renderer_->set   ("aoSamples"            , 1       );
  renderer_->set   ("aoTransparencyEnabled", false   );
  renderer_->set   ("oneSidedLighting"     , true    );
  renderer_->set   ("spp"                  , 1       );
  renderer_->set   ("bgColor"              , 1.0F, 1.0F, 1.0F, 1.0F);
  renderer_->set   ("camera"               , *camera_);
  renderer_->set   ("lights"               , *lights_);
  renderer_->set   ("model"                , *model_ );
  renderer_->commit();

  framebuffer_ = std::make_unique<ospray::cpp::FrameBuffer>(ospcommon::vec2i(32, 32), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
raytracer::~raytracer()
{
  framebuffer_.reset();
  lights_     .reset();
  index_data_ .clear();
  color_data_ .clear();
  vertex_data_.clear();
  geometry_   .clear();
  model_      .reset();
  camera_     .reset();
  renderer_   .reset();
  ospShutdown();
}

void  raytracer::set_streamlines(streamlines<>*                      streamlines)
{
  auto material = ospray::cpp::Material("scivis", "OBJMaterial");
  material.set   ("Ks", 0.7F, 0.7F, 0.7F);
  material.set   ("Ns", 10.0F);
  material.commit();

  model_ = std::make_unique<ospray::cpp::Model>();
  geometry_   .clear();
  vertex_data_.clear();
  color_data_ .clear();
  index_data_ .clear();
  geometry_   .reserve(streamlines->chunks.size());
  vertex_data_.reserve(streamlines->chunks.size());
  color_data_ .reserve(streamlines->chunks.size());
  index_data_ .reserve(streamlines->chunks.size());

  for(auto& chunk : streamlines->chunks)
  {
    if (chunk.indices.empty())
      continue;

    const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
    const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
    const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
    const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
    geometry->setMaterial(material);
    geometry->set        ("vertex"      , *vertex);
    geometry->set        ("vertex.color", *color );
    geometry->set        ("index"       , *index );
    geometry->set        ("radius"      , streamlines->radius);
    geometry->commit     ();
    model_  ->addGeometry(*geometry);
  }

  ospcommon::vec3f region_lower_bounds { streamlines->bounds[0][0], streamlines->bounds[0][1], streamlines->bounds[0][2] };
  ospcommon::vec3f region_upper_bounds { streamlines->bounds[1][0], streamlines->bounds[1][1], streamlines->bounds[1][2] };
  model_->set   ("id"          , communicator_->rank());
  model_->set   ("region.lower", region_lower_bounds  );
  model_->set   ("region.upper", region_upper_bounds  );
  model_->commit();

  renderer_->set   ("model", *model_);
  renderer_->commit();

  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::set_camera     (const camera&                       camera     )
{
  camera_->set   ("pos", ospcommon::vec3f{camera.position[0], camera.position[1], camera.position[2]});
  camera_->set   ("dir", ospcommon::vec3f{camera.forward [0], camera.forward [1], camera.forward [2]});
  camera_->set   ("up" , ospcommon::vec3f{camera.up      [0], camera.up      [1], camera.up      [2]});
  camera_->commit();

  framebuffer_->clear (OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::set_image_size (const std::array<std::uint64_t, 2>& image_size )
{
  framebuffer_size_ = image_size;

  camera_->set   ("aspect", framebuffer_size_[0] / static_cast<float>(framebuffer_size_[1]));
  camera_->commit();

  framebuffer_ = std::make_unique<ospray::cpp::FrameBuffer>(
    ospcommon::vec2i(static_cast<std::int32_t>(framebuffer_size_[0]), static_cast<std::int32_t>(framebuffer_size_[1])),
    OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}

void  raytracer::trace          (const std::size_t                   iterations )
{
  for (std::size_t i = 0; i < iterations; ++i)
    renderer_->renderFrame(*framebuffer_, OSP_FB_COLOR | OSP_FB_ACCUM);
}
image raytracer::serialize      ()
{
  image image;
  const auto bytes = static_cast<std::uint8_t*>(framebuffer_->map(OSP_FB_COLOR));
  image.set_data    (bytes, framebuffer_size_[0] * framebuffer_size_[1] * sizeof(std::array<std::uint8_t, 4>));
  image.mutable_size()->Resize(2, 0);
  image.mutable_size()->Set   (0, framebuffer_size_[0]);
  image.mutable_size()->Set   (1, framebuffer_size_[1]);
  framebuffer_->unmap(bytes);
  return image;
}
}

I can open up the whole repo if you like.
I'll get back onto this in 11 days.

Hey @acdemiralp , the code that works on 1.7.3 looks like it should also work on 1.8, since the region bounds are set on each model. One possible issue is using the camera image start end parameters to do the y flip, this may not be handled by the compositor properly when it's determining which data touches which tile. It might be worth trying with that off, and see if the image comes out correct (though upside down). If you can share the repo that would be great, I'd be interested in seeing the code which sets up and uses the streamlines here, and if I can test it myself to see what's going on.

Hey 11 days or 111, I’m cool with whatever
On Wed, Feb 20, 2019 at 11:29 Ali Can Demiralp<EMAIL_ADDRESS>wrote:

I decided to try out Ospray 1.7.3 just as a last hour effort at work,
without any changes to code and it worked, showing me the image I expected
to see:
[image: test3 json nodes_2 threads_8 rank_0]
https://user-images.githubusercontent.com/6049621/53118138-c3c89580-354c-11e9-892d-02156caacf63.png
Due to time constraints (EGPGV), I cannot spend more time on fixing what's
wrong with the latest distributed renderer at the moment, since I still
have to make that mid-line disappear and write a half-paper on top of that
in 11 days.
I will get back onto it after the deadline, because your library is
awesome in many ways and I wanna make it better, although it really makes
me sad when behavior changes randomly version to version, resulting in 2
research paper ragequits until today.
@will https://github.com/will:

The rank==0 was due to debugging as I mentioned earlier. It
nevertheless gave the earlier results you mentioned without that if
statement.
Those are not volumes but very tiny streamlines seen from afar.
This code works in 1.7.3 but not in 1.8.0:

#include <prs/stages/raytracer.hpp>
namespace prs
{raytracer::raytracer (boost::mpi::communicator* communicator, const std::size_t thread_count) : communicator_(communicator), raw_communicator_(*communicator)
{
if (ospLoadModule("ispc") != OSP_NO_ERROR)
throw std::runtime_error("Failed to load OSPRay ISPC module");
if (ospLoadModule("mpi") != OSP_NO_ERROR)
throw std::runtime_error("Failed to load OSPRay MPI module");
const auto device = ospray::cpp::Device("mpi_distributed");
ospDeviceSetStatusFunc(device.handle(), [ ] (                const char* message) { std::cout << message << "\n"; });
ospDeviceSetErrorFunc (device.handle(), [ ] (OSPError error, const char* message) { std::cout << message << "\n"; });
device.set       ("numThreads"       , thread_count      );
device.set       ("setAffinity"      , 1                 );
device.set       ("masterRank"       , 0                 );
device.set       ("worldCommunicator", &raw_communicator_);//#if _DEBUG
device.set       ("logLevel"         , 3                 );
device.set       ("logOutput"        , "cout"            );
device.set       ("errorOutput"      , "cerr"            );//#endif
device.commit    ();
device.setCurrent();
auto material = ospray::cpp::Material("scivis", "OBJMaterial");
material.set   ("Ks", 0.7F, 0.7F, 0.7F);
material.set   ("Ns", 10.0F);
material.commit();
geometry_   .emplace_back(std::make_uniqueospray::cpp::Geometry("streamlines"));
vertex_data_.emplace_back(std::make_uniqueospray::cpp::Data(0, OSP_FLOAT3A, nullptr)); vertex_data_.back()->commit();
color_data_ .emplace_back(std::make_uniqueospray::cpp::Data(0, OSP_FLOAT4 , nullptr)); color_data_ .back()->commit();
index_data_ .emplace_back(std::make_uniqueospray::cpp::Data(0, OSP_INT    , nullptr)); index_data_ .back()->commit();
geometry_[0]->set        ("vertex"      , *vertex_data_.back());
geometry_[0]->set        ("vertex.color", *color_data_ .back());
geometry_[0]->set        ("index"       , *index_data_ .back());
geometry_[0]->set        ("radius"      , 0.1F);
geometry_[0]->setMaterial(material);
geometry_[0]->commit     ();
model_ = std::make_uniqueospray::cpp::Model();
model_->addGeometry(*geometry_[0]);
model_->set        ("id", communicator_->rank());
model_->commit     ();
camera_ = std::make_uniqueospray::cpp::Camera("perspective");
camera_->set   ("fovy"      , 68.0F);
camera_->set   ("imageStart", ospcommon::vec2f{0.0F, 1.0F});
camera_->set   ("imageEnd"  , ospcommon::vec2f{1.0F, 0.0F});
camera_->commit();
auto ambient_light = ospray::cpp::Light("ambient");
ambient_light.set   ("intensity", 0.2F);
ambient_light.commit();
const auto ambient_handle = ambient_light.handle();
auto distant_light = ospray::cpp::Light("distant");
distant_light.set   ("direction"      , 1.0F, 1.0F, -0.5F);
distant_light.set   ("color"          , 1.0F, 1.0F,  0.8F);
distant_light.set   ("intensity"      , 0.8F);
distant_light.set   ("angularDiameter", 1.0F);
distant_light.commit();
const auto distant_handle = distant_light.handle();
std::vector lights_list = {ambient_handle, distant_handle};
lights_ = std::make_uniqueospray::cpp::Data(lights_list.size(), OSP_LIGHT, lights_list.data());
lights_->commit();
renderer_ = std::make_uniqueospray::cpp::Renderer("mpi_raycast");
renderer_->set   ("aoSamples"            , 1       );
renderer_->set   ("aoTransparencyEnabled", false   );
renderer_->set   ("oneSidedLighting"     , true    );
renderer_->set   ("spp"                  , 1       );
renderer_->set   ("bgColor"              , 1.0F, 1.0F, 1.0F, 1.0F);
renderer_->set   ("camera"               , *camera_);
renderer_->set   ("lights"               , *lights_);
renderer_->set   ("model"                , *model_ );
renderer_->commit();
framebuffer_ = std::make_uniqueospray::cpp::FrameBuffer(ospcommon::vec2i(32, 32), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}raytracer::~raytracer()
{
framebuffer_.reset();
lights_     .reset();
index_data_ .clear();
color_data_ .clear();
vertex_data_.clear();
geometry_   .clear();
model_      .reset();
camera_     .reset();
renderer_   .reset();
ospShutdown();
}
void  raytracer::set_streamlines(streamlines<>*                      streamlines)
{
auto material = ospray::cpp::Material("scivis", "OBJMaterial");
material.set   ("Ks", 0.7F, 0.7F, 0.7F);
material.set   ("Ns", 10.0F);
material.commit();
model_ = std::make_uniqueospray::cpp::Model();
geometry_   .clear();
vertex_data_.clear();
color_data_ .clear();
index_data_ .clear();
geometry_   .reserve(streamlines->chunks.size());
vertex_data_.reserve(streamlines->chunks.size());
color_data_ .reserve(streamlines->chunks.size());
index_data_ .reserve(streamlines->chunks.size());
for(auto& chunk : streamlines->chunks)
{
if (chunk.indices.empty())
continue;
const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
geometry->setMaterial(material);
geometry->set        ("vertex"      , *vertex);
geometry->set        ("vertex.color", *color );
geometry->set        ("index"       , *index );
geometry->set        ("radius"      , streamlines->radius);
geometry->commit     ();
model_  ->addGeometry(*geometry);

}
ospcommon::vec3f region_lower_bounds { streamlines->bounds[0][0], streamlines->bounds[0][1], streamlines->bounds[0][2] };
ospcommon::vec3f region_upper_bounds { streamlines->bounds[1][0], streamlines->bounds[1][1], streamlines->bounds[1][2] };
model_->set   ("id"          , communicator_->rank());
model_->set   ("region.lower", region_lower_bounds  );
model_->set   ("region.upper", region_upper_bounds  );
model_->commit();
renderer_->set   ("model", *model_);
renderer_->commit();
framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}void  raytracer::set_camera     (const camera&                       camera     )
{
camera_->set   ("pos", ospcommon::vec3f{camera.position[0], camera.position[1], camera.position[2]});
camera_->set   ("dir", ospcommon::vec3f{camera.forward [0], camera.forward [1], camera.forward [2]});
camera_->set   ("up" , ospcommon::vec3f{camera.up      [0], camera.up      [1], camera.up      [2]});
camera_->commit();
framebuffer_->clear (OSP_FB_COLOR | OSP_FB_ACCUM);
}void  raytracer::set_image_size (const std::array<std::uint64_t, 2>& image_size )
{
framebuffer_size_ = image_size;
camera_->set   ("aspect", framebuffer_size_[0] / static_cast(framebuffer_size_[1]));
camera_->commit();
framebuffer_ = std::make_uniqueospray::cpp::FrameBuffer(
ospcommon::vec2i(static_caststd::int32_t(framebuffer_size_[0]), static_caststd::int32_t(framebuffer_size_[1])),
OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::trace          (const std::size_t                   iterations )
{
for (std::size_t i = 0; i < iterations; ++i)
renderer_->renderFrame(*framebuffer_, OSP_FB_COLOR | OSP_FB_ACCUM);
}
image raytracer::serialize      ()
{
image image;
const auto bytes = static_caststd::uint8_t*(framebuffer_->map(OSP_FB_COLOR));
image.set_data    (bytes, framebuffer_size_[0] * framebuffer_size_[1] * sizeof(std::array<std::uint8_t, 4>));
image.mutable_size()->Resize(2, 0);
image.mutable_size()->Set   (0, framebuffer_size_[0]);
image.mutable_size()->Set   (1, framebuffer_size_[1]);
framebuffer_->unmap(bytes);
return image;
}
}

I can open up the whole repo if you like.
I'll get back onto this in 11 days.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ospray/ospray/issues/300#issuecomment-465720697, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAHtVvJRxQtzac851vTGdPMaqlAag77ks5vPaIAgaJpZM4bDyaP
.

@acdemiralp for a quick test I changed the MPI distributed spheres tutorial to flip the image using the camera start/end parameters and also find that it hangs. I wonder if removing these will get it rendering? Then you could flip the framebuffer in a post-process on rank 0 for EGPGV, and I'll work on resolving this bug. Let me know if that works

I took a look at getting the image start/end camera parameters working properly in the distributed renderer, and think I should have this working now. At least, in the distributed tutorials I can change the image start/end parameters and get the cropped/flipped/etc. render I expect out.
@acdemiralp if you've got time, could you try applying the following patch to OSPRay's perspective camera?
diff --git a/origin/devel:ospray/camera/PerspectiveCamera.cpp b/will/mpi-cam-subregion:ospray/camera/PerspectiveCamera.cpp
index 22e5d40..aa5a001 100644
--- a/origin/devel:ospray/camera/PerspectiveCamera.cpp
+++ b/will/mpi-cam-subregion:ospray/camera/PerspectiveCamera.cpp
@@ -116,8 +116,9 @@ namespace ospray {
     const float t = 1.0 / denom;

     const vec3f screenDir = r * t - dir_00;
-    const vec2f sp = vec2f(dot(screenDir, normalize(dir_du)),
-                           dot(screenDir, normalize(dir_dv))) / imgPlaneSize;
+    vec2f sp = vec2f(dot(screenDir, normalize(dir_du)),
+                     dot(screenDir, normalize(dir_dv))) / imgPlaneSize;
+    sp = (sp - imageStart) / (imageEnd - imageStart);
     const float depth = sign(t) * length(v);
     // TODO: Depth of field radius
     return ProjectedPoint(vec3f(sp.x, sp.y, depth), 0);

If it's not working after this or having other issues, I'd be happy to take a look at the repo and see what's going on. Though if that's the case it might be better to stick with 1.7.3 for EGPGV. I think the line across issue may be due to not having ghost streamlines across the boundary of the two processes, so streamlines that may be owned by one process but cross into the other's domain are missing, or clipped. Or if they're not being computed in the first place, it could also be due to missing the ghost voxels across the boundary from the other process (though here I'm not as familiar with the code, so mostly a guess).


Feature: Add page routing

Thank you for writing the micro frontend blog and this repo.
I've learned a lot about Qwik and Cloudflare Workers.
Please add page routing support, ideally using Qwik-City.
Thanks again.

Maybe the easiest thing to do is to add Qwik-City routing to Main.
Each "page" will contain different FragmentPlaceholders to stitch fragments together.
Let me fork and give it a try.

OK @mingfang  - thanks.
Just be aware that QwikCity doesn't work out-of-the-box on Workers without "Cloudflare Pages", which this demo is not using, so you might find some friction getting it up and running.

Closing as this request is out of scope for the cloud-gallery demo.


Remove large file support as it adds a confusing error and step

for new installations

@axiomizer, if you could take a look at this, I'd love to discuss

@axiomizer I don't expect there's any issues with removing git large file support from the repo so we'd like to merge this, just let us know if there's anything we're missing.

Closing for now as #178 requires rerolling the entire repo and we have a lot of outstanding work in branches for beta 2, beta 3 and forward.


Constant assignments should have access to local variables

Consider this Mirah code
class Foo1
	CONST1 = [1,2,3]
end

This code should compile successfully, and it does.
Consider this Mirah code
class Foo2
	CONST2 = [1,2,3].map do |i|
		i+1
	end
end

This code should compile successfully, and it does not. The compiler complains:
ERROR: can't use local '$gensym1'. (class org.mirah.jvm.mirrors.ClassScope doesn't support locals)
      `result` = java::util::ArrayList.new(`list`.size)

Analyzing this further, also this Mirah code
class Foo3
	CONST3 = begin
		a = 5
		b = a+3
		b
	end
end

does not compile, but the compiler complains
ERROR: can't use local 'a'. (class org.mirah.jvm.mirrors.ClassScope doesn't support locals)
                b = a+3

Note that the compiler does not complain about the assignment of local variable "a", just about the use of local variable "a".
It is clear that, at least if the computation is wrapped in a begin ... end-block, there should be local variable access, as the code performed is put into a static initializer anyway.
The Ruby interpreter executes the code as expected and without errors.

We probably need a class initializer scope that's similar to a method scope. I think class initializers could just use method scope for now though.


scrolling sam textures

a global body texture used for sam's body parts.
being in different locations 'cuts out' a different part of a large 'global' texture.

need:

texture
CODE (u, v coords)

From the textures we've created it seems clear we're no longer going for this technique. Closing this off.


Update life.js to fix DNDBeyond Updates

Fixes #1
(Not the multi-classing bit but it does get it at least working a bit)

I created a new character who is a Life Cleric with the Monk subclass to test, and with the changes above it seems to be working now 🙂
I expect the issue you saw regarding subclassing was also related to the delays which occur while dndbeyond does its initial page load.
I will go ahead and merge your PR and get it deployed to the chrome store


Add --include-package-data option

As in the title, --include-package-data option can now be provided as well.
The reason why I am making this pull request is that I needed selenium's data to be included (from Nuitka's issues), so I added the option :)

@fedebotu if you have a list of the required data files, you may also want to consider making a PR that adds them to Nuitka's Yaml package configuration, we are going to launch a series of posts about the format, but actually for data files and dirs, it's pretty straight-forward.

@kayhayen If I understand correctly what you are saying, I believe it shouldn't be necessary. For the required data files, they should be automatically included when using the --include-package-data option. Using --include-package-data selenium automatically includes Selenium's data without any need for external files (as mentioned here)

"--include-package-data selenium"
I have an error writing:
"Usage: main.py [--module] [--run] [options] main_module.py
main.py: error: The '--include-package-data' option requires an argument with '--include-package-data='."
Windows 10 64bit

@Radomur poting below closed issues is a surefire way to get ignored, please open a new issue

@Radomur You have to do
--include-package-data=selenium
Like the command explanation itself lists.

This is the wrong repo for Nuitka issues, selenium has recently seen a PR to do this automatically, checkout main repo.


Moneyflow Scheme gas consumption - evaluate/test/improve

Only contract creations in constructor: 10 649 600 gas
With connections (addChild): 11 042 816
Empty constructor: 1 212 416
One splitter: 999 424

@enkogu

Deploy to Rinkeby
Use compiler optimizations
Only for WeiTopDownSplitter

Deploy splitter with optimization to RinkeBy is 824 998 gas


update metallb deployment to use pinned version

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 57.623%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1818

Relevant Lines:
3155

💛  - Coveralls

I was just trying this out, about to send a PR. But then I tried main and I couldn't repro the issue. Did you see the issue? Also we should change both the speaker and controller to be pinned

Please change controller:main to controller:v0.12.1 also, as that seems to be broken in the same way. Thanks.


Fix LPOP to support multiple arguments #1100

This pull request addresses issue #1100, where the LPOP command needed to support an additional argument to specify the number of elements to pop from the left side of the list.
Changes:
Modified the LPOP command to accept either 1 or 2 arguments:
LPOP key: Pops a single element from the list (default behavior).
LPOP key count: Pops up to count elements from the list.
Validation:
If count is 0, an empty response is returned.
If count is negative, an "out of range" error is returned.
If count exceeds the number of available elements, it pops as many elements as available without error.
Invalid count (non-integer values) returns an appropriate error.
Popping from a non-existent or empty list returns nil.
Behavior:
Returns a single element if one element is popped.
Returns a list of elements if multiple elements are popped.
This is the dicedb output

This is the output for redis over the same commands

Thanks for the fix! It would be great to have unit and integration tests validating these changes.

Thanks :), Yep I am on it.

this is the unit-tests

this is the integration tests

any changes to make?

Hey,
Its been a while since I heard back.
I saw that there was a slight conflict with my branch, so i fixed it with a merge.
Would love to get some updates on this issue.

Hey, Its been a while since I heard back. I saw that there was a slight conflict with my branch, so i fixed it with a merge. Would love to get some updates on this issue.

whoops, didn't realise the comments were addressed. Going through this PR.


Scanline iterator RLEImage macOS

This addresses
https://github.com/InsightSoftwareConsortium/ITK/issues/4537
In addition to SmartPointer arguments, we need a raw const pointer version.
Suggested-by: Niels Dekker<EMAIL_ADDRESS>

@jilei-hao can you check this PR too?

@jilei-hao can you check this PR too?

It works! Thanks so much!

And it worked even without the changes in https://github.com/InsightSoftwareConsortium/ITK/pull/4553


[CARBONDATA-757] Big decimal optimization

Currently Decimal is converted to bytes and using LV (length + value) format to write to store. And while getting back read the bytes in LV format and convert back the bigdecimal.
We can do following operations to improve storage and processing.

if decimal precision is less than 9 then we can fit in int (4 bytes)
if decimal precision is less than 18 then we can fit in long (8 bytes)
if decimal precision is more than 18 then we can fit in fixed length bytes(the length bytes can vary depends on precision but it is always fixed length)

So in this approach we no need store bigdecimal in LV format, we can store in fixed format.It reduces the memory.
Carbondata format changes -> Added fixedLength in datachunk to know about the column length of big decimal. This attribute can be used in case of char(fixedlength) or varchar(fixedlength) datatypes as well.

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/2795/

Build Failed with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/216/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/731/Failed Tests: 4carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 4org.apache.carbondata.spark.testsuite.partition.TestAllDataTypeForPartitionTable.allTypeTable_hash_smallIntorg.apache.carbondata.spark.testsuite.partition.TestAllDataTypeForPartitionTable.allTypeTable_hash_varcharorg.apache.carbondata.spark.testsuite.partition.TestAllDataTypeForPartitionTable.allTypeTable_list_dateorg.apache.carbondata.spark.testsuite.partition.TestAllDataTypeForPartitionTable.allTypeTable_range_string

Build Failed with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/239/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/2819/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/2820/

Build Success with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/240/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/3005/

Build Success with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/418/

Build Success with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/420/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/3008/

LGTM

Build Success with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/426/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/3015/

Build Failed with Spark 1.6, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/550/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/3142/

SDV Build Failed with Spark 2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/41/


Image package

I'm not convinced the 1.x Image class should stay. It's part voodoo, it's part a coding mess, and nobody knows exactly how it works, which makes maintaining difficult.
Perhaps we should drop it, and suggest http://imagine.readthedocs.org/en/latest/ instead. There's a composer package for it. Not sure it's possible to create a 1.x compatibility class for it, and not sure either it covers all functionality.
I'm open to ideas!

http://image.intervention.io/ might worth a look as well. Seems to be easier to use. However I am not even sure we should provide one. The old API is too much complex, both of the mentioned libraries are simple and heavily object oriented.

@sagikazarmark makes a good point, it might not be worth us implementing an image library given that there are quite a few now out there. What might be a good idea is looking at some libraries that we like the look of and submit PRs with fuel service providers for "integration" into the framework.

Both of the mentioned ones have laravel providers, so I think submitted PRs might be accepted. If not, we still can create integration packages.

Works for me then. I don't really fancy the idea of creating yet another image package.

Yeah. Not to mention, that in the age of packagist, it is super easy to install one or another based on your needs. So besides core "MVC" (application and some extra like orm, auth) features we don't really have to provide anything else (which has been specifically solved already).

I don't think it was ever the intention to create one. Possibly (as part of the legacy package) porting the current v1 class for BC reasons...


Continuously Improve Vineyard Documentation and Tutorials

Describe your problem
Tasks:

[ ] Assess the current state(Especially the outdated) of Vineyard documentation and tutorials.
[ ] Refer to the best practices of tutorials/docs (e.g., https://www.wikihow.com/Write-a-Tutorial) and leverage tools such as GPT to assist in creating and refining documentation content.

We already have many such broad master issues (e.g., #296，#488) for docs/tutorials.
Closing. Please create sub issues for these todos and associate them with existing master issues.


Qmaps 407

Update storage id with dynamic version
Add helper to pre-store poi on test (tool.store)

The test with the panel of the poi already in favorite needs to be updated too: https://github.com/QwantResearch/erdapfel/blob/master/tests/integration/tests/poi.js#L61


Refresh token

Thank you for great sample! It is interesting to know how it is possible to implement refresh_token to use short-lived access tokens and update them using refresh_tokens, that can be revoked

Hi,
if you don't want to implement your strategy, you can use the grant type of IdentityServer: Refresh tokens.
For revocation, you must instead use the revocation endpoint: http://localhost:5000/connect/revocation.
However, if I have time in the next few days, I'll try to integrate it in the application as an option, or I will insert it in a note.
Greetings

Hi,
I've added the support for refresh token. Now a refresh token is returned with the access token, and using the getNewToken method of AuthenticationService class you can request a new access token for the user at the desired time.
I also added methods to revoke token. For more details see the explanation.
Greetings

Thank you for great addition!


In openDB prune the ledger DB to the newest state

Prune the ledger DB to the newest state after initializing the ledger state
from disk.
Fixes #3250

bors r+


[1.1] Fix dependency issue with Vite 5

This PR fixes the following dependency resolution issue for users of Vite 5 by allowing it to be a peer dependency in the latest version of this plugin.
npm error code ERESOLVE
npm error ERESOLVE unable to resolve dependency tree
npm error
npm error While resolving: undefined@undefined
npm error Found<EMAIL_ADDRESS>npm error node_modules/vite
npm error   dev<EMAIL_ADDRESS>from the root project
npm error
npm error Could not resolve dependency:
npm error peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm error node_modules/laravel-vite-plugin
npm error   dev<EMAIL_ADDRESS>from the root project
npm error
npm error Fix the upstream dependency conflict, or retry
npm error this command with --force or --legacy-peer-deps
npm error to accept an incorrect (and potentially broken) dependency resolution.

Our understanding was that NPM would not attempt to install/upgrade this plugin to v1.1 and instead continue to use v1.0 until the project's Vite dependency was bumped to v6. However, it seems that NPM tries to install the latest versions purely based on the constraints in package.json and then fails if they're incompatible instead of using the latest mutually compatible versions as Composer does.
It would seem that with NPM, the only way to safely drop support for a peer dependency major version is with a major version release, so we've decided to reintroduce support for Vite 5, which matches official Vite plugins like @vitejs/plugin-vue.
Fixes #311
Fixes #312

You guys are rockstars. Thanks for the late night fix 💪👁👄👁💪


Saurav_requireField

two important changes needed for required field  :

add red start on the require field . and right below the form that * is required .
2)if i'm at for example dental chart tab and hit apply before filling all the require field , it does not show or tell me where is the tab that has require field .

money paid and total price should not have * required field .


Develop/microsoft calendar

🔖 Summary
the request for this feature is here : https://github.com/backstage/backstage/issues/16001
the plugin will give a card component where the user can see his daily calendar schedule, and switch between different calendars, if a meeting URL is mentioned, the user can directly click on it to join a meeting and cancelled meetings are stroked.
hovering over any event will pop over a card showing you the list of people invited with a badge over each showing their responses.
https://user-images.githubusercontent.com/23618736/215717491-25db5fa6-b237-487f-8c00-28f572e8da05.mp4
✌ Context
Creating a plugin to achieve the following feature:

Select the menu to switch between calendars.
card showing the list of events on a selected date and of the selected calendar
hovering over the event will pop over a card showing the event summary message, and list of attendees.
attendee's chips will have a badge over them symbolizing their responses
green --> accepted
red --> declined
nothing --> not responded yet

Would also be pretty good to write some test for these components that you've got too. I see a lot of test-id being used, but no tests :(

Would it be possible to write some tests for this? Just some simple ones that just render the components and make sure that the expected content is there? It's pretty hard for us to accept other contributions to this plugin when there's no test that ensure that the plugin still works :)

Another separate question - where did you get the svg icons? Are they open source and licensed properly that we can copy the contents here? This change will also need a changeset too which you can generate using yarn changeset.

the icons are from https://www.svgrepo.com  which provide open-licensed SVG vectors.
https://www.svgrepo.com/svg/475024/webcam. --> webcam
https://www.svgrepo.com/svg/263516/calendar  ---> calendar
i will add the changeset.

Would it be possible to write some tests for this? Just some simple ones that just render the components and make sure that the expected content is there? It's pretty hard for us to accept other contributions to this plugin when there's no test that ensure that the plugin still works :)

i have made changes in dev folder and ensured that there is a working demo available for every user, just one command away. test case's i will try to contribute in the future or will raise it as an issue so if someone is eager can pick it up early.
@benjdlambert
everything is set , plugin is working too. what do you say can we get this integrated ?

@benjdlambert i have done the changes that u asked for, please have a look.

hey @benjdlambert , i can see that you have made the required changes, is there anything from my side left to do , or we are ready to merge this one.

Thanks for the effort and the hard work! :pray:

@benjdlambert  @awanlin for your support and appreciations.


[𝙵𝚄𝙻𝙻 𝙽𝙾𝚆@𝚂𝙴𝚇] Minahil Malik Nazeba ʟᴇᴀᴋᴇᴅ ᴠɪᴅᴇᴏ Viral Trending On Social Media X Twitter [ZHD~320G√]

13 sec ago Minahil Malik Nazeba ʟᴇᴀᴋᴇᴅ ᴠɪᴅᴇᴏ Viral On Social Media. [𝙵𝚄𝙻𝙻 𝙽𝙾𝚆@𝚂𝙴𝚇] Minahil Malik Nazeba ʟᴇᴀᴋᴇᴅ ᴠɪᴅᴇᴏ Viral Trending On Social Media X Twitter [ZHD~320G√]
Minahil Malik Nazeba’s latest video has captivated the internet, sparking lively discussions across various social media platforms. The emerging digital star, renowned for her engaging content—particularly her recent Spiderman-themed clip—has experienced a significant surge in popularity, with fans eagerly seeking the full version. Viewers are flocking to Instagram, X (formerly Twitter), and Discord to interact with her latest creation.
As excitement builds, fans are fervently sharing their reactions online, underscoring the crucial role social media plays in elevating creators like Minahil Malik Nazeba into the limelight. With her distinctive style and innovative approach, she has quickly made a substantial impact in the digital landscape, leaving many eager to see what she will do next.
Thank You

FG

.
.
.

vf
.
.

.

.


Sync files?

I tried to find an answer in your documentation and issue tracker but didn't find one so my apologies if this is a duplicate...   I know Flynn supports a clustered database (Postgres) out of the box, but does it also support file syncing as well?  For example, something like powerstrip-flocker or seafiles?  Use case is end-user file uploads that I don't want to commit to version control.

No, we do not support any clustered filesystems currently. Internally we use the postgres large object APIs to store arbitrary binary blobs.

Thanks for the fast response!


In the instance config, entries with same key are treated as array which can cause unexpected behavior

E.g. In the controller config:
controller.enable.split.commit=true
controller.enable.split.commit=true

will be treated as
controller.enable.split.commit=true,true

and then when pinot parses the value, true,true will be parsed as false
This can cause big confusion. We should consider not allowing multiple entries with the same key, but that can cause backward incompatibility. Need to figure out a solution

Yeah, I have been a victim of this as well. It is because of apache CommonsConfiguration class we are using.

I think the issue is from CommonsConfigurationUtils.toMap() which automatically combine values with the same key as a list. I tried directly using PropertiesConfiguration.getBoolean() on the same config, and it can correctly pick the first config, and returns true.


Custom enchants

I have a custom enchants plugin you can go off of

Can u provide link?

There is no link to it because it's custom and I don't know PHP very good so I need help adding it to my server


DeprecationWarning

DeprecationWarning: collection.count is deprecated, and will be removed in a future version. Use collection.countDocuments or collection.estimatedDocumentCount instead

This issue has already been registered.
I recommend you use the mongoose-paginate-v2 package.

thanks for response

Hello, @dyarleniber
what is the difference between those two packages (mongoose-paginate vs mongoose-paginate-v2) ?


Feat/multiple backends + rename

rename package to ovos-backend-client
add api_registry for different backends to advertise functionality
add backend_type concept
add ovos api backend option
add neon mq backend option
offline backend option with support for your own keys

API
Offline
Personal
Selene
OVOS
Neon

Admin
yes [2]
yes
no
no [1]
no [1]

Device
yes [3]
yes
yes
no [1]
no [1]

Wolfram
yes [5]
yes
yes
yes
yes

Geolocate
yes
yes
yes
yes
yes

STT
no
yes
yes
yes
no

Weather
yes [5]
yes
yes
yes
partial [4]

[1] will be cast to Offline type for compatibility
[2] will update user level mycroft.conf
[3] shared metrics db with personal backend for UI compat
[4] hourly/daily forecast endpoints not available
[5] needs api key key

companion PR in ovos-core https://github.com/OpenVoiceOS/ovos-core/pull/201


Pointing docs to hosted HTML files for core API.

@tjruwase I imported all of your HTML files into the gh-pages branch. This PR just links our Markdown pages to those HTML files. Thanks for putting everything together!


Letsencrypt doesn't work when email is configured

This happens because kubernetes doesnt support having @ in the secret name.

If this bug restricts your use of space-cloud, give it a thumbs up reaction, so that we can determine which bugs need to be fixed immediately.
👍

Coming in v0.19.5


⚠ Fortnite has degraded performance

In bf7740b, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1553 ms

Resolved: Fortnite performance has improved in 0e32fda.


GPS location code-block was incorrect, now working

I found that the code was wrong (outdated?). It used
'media.exif.long' when the correct is 'media.exif.gps.lon'.
It was also a bit unuseful since it showed it on the most zoomed
out map (continent level) and no marker.
The current link will go to a faily zoomed in (map=18) and put
a marker in the map.

Thanks!


linux,fs: fix calling p{read,write}v with a 64bit offset

According to 0 we need to pass it in 2 32bit registers. Fix
inspired by the musl libc implementation 1.
R=@bnoordhuis
CI: https://ci.nodejs.org/job/libuv+any-pr+multi/204/

CI seems as happy as usual, and a local MIPS qemu image too, but this is uncharted territory for me :-)
/cc @bwijen #498, #473

Addressed the comments except the one about only doing the split for required architectures.

LGTM with the nit that the first line of the commit log > 50 characters.

Thanks Ben, landed with a reworded commit message.

Guys, what about other architecture specific requirements?
Take a look @ bwijen/syscallfix that was my first try. You need to take into account byteorder (use __LONG_LONG_PAIR) and offset size (32/64)
On 32 bit, since off_t is signed, you also need sign extension.
Furthermore some architectures need an aligned arguments, so sometimes you need to skip one.
I don't think this pull request covers them all...

If the syscall code can't be removed, I think @lucab fix is the way to go...

I don't think this pull request covers them all...

I only have access to whatever Qemu is able to emulate, what architectures/scenarios do you think wouldn't be covered?

You need to take into account byteorder (use __LONG_LONG_PAIR) and offset size (32/64)

I was wondering about that.  I know glibc uses __LONG_LONG_PAIR() but musl doesn't.  Both run on big and little endian.
Looking at the kernel source, the canonical preadv and pwritev system calls have this signature:
SYSCALL_DEFINE5(preadv, unsigned long, fd, const struct iovec __user *, vec,
                unsigned long, vlen, unsigned long, pos_l, unsigned long, pos_h)

x86 and x86_64 (but not x32 and other architectures) have preadv64 / pwritev64 compatibility wrappers:
COMPAT_SYSCALL_DEFINE4(preadv64, unsigned long, fd,
                const struct compat_iovec __user *,vec,
                unsigned long, vlen, loff_t, pos)

And finally there are generic compatibility wrappers:
COMPAT_SYSCALL_DEFINE5(preadv, compat_ulong_t, fd,
                const struct compat_iovec __user *,vec,
                compat_ulong_t, vlen, u32, pos_low, u32, pos_high)

So, as far as I can tell, it's only the x86-specific preadv64 compatibility wrapper that would be problematic but we're not (or at least shouldn't be) calling that.
Also, it means I was wrong about only needing to pass the fifth argument on some architectures. :-)

I don't have a developer machine at hand at the moment, but I can build this there later.
From past tests, most of debian big-endian architectures had issues with a similar patch.
Specifically: hppa, powerpc, mips (all BE flavors).

@lucab it would be great if you could run the tests with this commit. I did test MIPS (with Qemu) and thsi fixed the problem.

Sorry for the delay, I just tested it on powerpc and mips, and it seems to be fine. I'm probably as much confused as @bwijen right now. I'll just upload the latest 1.8.0 without my patch, and see what happens on the autobuilders.

All autobuilders are green, so this fixed it.

Thanks for confirming @lucab!


add control of the arrows on FLOWLINE plots

Reported by Ansley B Manke on 5 Sep 2003 17:49 UTC
Let the user change the size of or get rid of
the arrows on a FLOWLINE plot, perhaps using the
/LENGTH qualifier.  Currently sets arrow size
based on the velocity.
yes? USE coads_climatology
yes? SET REGION/x=150e:130w/y=40s:50n/l=6
yes? FLOW/ASPECT/DENSITY=3 uwnd,vwnd
yes? VEC/ASPECT/OV uwnd,vwnd
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/601

Comment by<EMAIL_ADDRESS>on 21 Oct 2003 20:16 UTC
Moving from the<EMAIL_ADDRESS>user to the<EMAIL_ADDRESS>user.

Comment by<EMAIL_ADDRESS>on 10 Nov 2003 17:33 UTC
move to Ferret tasks, reassign to Ansley

Comment by<EMAIL_ADDRESS>on 5 Nov 2004 19:05 UTC
changing "Severity" tag of bugs under Ferret Tasks to "enhancement"

Comment by<EMAIL_ADDRESS>on 5 Nov 2004 19:19 UTC
Changing Component of bugs in "Ferret Tasks" from All to Main

Comment by<EMAIL_ADDRESS>on 1 Mar 2006 17:30 UTC
*** Bug 1321 has been marked as a duplicate of this bug. ***


RFC3339 and RFC3339Nano

How does one parse the formats below. Golang defines RFC3339 and RFC3339Nano, see https://golang.org/pkg/time/#pkg-constants for the documentation and /usr/local/go/src/time/format.go for the definition.
...
	RFC3339     = "2006-01-02T15:04:05Z07:00"
	RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
...

> moment("2006-01-02T15:04:05Z07:00").format()
"Invalid date"

Raised the same issue in the luxon repository, https://github.com/moment/luxon/issues/474.

Yep, let's please define this string in the moment.js library itself, in order to reduce the risk that users introduce a typo when they try to define this themselves.

Any news for this?
This format will be helpful to use with Google Firestore.

It appears that the examples in the Go docs for RFC3339 are wrong.  The format is really just a profile of ISO8601.
The problem with the Go doc examples for this format is that they include both Z and an offset together.  This is incorrect.  It is either Z (for Zulu or +00:00) or an offset.
If you correct the examples before trying to parse them, they are parseable by Moment.js:
> moment("2006-01-02T15:04:05-07:00").format()
'2006-01-02T17:04:05-05:00'
> moment("2006-01-02T15:04:05+07:00").format()
'2006-01-02T03:04:05-05:00'

@mbana @dobromyslov

If RFC3339 is parseable with Moment already, then I don't think we should do this.

Oh, I see, the two formats are inconsistent at the "T00:00" section. I am not sure RFC3339 always uses "Z" where listed in the Go package documentation.
https://stackoverflow.com/questions/522251/whats-the-difference-between-iso-8601-and-rfc-3339-date-formats

I may be wrong here. If so, please let me know and I can re-open this issue.


Remove all accounts regardless of failure in one account

Coverage increased (+0.0004%) to 37.649% when pulling 9db7f8d6f8a946dd0c215450af6eacfba9baa954 on amir/azureAuth/removeAccounts into 66632e6262bb01108f785c7d82d216abbfeb540d on main.


support proxy

add configuration for use proxy

The program will pickup proxy settings in environment variables like HTTPS_PROXY.
Let me know if you have any specific problems.


feat(Mongo): Config variable can disable MongoDB usage and dependency in Docker

:tada: This PR is included in version 5.4.0 :tada:
Your semantic-release bot :package::rocket:


Mount points do not resume after stop/start/reboot/etc of podman instance

Description
When using limactl with a podman instance and fuse sshfs for volume mounts, the podman instance will come up successfully just fine, but on subsequent attempts, it fails to mount the sshfs systems:
limactl stop podman
INFO[0000] Sending SIGINT to hostagent process 15141
INFO[0000] Waiting for the host agent and the qemu processes to shut down
INFO[0000] [hostagent] fusermount: mount failed: Operation not permitted
INFO[0000] [hostagent] fusermount: mount failed: Operation not permitted
INFO[0000] [hostagent] Received SIGINT, shutting down the host agent
INFO[0000] [hostagent] Shutting down the host agent
INFO[0000] [hostagent] Stopping forwarding "/run/user/502/podman/podman.sock" (guest) to "/Users/marcmercer/.lima/podman/sock/podman.sock" (host)
INFO[0000] [hostagent] Stopping forwarding "/run/lima-guestagent.sock" (guest) to "/Users/marcmercer/.lima/podman/ga.sock" (host)
INFO[0000] [hostagent] Unmounting "/Users/marcmercer"
INFO[0000] [hostagent] Unmounting "/tmp/lima"
INFO[0000] [hostagent] Shutting down QEMU with ACPI
INFO[0000] [hostagent] Sending QMP system_powerdown command
INFO[0001] [hostagent] QEMU has exited

limactl start podman
INFO[0000] Using the existing instance "podman"
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/marcmercer/.lima/podman/serial.log")
INFO[0000] SSH Local Port: 53492
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0019] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0019] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0030] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0031] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0031] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0031] [hostagent] The essential requirement 3 of 5 is satisfied
INFO[0031] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf to contain \"user_allow_other\""
INFO[0031] [hostagent] The essential requirement 4 of 5 is satisfied
INFO[0031] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
INFO[0031] [hostagent] The essential requirement 5 of 5 is satisfied
INFO[0031] [hostagent] Mounting "/Users/marcmercer"
INFO[0031] [hostagent] fusermount: mount failed: Operation not permitted
WARN[0063] [hostagent] failed to confirm whether /Users/marcmercer [remote] is successfully mounted
INFO[0063] [hostagent] Mounting "/tmp/lima"
INFO[0063] [hostagent] fusermount: mount failed: Operation not permitted
WARN[0094] [hostagent] failed to confirm whether /tmp/lima [remote] is successfully mounted
INFO[0094] [hostagent] Waiting for the optional requirement 1 of 1: "user probe 1/1"
INFO[0094] [hostagent] Forwarding "/run/user/502/podman/podman.sock" (guest) to "/Users/marcmercer/.lima/podman/sock/podman.sock" (host)
INFO[0094] [hostagent] The optional requirement 1 of 1 is satisfied
INFO[0094] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/marcmercer/.lima/podman/ga.sock" (host)
INFO[0094] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0095] [hostagent] Not forwarding TCP <IP_ADDRESS>:53
INFO[0095] [hostagent] Not forwarding TCP <IP_ADDRESS>:22
INFO[0095] [hostagent] Not forwarding TCP [::]:22
INFO[0095] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0095] READY. Run `limactl shell podman` to open the shell.
INFO[0095] Message from the instance "podman":
To run `podman` on the host (assumes podman-remote is installed):
$ export CONTAINER_HOST=unix:///Users/marcmercer/.lima/podman/sock/podman.sock

So far, the workaround is to delete and recreate the podman instance, forfeiting your currently built up infrastructure (images, containers, etc)
This is is SIMILAR to #584but not necessarily identical.

The workaround above does at least appear to work as a solution.  Thanks @afbjorklund ;; creating a separate issue for the next observed issue.


Add flag to prevent builders from running as root

If the --no-root flag is specified, the build will fail if the builder does not specify a USER with a numeric id that is not 0.

@bparees @mfojtik ptal

lgtm [merge]


$ErroractionPreference with scriptblock

its possible to have a scriptblock to conditionally break script

$ErroractionPreference = {
   # $_ is instance of $error[0]
   if($_.exception -is [ItemNotFoundException] -and $_.Exception.ItemName.Contains('system')) {
      break
   }
}

# some code here
gi k:\system\foo

You could use ErrorAction Stop and try-catch.

yes i know but $erroractionpreference in top of script reduce the case of use try/catch and the effect is global

We have RFC https://github.com/PowerShell/PowerShell-RFC/blob/9d0b313c76144e9af7b84294bba470da710810e8/1-Draft/RFCNNNN-ScriptBlock-Action-Preference.md#add--verboseaction--debugaction-and--progressaction-common-parameters
/cc @KirkMunro

You can also use the trap statement to install a single error handler for a script:
& {
    trap {
        if ($_.Exception -is [System.Management.Automation.ItemNotFoundException] -and
            $_.Exception.ItemName.Contains('system')) {
            "Couldn't find item with 'system' in the path"
            break
        }
        else {
            "Could not find $($_.Exception.ItemName) but hey, it's OK. Let's keep going."
            continue
        }
    }

    Get-Item k:\foo -ErrorAction Stop

    Get-Item k:\system\foo -ErrorAction Stop
}

You can also use the trap statement to install a single error handler for a script:
& {
    $ErrorActionPreference = 'Stop'
    trap {
        if ($_.Exception -is [System.Management.Automation.ItemNotFoundException] -and
            $_.Exception.ItemName.Contains('system')) {
            "Couldn't find item with 'system' in the path"
            break
        }
        else {
            "Could not find $($_.Exception.ItemName) but hey, it's OK. Let's keep going."
            continue
        }
    }

    Get-Item k:\foo

    Get-Item k:\system\foo
}

but this code not break in specifik error
PS > $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.0.0-rc.1
PSEdition                      Core
GitCommitId                    7.0.0-rc.1
OS                             Microsoft Windows 6.3.9600
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

PS > cat .\n.ps1
& {
    $ErrorActionPreference = 'Stop'
    trap {
        if ($_.Exception -is [System.Management.Automation.ItemNotFoundException
] -and
            $_.Exception.ItemName.Contains('system')) {
            "Couldn't find item with 'system' in the path"
            break
        }
        else {
            "Could not find $($_.Exception.ItemName) but hey, it's OK. Let's kee
p going."
            continue
        }
    }

    Get-Item k:\foo

    Get-Item k:\system\foo
}
PS > .\n.ps1
Could not find k but hey, it's OK. Let's keep going.
Could not find k but hey, it's OK. Let's keep going.
PS >

It is likely you are getting the same error for both:
Error is Cannot find drive. A drive with the name 'k' does not exist.

Change Get-Item k:\system\foo to Get-Item c:\system\foo (ditto for k:\foo) and try again.  With that change to the script I get:
12> .\n.ps1
Could not find k:\foo but hey, it's OK. Let's keep going.
Couldn't find item with 'system' in the path
Get-Item: C:\Users\Keith\n.ps1
Line |
  17 |     Get-Item c:\system\foo
     |     ^ Cannot find path 'C:\system\foo' because it does not exist.


Refactor _setDate params in dayCellClick

After #2122.

Yeah the click function will need some serious rethinking with all those editable templates.
Thank you @vsn4ik !


ReseedingSplittableRandomAdapter too slow on some environments

ReseedingSplittableRandomAdapterTest methods testDistribution and testStandardDeviation are timing out on Cloud9 without 2-minute timeouts, and on my laptop even then.

Blocked by #1.

Fixed by https://github.com/Pr0methean/BetterRandom/commit/c00f4e04104140129fcb76f8faadc1c0fa03a81c

NB: Although the fix for this is in trunk, its release is currently blocked on #4 .


NIFI-11337: Update PutDatabaseRecord's Max Batch Size property documentation to reflect actual behavior

Summary
NIFI-11337 This PR removes thedependsOn from the definition of the Maximum Batch Size property as the dependency is not honored in the code. The documentation is updated to reflect the actual behavior, and the default value is changed to 1000 to avoid memory usage issues for large incoming FlowFiles. No unit tests were added as the existing tests show there is no regression.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[x] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Thanks @mattyb149 changes look good +1. Will let Github Actions complete and then can merge.


Feature: Migrate docs

This feature is intended to be run at initial docs-migration stage, to later use git as the upstream source for documentation on discourse and/or charmhub.
Key features

It is backwards compatible with the existing reconcile mode since it accepts github_token as an optional parameter.
It supports dry_run mode which only logs actions that will be taken.
It supports custom branch name support to allow branch names other than the default upload-charm-docs.

Moving forward
Moving forward, further improvements can be made to allow migration mode to be triggered even after the initial run. Since the presence of the docs directory determines the execution flow between migration mode and reconcile mode, it is currently impossible to re-trigger the action after having fixed the upstream without deleting the branch and the PR and then rerunning the action.
Examples
Example run on indico-operator, Generated PR link
Example dry mode run on indico-operator.

As discussed, the reason for dry_run mode is for users to be able to review changes before making them public, with the migration, since there are no changes to the published documentation, the dry_run mode should have no effect, i.e., the migration PR should be created even if the dry_run mode is enabled

Also as discussed, if there are any failures during migration, the current behaviour of not creating the PR is fine for now. Please add a follow up issue to crate a partial PR if the migration fails

Just to clarify, is the following still accurate after our discussion?

Moving forward
Moving forward, further improvements can be made to allow migration mode to be triggered even after the initial run. Since the presence of the docs directory determines the execution flow between migration mode and reconcile mode, it is currently impossible to re-trigger the action after having fixed the upstream without deleting the branch and the PR and then rerunning the action.

@jdkandersson
It is no longer accurate, updating the PR!


xcsr: X-CSQA simply uses english for all alleged non-english data

Describe the bug
All the alleged non-english subcollections for the X-CSQA task in the xcsr benchmark dataset  seem to be copies of the english subcollection, rather than translations. This is in contrast to the data description:

we automatically translate the original CSQA and CODAH datasets, which only have English versions, to 15 other languages, forming development and test sets for studying X-CSR

Steps to reproduce the bug
# let's say you want to load the french X-CSQA subcollection
french = datasets.load_dataset("xcsr", "X-CSQA-fr")
# for good measure, let's load english too
english = datasets.load_dataset("xcsr", "X-CSQA-en")

# let's inspect
"".join(english['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
"".join(french['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
# what? Why are they both in english?

# I've checked this for validation and train splits too, across many datapoints. It's all the same english dataset
# maybe i need to look better?
french['test'].unique('lang')
# output: ['en']
# no, it's all english

Expected results
Accessing a subcollection in language X should return a subcollection containg samples in language X
Actual results
Accessing a subcollection in language X returns a subcollection containing samples in English.
Environment info

datasets version: 2.5.1
Platform: macOS-10.15.7-x86_64-i386-64bit
Python version: 3.8.13
PyArrow version: 9.0.0
Pandas version: 1.4.3

Thanks for reporting, @thesofakillers. Good catch. We are fixing this.


Add links to setting up service principal to the ScubaGear Functional Testing Automation instructions document

Scope
There is a markdown document in the /Testing folder that details how to execute the functional test orchestrator as a user. The document is named "ScubaGear Functional Testing Automation". The scope of this issue is to add the links below to that document for easy reference. The links below are documentation issues that I had created which describe how to setup the service principal used by the automated functional test orchestrator in GitHub and by ScubaGear developers. Those links also describe how to setup the respective conditional access policies that are necessary when running the functional test orchestrator against the Entra Id baseline.

#589
#591

Justification
The reason this is helpful is because these issues are closed and we need to refer back to them every so often when fixing configurations in the tenant or setting up a new testing configuration and this will make it much easier to find them instead of having to search the old closed issues list.
Implementation

[ ] Create a new section in the testing instructions document (this section is for sys admins). Call it Functional Testing Infrastructure Setup or something like that. Add the links above

I received internal feedback from the dev team with a recommendation on how to implement this. I think this is a good approach. Their feedback is included below for reference. If there are any questions please contact @james-garriss for context and further recommendations.

Create a new subfolder under Testing called docs.

Create Markdown files in there with the documentation from the two hyperlinks in the Scope section above. This is basically converting the information from GitHub issues to markdown files.

Create a new section in the /Testing/Readme.md file named Setup Documentation for Sys Admins

Add hyperlinks and a quick description of the two new markdown files into the new Setup Documentation for Sys Admins section.


(PUP-8974) Deserialize stringified array using JSON

The augeas provider used Kernel#eval to convert stringified arrays to
Ruby arrays. For example, it extracted the array part of the "clause"
below:
onlyif => 'values HostKey == ["/etc/ssh/ssh_host_rsa_key"]'

and called Kernel#eval with '["/etc/ssh/ssh_host_rsa_key"]'. Using eval
is bad because it executes arbitrary code.
This commit changes the provider to use JSON.parse to safely convert to
a Ruby array. Note we have to convert single quotes to double quotes.
I didn't add any tests, because the existing tests fail when the
quote translation doesn't occur.

jenkins please test this with puppet-agent#5.5.x

CLA signed by all contributors.

jenkins please test this with puppet-agent#5.5.x

ping @branan @jhelwig

This has a different set of issues. Going to close this while we figure out a more complete solution.


Profiling via plugin manager reports slow initialization with file

Introduction
To be fair, this is quite obvious since I have been lazy-loading loads of plugins whenever an event BufReadXXX has been fired. With that, I get the longest initialization of all time which is about ~500ms (as seen in the picture below) with a lowest of 300ms with an average range of 350-450ms.

I will be putting this in TODO in the meantime unless I would want to do it out of a sudden.
Profiling
For insights, the following are the screenshots containing profiling done by lazy.nvim.

At commit c00a8b15984a719ccd776dd7263dc616ee2469cc, things went a little bit better.
With file
First Initialization

Potentially Cached

Definitely Cached

Without file
First Initialization

Potentially Cached

Definitely Cached

At this point, I think this would fast, but I would want to achieve 140ms less or ideally, 100ms less, but for now, I'm delighted with this result.


Redis::delete is deprecated

[x] I was not able to find an open or closed issue matching what I'm seeing.
[x] This is not a question. (Questions should be asked on chat (Signup here) or our forums.)

Provide a narrative description of what you are trying to accomplish.
Code to reproduce the issue
every call of $redis->delete() in here:
https://github.com/zendframework/zend-cache/blob/master/src/Storage/Adapter/Redis.php
Expected results
not throwing deprecation errors
redis delete is deprecated. should be replaced with del https://redis.io/commands/del
(using redis 3.2 on my server)
Actual results
ErrorException
File
/app/vendor/zendframework/zend-cache/src/Storage/Adapter/Redis.php:359

Message
Function Redis::delete() is deprecated

Stack trace
#0 /app/vendor/sentry/sentry/lib/Raven/Breadcrumbs/ErrorHandler.php(34): Bootstrap\Bootstrap->exceptionErrorHandler(8192, 'Function Redis:...', '/app/vendor/zen...', 359, Array)
#1 /app/vendor/sentry/sentry/lib/Raven/ErrorHandler.php(127): Raven_Breadcrumbs_ErrorHandler->handleError(8192, 'Function Redis:...', '/app/vendor/zen...', 359, Array)
#2 /app/vendor/sentry/sentry/lib/Raven/Breadcrumbs/ErrorHandler.php(34): Raven_ErrorHandler->handleError(8192, 'Function Redis:...', '/app/vendor/zen...', 359, Array)
#3 /app/vendor/sentry/sentry/lib/Raven/Breadcrumbs/ErrorHandler.php(34): Raven_Breadcrumbs_ErrorHandler->handleError(8192, 'Function Redis:...', '/app/vendor/zen...', 359, Array)
#4 /app/vendor/zendframework/zend-cache/src/Storage/Adapter/Redis.php(359): Raven_Breadcrumbs_ErrorHandler->handleError(8192, 'Function Redis:...', '/app/vendor/zen...', 359, Array)
#5 /app/vendor/zendframework/zend-cache/src/Storage/Adapter/AbstractAdapter.php(1174): Zend\Cache\Storage\Adapter\Redis->internalRemoveItem('64fd5d37681d43d...')
#6 /app/vendor/zendframework/zend-session/src/SaveHandler/Cache.php(112): Zend\Cache\Storage\Adapter\AbstractAdapter->removeItem('64fd5d37681d43d...')
#7 [internal function]: Zend\Session\SaveHandler\Cache->destroy('64fd5d37681d43d...')
#8 /app/vendor/zendframework/zend-session/src/SessionManager.php(333): session_regenerate_id(true)

@gabrovex Would you like provide PR with this change? Thanks!

@gabrovex @webimpress This happens since redis extension v5.0.0. As a workaround, you can downgrade to 4.3.0. My PR should fix that behaviour.

https://travis-ci.org/zendframework/zend-cache/jobs/565853773#L1058
Build on PHP 5.6 fails because of this. I'll try to find some time to check this out.

@ppaulis

This happens since redis extension v5.0.0. As a workaround, you can downgrade to 4.3.0.

This is actually very good point. Now just wonder if by this change we are not introducing BC Break. We do not have any requirements on Redis version so I guess we should add. Probably also we would need update tests to run with previous version of Redis (4.x, maybe even 3.x?)

@ppaulis

This happens since redis extension v5.0.0. As a workaround, you can downgrade to 4.3.0.

This is actually very good point. Now just wonder if by this change we are not introducing BC Break. We do not have any requirements on Redis version so I guess we should add. Probably also we would need update tests to run with previous version of Redis (4.x, maybe even 3.x?)

@webimpress That's a very good point too. If the tests on all redis-server versions pass (no BC break), a new bugfix release could be made.
If, however, that's not the case, we would need to create a new major version (3.0.0 ?) with a requirement on the redis-server version? I don't see any alternative there, because the redis extension is already released.

@webimpress What I currently don't get, is, why the build fails on memcache for PHP 5.6 when I change a few calls in the Redis adapter.

@ppaulis

why the build fails on memcache for PHP 5.6 when I change a few calls in the Redis adapter

I think because other version of service/extension is now installed. I've checked latest passing build on master and for example I see there Redis 4.0.6, your build uses Redis 5.0.4 (but I guess there could be similar reason?)

Any update on this? Its stopping us for using Zend\Cache at this point with the latest PHP version

I'd suggest adding a "conflict": {"ext-redis": ">=5.0.0"} for existing versions, and fixing this (by adding "conflict": {"ext-redis": "<4.0.0"} in a new minor.


Updated README's formatting

Minor changes, made all data uniform & corrected little mistakes

[x] The PR title and message describe the solution briefly.
[x] Documentation has been updated/added if relevant

What is the current behavior? Good

What is the new behavior?
Everything works as intended
Other information?
 No
 Previewing

@rara4 thank you for the contribution, much appreciated! 😊

You're welcome !


Enhancement: Require fzaninotto/faker:^1.5

:exclamation: Blocked by https://github.com/thephpleague/factory-muffin/pull/400.
This PR

[x] requires fzaninotto/faker:^1.5 so we can use it in projects already requiring the latest versions of fzaninotto/faker
[x] fixes the test by asserting that the image url starts with the expected url

@GrahamCampbell
Would be cool if you could take a look soon at #400 and this (happy to rebase once #400 has been merged). Ideally, also tag soon?

Faker 1.5 is available in Factory Muffin 3. We can't upgrade 2.x because it's a BC break.

BC break as proven by the fact our tests broke and you had to change them.

So when is 3.0 going to be released?

3.0 is already "stable". I just don't want to tag v3.0.0 until the docs are up to standard.


[FLINK-4574] [kinesis] Strengthen fetch interval implementation in Kinesis consumer

I used Timer to implement it.
If "flink.shard.getrecords.intervalmillis" is set by default value, which is 0, the timer will schedule ShardConsumerFetcher once and run it forever.
If "flink.shard.getrecords.intervalmillis" is greater than 0, the timer will schedule ShardConsumerFetcher at a fixed ratio by using timer.scheduleAtFixedRate, which makes sure two consecutive function call would be a fixed interval.
But if the getRecords took too much time and couldn't be finished on time, ShardConsumerFetcher would log the warning and drop the next delayed task.
Ideally :
|----p1----|----p2----|----p3----|
|=====>...|====>......|====>......|
task1.........task2.........task3
task2 is delayed by task1: task2 will be dropped
|----p1----|----p2----|----p3----|
|============>..... |====>......|
task1...........................task3

@tzulitai I think it would be great to get your input on this pull request, when you find the time.

I'm definitely planning to look at this over the next few days :) Currently quite overwhelmed right now.
Thanks for all your recent work on the Kinesis connector @tony810430, and very sorry for the late reviews. Please bear with me for a little while, I'll get back to the PRs soon ;)

Thanks for working on this @tony810430!
I'd like to discuss a bit on the design before a more detailed review for this PR. If I'm correct, with this PR we will be creating a separate shard consuming thread with the Timer even if the fetch interval is 0 (which is our default). In this case, especially that it's our default, this extra thread usage seems sub-optimal to me, and I'm wondering if there's a better design for this. What do you think?

I think it would not be sub-optimal, because it still need to separate consuming thread and ingesting thread, which emits records to next operator. The reason for this implementation is because Timer can't support 0 period, so I have to make it run forever.

@tony810430 ok, thanks for your response and patience on the review.
I'll proceed with reviewing this and testing it against Kinesis either today or tomorrow.

Hi, @tzulitai
I think there is no better way if trying to use Timer to deal with 0 fetch interval. Even though I implement my own Timer scheduler, it just move the runForever inside from ShardConsumerTask. Because fetch interval is 0, there is no way to get the next expected execution time and the solution would be running the function forever.
I think a better solution is defined non-positive fetch interval differs from positive fetch interval and would be executed in another way in the document. The implementation will be two parts: the original way remains the same without runForever in ShardConsumerFetcherTask and the other is for non-positive fetch interval implemented by using a Thread running ShardConsumerFetcherTask::run() forever.
It is more reasonable for me to distinguish these two configuration and implement them by using two ways in ShardConsumer::run(). What do you think?


SRC-7; Arbitrary Asset Metadata Standard

Abstract
The following standard attempts to define the retrieval of on-chain arbitrary metadata for any Native Asset. Any contract that implements the SRC-7 standard MUST implement the SRC-20 standard.
Motivation
The SRC-7 standard seeks to enable data-rich assets on the Fuel Network while maintaining compatibility between multiple assets minted by the same contract. The standard ensures type safety with the use of an enum and an Option. All metadata queries are done through a single function to facilitate cross-contract calls.
Prior Art
The use of generic metadata was originally found in the Sway-Lib's NFT Library which did not use Fuel's Native Assets. This library has since been deprecated.
A previous definition for a metadata standard was written in the original edit of the now defunct SRC-721. This has since been replaced with the SRC-20 standard as SubId was introduced to enable multiple assets to be minted from a single contract.
The standard takes inspiration from ENS's public resolver with the use of a String as the key. This should enable human-readable keys to help minimize errors and enable the standardization of certain keys, such as "image" as opposed to an enum or u64 representation of keys.
We also take a look at existing common metadata practices such as OpenSea's Metadata Standards and seek to stay backwards compatible with them while enabling more functionality. Through the combination of String keys and various return types, both pre-defined URIs or specific attributes may be stored and retrieved with the SRC-7 standard.
Specification
Metadata Type
The following describes an enum that wraps various metadata types into a single return type. There SHALL be the following variants in the MetadataType enum:
- String: String
The String variant SHALL be used when the stored metadata for the corresponding asset and key is of the String type. The String variant MUST be used when a URI is required but MAY contain any arbitrary String.
- Int: u64
The Int variant SHALL be used when the stored metadata for the corresponding asset and key is of the u64 type.
- Bytes: Bytes
The Bytes variant SHALL be used when the stored metadata for the corresponding asset and key is of the Bytes type. The Bytes variant should be used when storing custom data such as but not limited to structs and enums.
Require Functions
fn metadata(asset: AssetId, key: String) -> Option<MetadataType>
This function MUST return valid metadata for the corresponding asset and key, where the data is either a String, Int, or Bytes variant. If no metadata exists, the function MUST return None.
Rationale
The SRC-7 standard should allow for data-rich assets to interact with one another in a safe manner.
Backwards Compatibility
This standard is compatible with Fuel's Native Assets and the SRC-20 standard. It also maintains compatibility with existing standards in other ecosystems.
Security Considerations
This standard does not introduce any security concerns, as it does not call external contracts, nor does it define any mutations of the contract state.
Example ABI
abi SRC7Metadata {
     #[storage(read)]
     fn metadata(asset: AssetId, key: String) -> Option<MetadataType>;
}

Hi,
I copied here my comment from the PR to hope its end up in the right place.
TL;DR

I suggest you engage in discussion with stakeholders like wallets and data indexers and ask what they wish to see for the metadata standard
Standard should define core metadata everyone should provide, so that assets can be properly presented in various user interfaces
Off-chain data storage is privacy and security risk, and there is less or little reason to do off-chain in 2023

Here is some feedback regarding this standard (discussed with @dmihal earlier)

Because on-chain is no longer expensive (it's not 2016 anymore) and metadata is very small, most of the data should be stored on-chain instead of an external source to avoid privacy and security issues (see here)
This information is going to be consumed by wallets, frontends and data  indexers: have you engaged to discussions on any of these groups what they want to see on the metadata standard?
Not only the interface how to fetch data, but the data content itself must be standardised
You definitely want to have a method that fetches all data attributes once (to a some certain nice round limit like 256) attributes

When attributes you want to standardise

Website
Logo, various formats

Dark background
White background
Standard dimensions like 256x256, pick whatever Apple has decided
Note that sufficiently small logo files can be directly stored inline compressed on-chain, avoiding any issues with privacy leaks and exploits when fetching images

Social media links
- Twitter
- Discord
- Telegram
- etc.
Trading related

name; keypairs where to official trade the token
to avoid users landing on a phishing site through a Google

Security incident contact email

For more inspiration, look what EtherScan, CoinGecko and TrustWalllet token repository offers as a metadata about the tokens.
Happy to help you to come up with descriptions of key metadata attributes any asset should have.


Fix Collection on Select

Do you have usecases where you want to pass the collection as positional arguments? Otherwise I would recommend to just require an array.

These are intentionally positional to make the syntax cleaner.