locustfile as configmap -> Could not find any locustfile! Ensure file ends in '.py'

Description of issue
I try to run a locust test file as config map. I have created the config map from the .py file.
However, when I deploy the helm chart, the master pod crashes with the above error.
Expected behavior
To find the locustfile that I have provided as config map [as stated in the documentation], and run without problems.
Actual behavior
The master pod crashes with error "Could not find any locustfile! Ensure file ends in '.py'"
Environment settings

OS: Linux on Vagrant
Python version: 2.7
Locust version: 0.8.1

Steps to reproduce (for bug reports)
from rabbitmq import get_client
from locust import HttpLocust, Locust, TaskSet, task, events
import pika
from random import randint,choice
from string import ascii_uppercase

class UserBehavior(TaskSet):

  host="<IP_ADDRESS>"
  port=5672
  username="guest"
  password="guest"
  vhost="/"

  # create a random string between 1000 and 10000 chars
  msg=''.join(choice(ascii_uppercase) for i in range(randint(1000,10000)))

  @task
  def hit_rabbit(self):
    min_wait = 2000
    max_wait = 5000

    # # # #
    credentials = pika.PlainCredentials(self.username, self.password)
    parameters  = pika.ConnectionParameters(self.host, self.port, self.vhost, credentials)
    connection  = pika.BlockingConnection(parameters)
    channel     = connection.channel()

    channel.queue_declare(queue='locust_queue')

    channel.basic_publish(exchange='', routing_key='route-key-1', body=self.msg)
    print("[OK] Sent msg")

    connection.close()
    # # # #

    @task
    def stop(self):
      self.interrupt()

class WebsiteUser(HttpLocust):
  # host = target_url
  task_set = UserBehavior
  # min_wait = 5000
  # max_wait = 15000

To find the locustfile that I have provided as
config map [as stated in the documentation]

I'm not sure what a config map is or which documentation you are referring to... but the Locust docs don't mention any such thing.

@wehappyfew , are you referring to this documentation?
https://github.com/helm/charts/tree/master/stable/locust

Hello @hsellik !
Yes exactly.


☂ Issue for Documentation

[x] Cleanup current outdated state
[x] Instruction: How to build a blueprint/component
[x] Operator guide: Hands-On Deploy landscaper and install first component
[ ] Concept overview

@In-Ko closed because outdated and/or copied into internal project


Manifest is not a function

Bug Report
Your Environment

Software
Version(s)

the-traveler
0.7.4

Node
8.9.1

Operating System
Ubuntu Server 16.04 LTS

Expected Behavior
Creating an instance of manifest using the path to the manifest.content file should create an sqlite3 database handle to the manifest database as specified in the documentation
Current Behavior
Process fails with the error
(node:31520) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Manifest is not a constructor
Code Sample
const Manifest = require( 'the-traveler/build/Manifest' );
const manifest = new Manifest( __dirname + '/../destiny-2/manifest.content' );
const queryResult = await manifest.queryManifest( 'SELECT name FROM sqlite_master WHERE type="table"' );

console.log( queryResult );

Other Notes
due to issue #39 I manually  unzipped the manifest, renamed it manifest.content, and placed it in the appropriate directory.

this issue persists after the closing of issue #39

I found the root of the issue. the documentation states
import Manifest from 'the-traveler/build/Manifest' which I couldn't get to work. So I used
const Manifest = require( 'the-traveler/build/Manifest' );
Looking at the source code I see that what I needed was const Manifest=require( 'the-traveler/build/Manifest' ).default
Are you able to use import statements with node 8? From what I'm reading we are still using require unless you specify the flag --experimental-modules

Right now, import statements are not supported in node 8 as far as I know. So you would need to stick to const Manifest = require( 'the-traveler/build/Manifest' ).default to use the Manifest class in node 8, and generally using the require keyword in node 8.


[proof of concept/ wip] obsm, varm refactor

@flying-sheep based on our conversation in https://github.com/theislab/scanpy/issues/562#issuecomment-483235556, I figured I'd give it a try. This branch is an attempt at replacing BoundRecArray with a MutableMapping subclass.
I think this is beneficial largely from a code simplicity point of view.

recarrays are poorly documented by numpy, and I personally find more difficult than alternatives
Everyone knows how to use a dict
This can hold sparse arrays and dataframes, unlike recarrays

Meaning @flying-sheep's suggested solution for #136 would work

This could also potentially hold any n>1 dimensional values.

This is WIP since it's not fully featured, and would definitely require a reworking of tests if implemented. One definitely broken thing is IO, but I don't think this will be difficult to solve. Currently this seems to work with the core of scanpy's api (pretty sure all the commands from to seurat based tutorial for sc.datasets.pbmc3k work).

This is pretty close to being a real implementation now. All non-Raw tests now pass for both anndata and scanpy.
Still to go:

Tests for sparse matrices, dataframes, etc. in .obsm, .varm

Xarray, zarr?

Investigate on disk representation
Think about compatibility

Since the on disk representation will likely change, how are changes to forwards compatibility handled?

Good one!
I think we should figure out how to call the type “any matrix type we support”:
>>> import numpy; numpy.ndarray.mro()
[numpy.ndarray, object]
>>> import scipy.sparse; scipy.sparse.spmatrix.mro()
[scipy.sparse.base.spmatrix, object]
>>> import zarr; zarr.Array.mro()
[zarr.core.Array, object]
>>> import xarray; xarray.DataArray.mro()
[xarray.core.dataarray.DataArray,
 xarray.core.common.AbstractArray,
 xarray.core.common.ImplementsArrayReduce,
 xarray.core.common.DataWithCoords,
 xarray.core.arithmetic.SupportsArithmetic,
 xarray.core.common.AttrAccessMixin,
 object]

So they don’t share any ancestor.
>>> {c: set(dir(c)) & {'toarray', '__array__', 'shape', 'ndim'} for c in [numpy.ndarray, scipy.sparse.spmatrix, zarr.Array, xarray.DataArray]}
{numpy.ndarray: {'__array__', 'ndim', 'shape'},
 scipy.sparse.base.spmatrix: {'ndim', 'shape', 'toarray'},
 zarr.core.Array: {'__array__', 'ndim', 'shape'},
 xarray.core.dataarray.DataArray: {'__array__', 'ndim', 'shape'}}

but they do all have shape, ndim, and either __array__ or toarray!

So we could accept anything that has that.
When slicing the whole thing and that fails, we need to emit a good error message telling people which obs exact entry in which annotation failed to be slicable.

I was thinking I'd go for more of a duck typing approach for now, with types that we explicitly support covered by tests. A large part of that is because I'd like to reduce the total amount of code in AnnData (especially base.py!), apart from tests. I'd be up for introducing that kind of abstract type once I see how it will reduce complexity/lines of code.
Question about Raw, think that we might remove it in favor of layers? I'm trying to figure out how much effort I should put into making it work (i.e., workaround vs refactor).

ABCs are duck typing.

Looks really interesting: Raw has a different functional role (different number of variables, only row slicing) than layers (same dimensions as .X, row and colum slicing).
Thoughts on treating connectivities etc. within that instead of in .uns? It's a big flaw of views on  .uns that one needs a deepcopy. And this is just due to slicing of the n_obs x n_obs graph adjacency matrices.
PS: Brave undertaking for something that is mostly about simplifying the code. But, I, of course, don't mind if you have the bandwidth. :)

Further comments (hopefully not demotivating, just to make sure we're all on the same page):

You mean n>2 and not "any n>1 dimensional values", right? obsm already holds n>1 dimensional values like X_pca etc.
We made BoundRecArray so that it can be used like a dict. It has .keys, you can del .obsm[key] and you can append etc. So, I'd be surprised if usage became even simpler as it is right now.
There could be some advantage to having a dedicated slot for dense representations and another one for sparse ones. I'd need to think a little about different arguments.

@flying-sheep  Just to preface, the aphorism for duck-typing I'm familiar with is "if it looks like a duck, and acts like a duck, it's a duck." I googled to make sure I was remembering it right, but it turns out there's a bunch of ways of saying it.
I think creating an ABC would definitely cover "looks like a duck", but I'm not sure if covers "acts like a duck". It lets us know it has those methods, but not necessarily that they result in what we want. If the types were actually subclassed from common ABC I'd be more comfortable with having expectations about their behaviour, like how we expect .update and .items to work for a mutable mapping. For now I think it's easier to say both "it should have shape", and "when I index into it, it'll return something of the shape I expect" via runtime checks.
I also don't want to get overly ambitious with this. I definitely want this to support sparse arrays, ndarrays, and dataframes. I'll worry about the others once that's accomplished. Could you point me towards the other section you mentioned? I'd be interested in seeing that, as it could give me a better idea of the areas of application.

@falexwolf Yeah, I think simplifying the code will eventually make it easier to build on top of it. I'd also like to understand how AnnDatas are initialized, and right now it's kinda hard to follow. This seems like a (relatively) simple step towards that. Also simplifying now could reduce additional complexity later, like that this could remove the need for a special attribute for sparse annotations.
I definitely think connectivities could fit in obsm. That said, adata.uns["neighbors"] has a nested structure which is useful for keeping those related values together. I'm not sure what the best way to handle this is.
On your specific points:

I was thinking that this would allow obsm to hold any obs annotations with greater than one dimension. So this includes the current n==2 case, but I'm not sure the current implementation supports n==3.
What inspired me to do this is that BoundRecArray limits the types of values more than I think is necessary for what it does. I think it could be generalized, with easier to follow code. For instance: the current __setitem__ (https://github.com/theislab/anndata/blob/81d10e2f960873c0283b87ae9f7f983100a8d675/anndata/base.py#L115) is pretty complicated. Also inspired by the pain from #138.
I'd be interested to hear the advantages. To me, I see .obsm as being array-like annotations of the observations, but I'd be interested to hear how you see it.

@falexwolf and @flying-sheep
More specifically, I've hit a snag with figuring out what to do with setting values on views of an AnnData. I'd like to get your opinions on what the spec for this is.
I've got a little code sample that could be illustrative. I think it could be useful for you to look at it and decide what you think should happen, then try and see if what does happen matches up.
import scanpy as sc
import numpy as np

pbmc = sc.datasets.pbmc68k_reduced()
v = pbmc[pbmc.obs["louvain"] == "1"]
pbmc.obsm["zeros"] = np.zeros_like(pbmc.obsm["X_pca"])
v._isview # True
v.obsm["zeros"] # What should this be?

frozen_obsm = v.obsm.copy()

v.obsm["ones"] = np.ones_like(v.obsm["ones"])
v._isview # Should this be a view?

v = pbmc[pbmc.obs["louvain"] == "1"]
v # now has "zeros" field
v.obsm = frozen_obsm # What should this do? Is v still a view? If so, what's in `obsm`?
v.obsm = v.obsm # What about this?

I think simplifying the code will eventually make it easier to build on top of it.

Everything else above also makes a ton of sense! I'm aware that anndata/base.py is way too long and was developed way too iteratively just to satisfy certain feature requirements. It can be much simplified and better structured.

setting values on views

You shouldn't be able to set any value on views, except for writing to the data matrix. This is what is crucially necessary in the backed case (where you see a portion of the huge data on disk in a view and want to be able to modify it) and adds a lot of convenience in the memory-case (otherwise one wouldn't be able to write into the data matrix using indexing based on obs and var names...
But all other write operations are meaningless (I think), and that's why DictView, ArrayView etc. exist. They listen to whether you are modifying them and auto-generate a copy if you do so. This is also inspired by the "auto-view-switch" in numpy arrays.
v.obsm["ones"] = np.ones_like(v.obsm["ones"])
v._isview # Should this be a view?

This is should not be a view anymore (btw, there should be a public v.isview). Should also be in tests; if it's a view, it's a bug. I think this should also answer your other questions.

I definitely think connectivities could fit in obsm. That said, adata.uns["neighbors"] has a nested structure which is useful for keeping those related values together. I'm not sure what the best way to handle this is.

Cool! There is a special case about n_obs x n_obs sparse matrices though, as they need to be both row- and column-sliced, which is evidently not the default for fields in .obsm. Don't worry about a nested structure, we don't need it. The idea is to continue to have each tool 'x' write a dict of annotations to .uns['x'] as long as these things are nothing that we would want to slice. The only case where this occurs is sc.pp.neighbors with annotations distances and connectivities; params would stay in .uns['neighbors'], for instance.

Setting values on views

backed case

I hadn't been thinking of this. I'll have to look into it for a bit, but will mostly talking about the in-memory case in this comment.

You shouldn't be able to set any value on views, except for writing to the data matrix.

It's good to have this formalized. The current implementation of the .obsm property looks like it has code to update the parent if it's a view, though it doesn't seem to do this in practice.
I do think there could be API benefits to allowing modification via views. Some examples:
# Run a pca just on expression data
sc.pp.pca(adata[:, adata.var["modality"] == "rna"])

# Differential expression, but only within one batch
sc.tl.rank_genes_groups(adata[adata.obs["batch"] = "1"], groupby="celltype")

Another case is raised by theislab/scanpy#612 and is relates to

"auto-view-switch" in numpy arrays

If the model is numpy arrays, should adata[subset, :].{some subsetable attribute} = {values} work? This could be nice for when you do want to modify some array inplace, but want the indexing semantics of an AnnData.
As a side note, I personally don't like the automatic switching in numpy arrays. I think it should be explicit whether an operation returns a view or a copy but for numpy arrays the following code (without type checking) is ambiguous: b = a[idx].
Subsetting connectivities
That's a good point. Subsetting a graph can be pretty different from subsetting a matrix, in the sense that you can subset a graph on edges or nodes.

But all other write operations are meaningless (I think), and that's why DictView, ArrayView etc. exist. They listen to whether you are modifying them and auto-generate a copy if you do so.

FYI: that’s called a “copy on write” structure, or so I think COWDict would be a more fitting name here.

@flying-sheep by "this section" I meant whatever you were referring to by this:

we’re using a check like that already somewhere else

Mostly regular instance checks for different things we support:
https://github.com/theislab/anndata/blob/81d10e2f960873c0283b87ae9f7f983100a8d675/anndata/base.py#L51-L61
https://github.com/theislab/anndata/blob/81d10e2f960873c0283b87ae9f7f983100a8d675/anndata/base.py#L831-L838
https://github.com/theislab/anndata/blob/81d10e2f960873c0283b87ae9f7f983100a8d675/anndata/base.py#L743-L758

Just removed BoundRecArrays entirely from the codebase. Probably still have some cleaning up to do however. Next on my list is adding some tests to make sure views are working, and getting .raw to work. I also want to come up with a better name that DictM, since that was when I wasn't sure if I'd try to merge this. I'm thinking something like: AlignedAnnot?
Questions

There was one case I wasn't clear on for views:

pbmc = sc.datasets.pbmc68k_reduced()
v = pbmc[pbmc.obs["louvain"] == "1"]
pbmc.obsm["zeros"] = np.zeros_like(pbmc.obsm["X_pca"])
v._isview # True
v.obsm["zeros"] #  What should this return? Currently is a KeyError

Do you have opinions on changing the on disk representation? I think making obsm and varm groups instead of datasets will be necessary for supporting more container types, particularly sparse arrays.

Update on on-disk representation:
The reader and writer handle obsm/varm being a mapping (encoding it as a group) totally fine. Additionally sparse matrices seem to be handled with current implementation. Dataframes are being read wrong, however.

@flying-sheep I've been thinking about this bit, and looking over the code for AnnDataLayers and thinking this could be simplified even more. I'm thinking these could easily share a parent class (likely an ABC) since they're very similar concepts, and I think handling the "which axes to subset by" can be dealt with by passing subsetting indices as a tuple. Plus it'd be nice if the APIs were the similar, and this would help enforce that.
I've got a skeleton of the class implementations here. Mind taking a look?

Looks great! I just have design considerations (many of which are actually about the current design):

COW, not View. Views propagate changes made to them to the original. Since we have the behavior that we create a copy once we change something, we don’t have views but “copy on write” (aka COW) structures.

Use None instead of 'NoneType'. It’s a bit confusing that None gets used as a synonym for it’s own type, but they decided on that

I don’t understand what idx is for, can you explain? Will it just hold the COW’s subset into the actual structure? Then it should be part of the AlignedCOWMixin, right? You should also should fill in the type parameters for Tuple[?]. Tuple[str, ...] [sic] means that it’s a tuple of strings of unspecified length.

Remove as_dict. dict(thing) will automatically do the same as dict(thing.items()) if thing has an .items() method (which we have, being a MutableMapping)

I don’t think we need to override abstract properties as properties. I think we can just do
class LayersBase(AlignedMapping):
    attrname = "layers"

It'd be good to represent how modification happens in the name, but I think it's also important to make it clear it's a subset of an existing object (I don't think we necessarily want "View"s of the whole object). Maybe something like ImmutableView?

I always forget that. That type(None) returns NoneType doesn't help.

Currently AnnDataLayers has _oidx and _vidx attributes. self.idx is basically (self._oidx, self._vidx) since indexing into an array like a[oidx, vidx] is the same as a[(oidx, vidx)]. For obsm, idx would be a tuple like (oidx, slice(None)).

I'm keeping the contents untyped, because you can index with a bunch of different types, and some objects have different indexing semantics (like xarray.Dataset). The types could potentially be parametric, but I think it's simpler to keep it generic for now.

as_dict is only there for backwards compat right now, though I think that can be broken. Another thing that I think could be broken is the attribute access to the layers.

Can do, I'd only checked if that'd work (it doesn't) for instance attributes.

Good point. COWView? ViewCOW? Then we can leave the view/isview and other function names as long as we document what we really mean.

:+1:

Understood!

Very good opportunity to document this vague “bunch of different types” right now. We can use a TypeVar with all possible index types in it.

Yes. Let’s break them and add an explicit error function instead:
def as_dict(self):
    raise AttributeError(f'{type(self).__name__} does not have `as_dict() anymore. Use `dict(obj)`')
```

:+1:

@flying-sheep, mind taking another look? I've managed to get everything running on the same code, and having the tests pass
Things left to do:

Name things so COW is clear
Maybe as_dict should return a warning for now? It breaks a lot of things if I remove it.
It'd probably be good to add tests making sure copies are being made when expected
General code cleanup (probably new files, doesn't make sense for everything to be in layers.py)

Thanks for the review @flying-sheep ! Just getting to it now.
Two general design questions:
New files
I want to reorganize the code from base.py and layers.py. I'm thinking something like:

base/anndata.py

Contains AnnData and Raw

base/views.py

ArrayView, DictView, etc.

base/alignedarrays.py

AxisArrays, Layers

COW views
This has two parts:

Should adata[{some_indexing}].obsm[key][{other_indexing}] return one of the ArrayView, DataFrame, other subclasses of _ViewMixin? This isn't the case for layers currently, but would make sense to me.
If views from anndata are all COW, could we just document that and keep the simple names? I think we could just make it clear that the view is attached to an AnnData in the __repr__.

The docs could use a user guide, and this would be in a subsetting section.

This whole thread is awesome.
Re Renaming: I think the most frequently encountered convention is to call the central submodule core.py (or _core.py) not base.py. Hence, I would do

_core.py
_views.py
_alignedarrays.py

Or the same thing in a directory _core. Right now, we get autosuggest for all the modules that aren't actually meant for the user:

Re COW views: Both of your suggestions: Agreed! :wink:

How about:
core/anndata.py
core/views.py
core/alignedarrays.py

This would similar to how pandas, xarray, and MDAnalysis do it. I think I prefer non _private names for modules containing exported classes.
Also, why does your autocomplete look nicer than mine? I don't get any type info.

If views from anndata are all COW, could we just document that and keep the simple names

I’m always for simple, but never for misleading. “view” has a very specific meaning in python, which is “changes made to this object actually change the object this is a view of”.
When I do
>>> view = obj.view()
>>> view[foo] = bar
>>> obj[foo]
KeyError: ...

I’m very confused.

Also, why does your autocomplete look nicer than mine? I don't get any type info.

Because he uses a version of IPython where this PR has been merged: ipython/ipython#11506

I'm not sure it's so cut and dry in the stdlib:

memoryviews  can be mutable
While there are MappingViews, they don't share and api with the parent mapping. This seems like a different definition of view to me – more like currying maybe?
There is types.MappingProxyType, which fits the immutable view definition I think we're agreed on, but it's not called a view.

I'm also not sure we can claim total analogy with numpy. Where a is a np.ndarray:

a[idx] = val always modifies in place. adata[idx] = val or adata[idx].attr = val generally does not.
The following is ambiguous. For anndata a similar operation would return False unless values were set on X, where I believe it should always be True

a = np.ones((10, 10))
b = a[idx]
b[:] = 0
np.all(a[idx] == 0)

On the .view method, I think that seems fair. However that's not how users interact with AnnData, since it doesn't have a .view method. This has made me think I should rename the AlignedMapping.view method, or at least make it private. Could be AlignedMapping.subset?
We could potentially go with something like Subset which avoids the more loaded View in general? On the other hand, it would be really nice to not break current naming schemes. I think I'm also still pro allowing modifications of views in the future, as it could allow even more data to live on disk in the backed case.
@falexwolf, you mentioned before the inspiration of numpy's  "auto-view-switch" for how views worked. Would you mind elaborating on that/ linking to (possibly external) design docs that describe this? My initial understanding of this was that memory was shared when it's easy, but copies were made when sharing was hard.

I think I've got that PR merged for mine (I'm on the current version), I've just got jedi disabled until path completion works with it (I assume it's still broken if https://github.com/ipython/ipython/issues/11530 hasn't been closed?)

Codecov Report

Merging #140 into master will increase coverage by 1.25%.
The diff coverage is 92%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
+ Coverage   73.61%   74.87%   +1.25%
==========================================
  Files          11       11
  Lines        2009     2022      +13
==========================================
+ Hits         1479     1514      +35
+ Misses        530      508      -22

Impacted Files
Coverage Δ

anndata/layers.py
91.72% <91.36%> (+12.77%)
:arrow_up:

anndata/base.py
76.28% <94.44%> (+0.35%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc7f3cb...5df2ddb. Read the comment docs.

Codecov Report

Merging #140 into master will increase coverage by 1.25%.
The diff coverage is 92%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
+ Coverage   73.61%   74.87%   +1.25%
==========================================
  Files          11       11
  Lines        2009     2022      +13
==========================================
+ Hits         1479     1514      +35
+ Misses        530      508      -22

Impacted Files
Coverage Δ

anndata/layers.py
91.72% <91.36%> (+12.77%)
:arrow_up:

anndata/base.py
76.28% <94.44%> (+0.35%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc7f3cb...5df2ddb. Read the comment docs.

Turns out having views of mapping return _ViewMixin subclasses becomes more complicated than I expected, since _ViewMixin subclasses are assumed to be attributes of an AnnData instead of values of attributes. I think this can be worked around by saying view_args is either Tuple[AnnData, str] or Tuple[AnnData, str, str]. If it's the latter, the second str is used as a key. How does this sound?
This partially solves the deep copy problem for uns. Something else I realized (which I think @falexwolf will like), is that having a mapping where values are array-like objects where both axes align to one axis of the parent is a pretty trivial subclass of what I've done already. I think this would fit the behavior you've described previously for cell graphs.

@flying-sheep, I've been thinking about the renaming, and I think I'd like to keep this PR to non-breaking changes. My goal here is just some backend centralization of code, and adding a few features, but not yet worthy of a breaking version bump.
By splitting out that discussion we can also get more opinions on appropriate naming and behavior, which I think would be valuable. We could also bundle any api changes (like swapping out isview) with any other api changes we'd like to make.
Here's what my goals for this PR are:

Code reorganization, and consolidation of redundant code for mappings with aligned values
Increased consistency of api, mostly that views have copy-on-write behavior
Work towards a more flexible AnnData by allowing more types of array-like annotations for obsm, varm, and layers.

I think fully implementing this is out of scope for this PR, since I'd like IO changes to be separate.

What I think is left to do:

[ ] Return _SetItemMixin derived classes from operations like adata.obsm["x"][idx] and test they work as expected
[ ] Add narrative docs specifying that views are copy on write
[ ] Remove demonstrative {obs,var}p attribute

Kinda related things that could go here

Fix for #148, since I've already changing code that doesn't fit copy-on-write semantics
Maybe fix #145, since I'm already touching the code here: https://github.com/theislab/anndata/pull/140#discussion_r281048609

Issues this could close

#142
#136 (since sparse arrays can just go in {obs,var}m)
#45

Changing the type names is a non-breaking change, as they’re private types and aren’t documented. Anyone doing from anndata import ArrayView now does so at their own risk.

Oh, I'd thought you'd wanted to rename .isview also? Then it could be fine.
Also, isn't ArrayView one we wouldn't want to rename? Based on Alex's comments above, don't views have a mutable .X?
Either way, it could happen in another PR. I'd also like to play around with making as much of AnnData be backed, and having views on them all be mutable. If that goes well, we might not want to change the name.

Also, isn't ArrayView one we wouldn't want to rename? Based on Alex's comments above, don't views have a mutable .X?

I didn’t think about it, it was just an example :sweat_smile:

I wanna try. I’ll give a shot at explaining my rationale for this:
If your dataset is really big, it’s not just X that’s gonna use a lot of memory. So are any layers, obsm, varm, etc. If we can already deal with arrays being backed on disk, then it probably shouldn’t matter which array it is. I also think I’m just as likely (if not more) to modify any of the other arrays as I am X.
I also think it'd be convenient if scanpy would do the work of translating indices when I want to modify some data in the object. I think this is what was being tried in #148.

Just getting back to this, (had my confirmation, got real sick right after), but I think it's getting close to done.
Were we going to go back to 3.5 compatibility for scanpy? If not, are we gonna keep 3.5 compatibility for anndata?

I'm fine dropping 3.5 compat for AnnData.

Question about this PR. I'd like to merge it soon, but it will add features that aren't totally supported.
You can now put a dataframe or sparse matrix in obsm, varm, and layers, but those won't survive a roundtrip to disk. What's the best way to handle this until then? I'm thinking setting an unsupported value should either be a warning or an error.

I'm thinking setting an unsupported value should either be a warning or an error.

That is a perfect solution! I'm fine if we go ahead with this. Any necessity to release 0.6.21 before?

Since it's a big code change, I think it'd be nice to have it sit on master for a little bit before a release. Just a chance to catch any bugs I've missed.
I also ended up getting started on the io parts, and have made better progress than I expected. It's possible these can come out together. I'll have to work on it for another day or two to know for sure.

Great. Completely agree that we shouldn't move quick with the next release once this is on master. Should your other PR (obs_vector) go into a release 0.6.21 before or is it not necessary?

It would be great to have a release of the obs_vector stuff first, so I can merge some long standing PRs in scanpy.
I think this is pretty much ready to go, but I think it's going to have to be merged from another PR since there's only so much time I'm willing to spend on managing a git history.

Good, I'll make a release as soon as Volker has answered some trivial questions around read_loom(..., cleanup...). I'll let you know. Should be later today.
Feel free to merge all of this stuff here in any way you like after 0.6.21 is made. 🙂


Parsing Telgram 6.3.0 DB Error

I'm trying to use teleparser on cache4.db of Telegram 6.3.0 and it doesn't seem to work.
I run the program with the following command: python3 teleparser.py cache4.db out_folder and it gives me this error:
https://pastebin.com/g6AwV0Mz
out_folder is empty :(

Hi @ufgf ,
indeed it generates and error, most likely a bad serialized object. It is possible for you to provide the specific blob of data?
If you set verbosity, it will tell you which table,line and column generates the error.
Just copy and paste here for reference
construct.core.ConstError: Error in path (parsing) -> media -> message_media_structures -> media -> message_media_web_page -> webpage -> web_page_structures -> webpage -> cached_page -> page_structures -> cached_page -> vector_sig_photo
parsing expected 481674261 but parsed<PHONE_NUMBER>

Hey, any update on this? I am seeing this error as well.


[Bug] Can't install Canopy SDK

Is this a new bug?

[X] I believe this is a new bug
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
I can't install canopy SDK
Expected Behavior
Successful install
Steps To Reproduce
pip install canopy-sdk
Relevant log output
simsketch@Elons-MacBook-Pro rag % python --version
Python 3.11.0
simsketch@Elons-MacBook-Pro rag % pip install canopy-sdk
Collecting canopy-sdk
  Using cached canopy_sdk-0.8.1-py3-none-any.whl.metadata (18 kB)
Collecting fastapi<1.0.0,>=0.93.0 (from canopy-sdk)
  Using cached fastapi-0.110.0-py3-none-any.whl.metadata (25 kB)
Collecting gunicorn<22.0.0,>=21.2.0 (from canopy-sdk)
  Using cached gunicorn-21.2.0-py3-none-any.whl.metadata (4.1 kB)
Collecting jsonschema<5.0.0,>=4.2.0 (from canopy-sdk)
  Using cached jsonschema-4.21.1-py3-none-any.whl.metadata (7.8 kB)
Requirement already satisfied: openai<2.0.0,>=1.2.3 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from canopy-sdk) (1.14.2)
Collecting pandas==2.0.0 (from canopy-sdk)
  Using cached pandas-2.0.0-cp312-cp312-macosx_10_9_universal2.whl
Collecting pandas-stubs<<IP_ADDRESS>,>=<IP_ADDRESS>814 (from canopy-sdk)
  Using cached pandas_stubs-<IP_ADDRESS>316-py3-none-any.whl.metadata (9.5 kB)
Collecting pinecone-client<4.0.0,>=3.0.0 (from canopy-sdk)
  Using cached pinecone_client-3.2.1-py3-none-any.whl.metadata (16 kB)
Collecting pinecone-text<0.9.0,>=0.8.0 (from canopy-sdk)
  Using cached pinecone_text-0.8.0-py3-none-any.whl.metadata (10 kB)
Collecting prompt-toolkit<4.0.0,>=3.0.39 (from canopy-sdk)
  Using cached prompt_toolkit-3.0.43-py3-none-any.whl.metadata (6.5 kB)
Collecting pyarrow<15.0.0,>=14.0.1 (from canopy-sdk)
  Using cached pyarrow-14.0.2-cp312-cp312-macosx_11_0_arm64.whl.metadata (3.0 kB)
Requirement already satisfied: pydantic<3.0.0,>=2.0.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from canopy-sdk) (2.6.4)
Collecting python-dotenv<2.0.0,>=1.0.0 (from canopy-sdk)
  Using cached python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Collecting sentencepiece<0.2.0,>=0.1.99 (from canopy-sdk)
  Using cached sentencepiece-0.1.99.tar.gz (2.6 MB)
  Preparing metadata (setup.py) ... done
Collecting sse-starlette<2.0.0,>=1.6.5 (from canopy-sdk)
  Using cached sse_starlette-1.8.2-py3-none-any.whl.metadata (5.4 kB)
Collecting tenacity<9.0.0,>=8.2.1 (from canopy-sdk)
  Using cached tenacity-8.2.3-py3-none-any.whl.metadata (1.0 kB)
Collecting tiktoken<0.4.0,>=0.3.3 (from canopy-sdk)
  Using cached tiktoken-0.3.3-cp312-cp312-macosx_14_0_universal2.whl
Requirement already satisfied: tokenizers<0.16.0,>=0.15.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from canopy-sdk) (0.15.2)
Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from canopy-sdk) (4.66.2)
Collecting types-jsonschema<5.0.0,>=4.2.0 (from canopy-sdk)
  Using cached types_jsonschema-<IP_ADDRESS>40311-py3-none-any.whl.metadata (1.8 kB)
Collecting types-pyyaml<<IP_ADDRESS>,>=<IP_ADDRESS> (from canopy-sdk)
  Using cached types_PyYAML-<IP_ADDRESS>40311-py3-none-any.whl.metadata (1.5 kB)
Collecting types-tqdm<5.0.0,>=4.61.0 (from canopy-sdk)
  Using cached types_tqdm-<IP_ADDRESS>40106-py3-none-any.whl.metadata (1.5 kB)
Collecting uvicorn<1.0.0,>=0.20.0 (from canopy-sdk)
  Using cached uvicorn-0.29.0-py3-none-any.whl.metadata (6.3 kB)
Collecting python-dateutil>=2.8.2 (from pandas==2.0.0->canopy-sdk)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas==2.0.0->canopy-sdk)
  Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.1 (from pandas==2.0.0->canopy-sdk)
  Using cached tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)
Requirement already satisfied: numpy>=1.21.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from pandas==2.0.0->canopy-sdk) (1.26.4)
Collecting starlette<0.37.0,>=0.36.3 (from fastapi<1.0.0,>=0.93.0->canopy-sdk)
  Using cached starlette-0.36.3-py3-none-any.whl.metadata (5.9 kB)
Requirement already satisfied: typing-extensions>=4.8.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from fastapi<1.0.0,>=0.93.0->canopy-sdk) (4.10.0)
Requirement already satisfied: packaging in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from gunicorn<22.0.0,>=21.2.0->canopy-sdk) (24.0)
Collecting attrs>=22.2.0 (from jsonschema<5.0.0,>=4.2.0->canopy-sdk)
  Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema<5.0.0,>=4.2.0->canopy-sdk)
  Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting referencing>=0.28.4 (from jsonschema<5.0.0,>=4.2.0->canopy-sdk)
  Using cached referencing-0.34.0-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema<5.0.0,>=4.2.0->canopy-sdk)
  Using cached rpds_py-0.18.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (4.1 kB)
Requirement already satisfied: anyio<5,>=3.5.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from openai<2.0.0,>=1.2.3->canopy-sdk) (4.3.0)
Requirement already satisfied: distro<2,>=1.7.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from openai<2.0.0,>=1.2.3->canopy-sdk) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from openai<2.0.0,>=1.2.3->canopy-sdk) (0.25.2)
Requirement already satisfied: sniffio in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from openai<2.0.0,>=1.2.3->canopy-sdk) (1.3.1)
Collecting types-pytz>=2022.1.1 (from pandas-stubs<<IP_ADDRESS>,>=<IP_ADDRESS>814->canopy-sdk)
  Using cached types_pytz-2<IP_ADDRESS>40203-py3-none-any.whl.metadata (1.5 kB)
Requirement already satisfied: certifi>=2019.11.17 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from pinecone-client<4.0.0,>=3.0.0->canopy-sdk) (2024.2.2)
Requirement already satisfied: urllib3>=1.26.5 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from pinecone-client<4.0.0,>=3.0.0->canopy-sdk) (1.26.15)
Collecting mmh3<4.0.0,>=3.1.0 (from pinecone-text<0.9.0,>=0.8.0->canopy-sdk)
  Using cached mmh3-3.1.0-cp312-cp312-macosx_10_9_universal2.whl
Collecting nltk<4.0.0,>=3.6.5 (from pinecone-text<0.9.0,>=0.8.0->canopy-sdk)
  Using cached nltk-3.8.1-py3-none-any.whl.metadata (2.8 kB)
Requirement already satisfied: requests<3.0.0,>=2.25.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from pinecone-text<0.9.0,>=0.8.0->canopy-sdk) (2.31.0)
Collecting types-requests<3.0.0,>=2.25.0 (from pinecone-text<0.9.0,>=0.8.0->canopy-sdk)
  Using cached types_requests-<IP_ADDRESS>40311-py3-none-any.whl.metadata (1.8 kB)
Collecting wget<4.0,>=3.2 (from pinecone-text<0.9.0,>=0.8.0->canopy-sdk)
  Using cached wget-3.2-py3-none-any.whl
Collecting wcwidth (from prompt-toolkit<4.0.0,>=3.0.39->canopy-sdk)
  Using cached wcwidth-0.2.13-py2.py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: annotated-types>=0.4.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from pydantic<3.0.0,>=2.0.0->canopy-sdk) (0.6.0)
Requirement already satisfied: pydantic-core==2.16.3 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from pydantic<3.0.0,>=2.0.0->canopy-sdk) (2.16.3)
Requirement already satisfied: regex>=2022.1.18 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from tiktoken<0.4.0,>=0.3.3->canopy-sdk) (2023.12.25)
Requirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from tokenizers<0.16.0,>=0.15.0->canopy-sdk) (0.21.4)
Requirement already satisfied: click>=7.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from uvicorn<1.0.0,>=0.20.0->canopy-sdk) (8.1.7)
Requirement already satisfied: h11>=0.8 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from uvicorn<1.0.0,>=0.20.0->canopy-sdk) (0.14.0)
Requirement already satisfied: idna>=2.8 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.2.3->canopy-sdk) (3.6)
Requirement already satisfied: httpcore==1.* in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.2.3->canopy-sdk) (1.0.4)
Requirement already satisfied: filelock in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers<0.16.0,>=0.15.0->canopy-sdk) (3.13.1)
Requirement already satisfied: fsspec>=2023.5.0 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers<0.16.0,>=0.15.0->canopy-sdk) (2024.3.1)
Requirement already satisfied: pyyaml>=5.1 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers<0.16.0,>=0.15.0->canopy-sdk) (6.0.1)
Requirement already satisfied: joblib in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from nltk<4.0.0,>=3.6.5->pinecone-text<0.9.0,>=0.8.0->canopy-sdk) (1.3.2)
Requirement already satisfied: six>=1.5 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas==2.0.0->canopy-sdk) (1.16.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from requests<3.0.0,>=2.25.0->pinecone-text<0.9.0,>=0.8.0->canopy-sdk) (3.3.2)
Collecting urllib3>=1.26.5 (from pinecone-client<4.0.0,>=3.0.0->canopy-sdk)
  Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Using cached canopy_sdk-0.8.1-py3-none-any.whl (105 kB)
Using cached fastapi-0.110.0-py3-none-any.whl (92 kB)
Using cached gunicorn-21.2.0-py3-none-any.whl (80 kB)
Using cached jsonschema-4.21.1-py3-none-any.whl (85 kB)
Using cached pandas_stubs-<IP_ADDRESS>316-py3-none-any.whl (156 kB)
Using cached pinecone_client-3.2.1-py3-none-any.whl (213 kB)
Using cached pinecone_text-0.8.0-py3-none-any.whl (23 kB)
Using cached prompt_toolkit-3.0.43-py3-none-any.whl (386 kB)
Using cached pyarrow-14.0.2-cp312-cp312-macosx_11_0_arm64.whl (24.0 MB)
Using cached python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Using cached sse_starlette-1.8.2-py3-none-any.whl (8.9 kB)
Using cached tenacity-8.2.3-py3-none-any.whl (24 kB)
Using cached types_jsonschema-<IP_ADDRESS>40311-py3-none-any.whl (10 kB)
Using cached types_PyYAML-<IP_ADDRESS>40311-py3-none-any.whl (15 kB)
Using cached types_tqdm-<IP_ADDRESS>40106-py3-none-any.whl (19 kB)
Using cached uvicorn-0.29.0-py3-none-any.whl (60 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB)
Using cached referencing-0.34.0-py3-none-any.whl (26 kB)
Using cached rpds_py-0.18.0-cp312-cp312-macosx_11_0_arm64.whl (332 kB)
Using cached starlette-0.36.3-py3-none-any.whl (71 kB)
Using cached types_pytz-2<IP_ADDRESS>40203-py3-none-any.whl (5.1 kB)
Using cached types_requests-<IP_ADDRESS>40311-py3-none-any.whl (14 kB)
Using cached tzdata-2024.1-py2.py3-none-any.whl (345 kB)
Using cached urllib3-2.2.1-py3-none-any.whl (121 kB)
Using cached wcwidth-0.2.13-py2.py3-none-any.whl (34 kB)
Building wheels for collected packages: sentencepiece
  Building wheel for sentencepiece (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [66 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-312
      creating build/lib.macosx-10.9-universal2-cpython-312/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.macosx-10.9-universal2-cpython-312/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.macosx-10.9-universal2-cpython-312/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.macosx-10.9-universal2-cpython-312/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.macosx-10.9-universal2-cpython-312/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      ./build_bundled.sh: line 21: cmake: command not found
      ./build_bundled.sh: line 22: nproc: command not found
      ./build_bundled.sh: line 22: cmake: command not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/k3/z0jnlt3n49s3r306n75cdnk80000gn/T/pip-install-zf4nohiy/sentencepiece_4d97de02867a4c38899f72ed1af7de7e/setup.py", line 167, in <module>
          setup(
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/private/var/folders/k3/z0jnlt3n49s3r306n75cdnk80000gn/T/pip-install-zf4nohiy/sentencepiece_4d97de02867a4c38899f72ed1af7de7e/setup.py", line 87, in build_extension
          subprocess.check_call(['./build_bundled.sh', __version__])
        File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.99']' returned non-zero exit status 127.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
ERROR: Could not build wheels for sentencepiece, which is required to install pyproject.toml-based projects

Environment
- **OS**: Sonoma 14.4
- **Language version**: Python 3.11.0
- **Canopy version**: ?

Additional Context
No response

I was able to solve by passing -m flag
https://github.com/pinecone-io/canopy/issues/240
https://stackoverflow.com/a/40409521/1579789
py -m pip install canopy-sdk


clasp logs --watch --simplifed returns log with timestamp

It's highly possible that I am doing something wrong. But I'm receiving the timestamps when passing in both flags of --watch and --simplified.
I am not sure if this is the behavior that was intended.
Thank you!
Expected Behavior
$ clasp logs --watch --simplified
DEBUG      N/A {"message":"hello there!","serviceContext":{"service":"..."}}

Actual Behavior
$ clasp logs --watch --simplified
timestamp >= "2020-07-17T02:44:40.069Z"
timestamp >= "2020-07-17T02:44:46.071Z"
DEBUG      N/A {"message":"hello there!","serviceContext":{"service":"..."}}
timestamp >= "2020-07-17T02:44:46.071Z"

Specifications

Node version (node -v): v10.10.0
Version (clasp -v): 2.3.0
OS (Mac/Linux/Windows): Mac

After doing some cursory git-blaming from within Github, I believe this issue was introduced by https://github.com/google/clasp/pull/778, though, I may be incorrect.

After doing some cursory git-blaming from within Github, I believe this issue was introduced by https://github.com/google/clasp/pull/778, though, I may be incorrect.

This issue cause is likely here:
https://github.com/google/clasp/blob/ee70a6ae743d3b28b27b2c5cc6ca3e70c64f465b/src/commands/logs.ts#L212-L215
These lines are run every time ˋfetchAndPrintLogsˋ is invoked.
That part of the code would need some revision so that under ˋwatchˋ option it only displays once.
Anyone wants to draft a PR?

@leeyspaul   Just to be sure the issue still exist, can you try replacing @google/clasp 2.3.0 with forked-clasp which is more advanced?
npm uninstall -g @google/clasp
npm install -g forked-clasp


ShouldComponentUpdate-like behavior for store

I was wondering if there is a way to obtain a "componentShouldUpdate"-like behavior for store edits.
From what I understand if I set a value on the store the render is trigger even if the new value is the same as the old value, is there a way to avoid it?
// current store situation store = {test: 2}
store.test = 2 // -> will trigger re-render

From what I understand if I set a value on the store the render is trigger even if the new value is the same as the old value, is there a way to avoid it?

It won't re-render if the value did not change with a strict equality check (===). See this minimal repro (just spam the reset button a few times).
You can also use raw() from the underlying @nx-js/observer-util package to update/read the store without the added reactivity but we don't promote that a lot because it is a low-level API.
import { raw } from '@nx-js/observer-util';

// this re-renders everything that uses `myStore.prop`
myStore.prop = 12

// this does not re-render anything
raw(myStore).prop = 12

You also have the option to use shouldComponentUpdate or memo in your component and do complex store comparisons there.
And we may add some low-level hooks in the future to handle this use-case. We will add a couple of low-evel hooks for stores in the very next release which can intercept default language and reactivity behavior, I will think about adding a hook for this too. These will be part of the public and documented API but they will be mainly intended for library authors and for devs how would like to build custom solutions on top of easy-state.

Great! Thanks for the explanation!  It is really a great library. We use it on a big application and the only problem it is to understand what changes triggered the re-render of a specific component.
I am looking forward for the next release to be able to hook on a console.log debugger to better debug re-renders!
Thanks for the great work!


remove quotes around -n test variable

using "" around the variable causes -n to test an empty string instead of an undefined (or null) string

don't know why I added "in OS X" - in linux the DISPLAY var is always present and your probably always using xvfb ...


Camera calibration format

Decide the format to use for the camera calibration file. Note that we must be able to deal with:

a camera with fixed intrinsics parameters
a camera with intrinsics parameters changing over time (f, pp, and k3)
a camera rig composed of N (N = {1,2,3}) cameras

So far for the camera localization (voctree or cctag based) a simple txt file is used using a format like this
// the structure of the file is
// int #image width
// int #image height
// double #focal
// double #ppx principal point x-coord
// double #ppy principal point y-coord
// double #k0
// double #k1
// double #k2

as explained here https://github.com/poparteu/openMVG/blob/voctree_localizer/src/openMVG/dataio/IFeed.cpp#L18

Why not directly use the cereal serialization of the existing camera class

yes, the json it's the natural and obvious solution, but we need to define how to deal with a video stream in which the camera calibration is changing over time. So we need something like frame ID -> camera calibration, for each frame or just whenever it changes. I don't know

The SfM_Data format already support that.
Why not create a sfm_data file for the video, with different intrinsic for each frame.
So you save, load it for an entire video...

yes, the thing i forgot to mention (i will added to the description) is that we deal with a rig of 3 cameras. So we need to take also that into account. (ie one of the camera may change the parameters while the other does not). BTW did you do anything to deal with a rig of cameras, ie like a class modeling the rig or other?

In the SfM_Data class to make some intrinsics shared or not, so you can do whatever you want.
Regarding the rig:
I have started a functional draft here: https://github.com/pmoulon/openMVG/blob/develop_rig_subposes/src/openMVG/cameras/Camera_Pinhole.hpp#L136
In which you must update the subpose with known data.
Additionnaly a RigView concept have been created:
https://github.com/pmoulon/openMVG/blob/develop_rig_subposes/src/openMVG/sfm/sfm_view.hpp#L73
The idea is to store additional info like the RIG id & subchannel id

cool, that could be useful.
If i understand correctly a rig is composed of several Rig_Pinhole_Intrinsic, each having a subpose wrt some reference camera. And then in sfmdata each view is like a normal view but it also contains the reference to the rig ID (in case there are several rigs) and an its internal ID inside the rig

You got it.
I think it was a flexible solution.
The other solution was to make all intrinsic have a subpose and let it bet Id by default (but this solution impose to check always if the subpose is Id or not).
I have preferred to make a new class first to test and ensure that the design fit the needs.

Ok thanks!


Select Team list container should scale in width based on browser window width #14197

Summary
changes done for the issues mentioned in the ticket

Ticket Link
Fixes https://github.com/mattermost/mattermost-server/issues/14197
JIRA https://mattermost.atlassian.net/browse/MM-23749
Related Pull Requests

Has server changes (please link here)
Has redux changes (please link here)
Has mobile changes (please link here)

Screenshots

overflow-y addition on the mobile view

Before the change when the max-width was 400px

after the max-width is set to 800px

Thanks @sridhar02, looks good from my end! Adding @andrewbrown00 for a UX review

/update-branch

I noticed login screen email/password fields width has changed on the test server.
Is this intentional?

@jgilliam17,actullay it was mentioned in the task to change the width of the teams to 800px, I have changed it and it is effecting the sign-in also as they are using the same styles for both the teams and the sign-in pages.

@sridhar02 thanks for the contribution here 👍
Changes to the Teams page looks good ✅

Item to resolve

[ ] Revert the Login page back the previous width, it's too wide

Current
Expected (previous)

Thank you @sridhar02
Team list page looks good.
@andrewbrown00 found two more instance where field width was increased. Should this also be reverted? Team URL screen actually looks better with the wider field, where most of the url is visible.
Create a Team

Team URL screen

@andrewbrown00 found two more instance where field width was increased. Should this also be reverted? Team URL screen actually looks better with the wider field, where most of the url is visible.

Thanks @jgilliam17, yes, all non-team page instances should be reverted 🙂

@jgilliam17 @andrewbrown00 I have reverted the changes and made changes that will be applicable to the team container u can check it.

@jgilliam17 @andrewbrown00 I have reverted the changes and made changes that will be applicable to the team container u can check it.

Looks good from my end. Thanks for quick updates 🙂 , I'll let @jgilliam17 do her review too.

Thank you @sridhar02


updates Localytics SDK

Localytics SDK is on version 3.6.1

By analyzing the blame information on this pull request, we identified @f2prateek to be a potential reviewer.

@f2prateek :eyes:


Fixed resource address in CosmosException.

Fixed resourceAddress in CosmosException
Added new API to expose regions contacted on CosmosDiagnostics
TODO: @simplynaveen20 has a fix for regionsContacted API in ClientTelemetry PR, once that is in, it will fill up the regionsContacted for Gateway mode as well. Skipped that fix in this PR to avoid future conflicts

Sample resourceAddress and regionsContacted diagnostics:
ResourceAddress sample DIRECT mode:
Exception is : NotFoundException{userAgent=azsdk-java-cosmos/4.9.0-beta.1 MacOSX/10.15.7 JRE/11.0.6, error={"Errors":["Resource Not Found. Learn more: https://aka.ms/cosmosdb-tsg-not-found"]}, resourceAddress='rntbd://cdb-ms-prod-westus2-fd31.documents.azure.com:14353/apps/e1145062-6507-492f-845f-f448225c36f6/services/55789df2-d46c-4435-9bc8-1d2ff3266202/partitions/8dedad13-a762-4b1d-b782-a38817f67371/replicas/132491704112230548s/, statusCode=404, message=["Resource Not Found. Learn more: https://aka.ms/cosmosdb-tsg-not-found"], ...
ResourceAddress sample GATEWAY mode:
resourceAddress='https://paged-flux-test-account-westus2.documents.azure.com:443/dbs/RxJava.SDKTest.SharedDatabase_20201110T150115_uRJ/colls/dbcd0b24-c2e4-41ed-b01e-2a0534ff3461/docs/cac7dadd-0f6f-4d7a-9490-c9494bf20019, statusCode=404, message=Entity with the specified id does not exist in the system. More info: https://aka.ms/cosmosdb-tsg-not-found, ...
RegionsContacted sample:
2020-11-06 13:32:32,013       [TestNG-method=directDiagnostics-1] INFO  TestSuiteBase - Regions contacted: [https://paged-flux-test-account-westus2.documents.azure.com:443/]

Closes #17279
Closes #16819

/azp run java - cosmos - tests

/azp run java - cosmos - tests


publish a new release on pypi?

Hi! It's been about a year since the last release — and my previous ask for it ;) — and I was wondering if folks had bandwidth to publish a new (v2.0.4?) release to pypi. It would be great to get unblocked on upgrading scikit-learn to v0.20. Thanks in advance!

I'm all for pushing a new release.  Who has access to create a new release other then me?  It would also be great to have someone review and merge in the two open PRs before the new release.

That's a good question... @pakelley or @b4hand , maybe?

I don't have permission to merge or release a package, but I'd be happy to help review those PRs if you want @matt-peters

Would either of you like to have permissions to merge PRs and publish releases? If so... @matt-peters , would you be able to grant them? I'd volunteer but I'm not confident that I have enough bandwidth to maintain another project... Maybe I can be a backup plan? :)

Yeah, I'd be happy to help out with that if @matt-peters is open to it

I looked at both open PRs, but I'm not sure if either is ready to merge. It's probably better to just go ahead and create a new release. I can grant permissions to the project if @matt-peters approves.

I'm all for adding additional collaborators with write access to the repo, and ability to push a new release to pypi.  My bandwidth for maintaining this is pretty low at this point, but it is still fairly actively used as far as I know.

Hi @matt-peters , just following up on this. Could you grant repo write access to @pakelley, and make him a maintainer of the package on pypi? If you'd like an additional pair of helping hands, I volunteer as tribute, and will coordinate with Patrick.

I've granted @bdewilde and @pakelley write access to the repo.

I can grant access to pypi, but I'll need your usernames there. I couldn't find either of you listed as @bdewilde and @pakelley.

Thanks a ton, @b4hand ! On PyPi, I'm @bjdewilde.

bjdewilde has been added to pypi.

Alright! Let's see now about publishing a minor 2.0.4 release. Is there anything special that has to be done for the cython files? If not, here's a checklist:

[ ] write up a changelog (and add a CHANGES.md file to the repo, maybe?)
[ ] bump package version in setup.py
[ ] tag a release on the github repo
[ ] build source dist and wheels for PY3 (and PY2?) via python setup.py sdist bdist_wheel
[ ] upload the distribution archives to pypi via python -m twine upload

Anything I've missed here?

I didn't have my own PyPi account before, but I created one (@pakelley). I'll be away for the rest of this week, but I can help out with preparing the release next week 👍

Awesome, thanks everyone for taking this on! I can't remember the details regarding the cython files, but don't think there was anything special as they are distributed with the package and built locally on installation.

Probably a good idea to put the release instructions in the README just so we don't forget later. It's always a pain to try to figure it out when you just want to make a release quickly.

pakelley is added to pypi as well

Just a heads-up: I've gotten through all the steps except for publishing the release on PyPi and tagging it here on GitHub. I built a source distribution as well as PY2 + PY3 wheels, and all existing tests pass, so I'm assuming that everything is okay.
Apologies in advance if I've screwed something up. 🙏

PyPi: https://github.com/dragnet-org/dragnet/releases/tag/2.0.4
GitHub: https://github.com/dragnet-org/dragnet/releases/tag/2.0.4


Code Gen failing for React.

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v16.13.0
Amplify CLI Version
7.4.1
What operating system are you using?
Amazon Linux
Amplify Categories
api
Amplify Commands
codegen
Describe the bug
Codegen fails when trying to run saying it can't find the file:
Code generation failed with the following error
ENOENT: no such file or directory, open '/home/ec2-user/environment/testingcodegen/unicornsports/amplify/backend/api/unicornsport/build/schema.graphql'.

Reproduction steps

amplify init

| Name: unicornsports
| Environment: dev
| Default editor: Visual Studio Code
| App type: javascript
| Javascript framework: react
| Source Directory Path: src
| Distribution Directory Path: build
| Build Command: npm run-script build
| Start Command: npm run-script start

amplify add api

Sam:~/environment/testingcodegen/unicornsports (workshopify) $ amplify add api
? Select from one of the below mentioned services: GraphQL
? Choose the default authorization type for the API Amazon Cognito User Pool
Using service: Cognito, provided by: awscloudformation

 The current configured provider is Amazon Cognito.

 Do you want to use the default authentication and security configuration? Default configuration
 Warning: you will not be able to edit these selections.
 How do you want users to be able to sign in? Username
 Do you want to configure advanced settings? No, I am done.
✅ Successfully added auth resource unicornsports607701be locally

✅ Some next steps:
"amplify push" will build all your local backend resources and provision it in the cloud
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

? Configure additional auth types? No
? Here is the GraphQL API that we will create. Select a setting to edit or continue Continue
? Choose a schema template: Blank Schema

Set the schema to one provided below:
amplify push

? Do you want to generate code for your newly created GraphQL API Yes
? Choose the code generation language target javascript
? Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
? Do you want to generate/update all possible GraphQL operations - queries, mutations and subscriptions Yes
?


Bumped versions in POM files and README for Key Vault's March 2021 stable release

Missing CHANGELOG updates.

I made them in an earlier commit when merging the master branch into this one, forgot CHANGELOG changes should go here.


generate menu for docs

getting to be long now

Looks like a menu/table of contents was first added to the docs in https://github.com/mochajs/mocha/commit/7c80ce2381614809dc7c5a0969f6a26aa769b74f (Nov 2012) Hope you don't mind if I close this. :)


Can't find the sample app

In your readme, you state:
Extract the amazonaws folder from Mobile Hub package archive to android/app/src/main/java/com/amazonaws.

Could you please state where it is? I checked the Integrate button, but there is no reference to the sample app there, neither in the entire internet.

too many discrepancies between your blog and the real environment, it's almost impossible to build the  app.

You no longer need to use this, you can use AWS Amplify and enable Push in a react native app


Updated scrypt.cpp

All other variable definitions are that way!

So, uint8_t should be replaced with unsigned char?

Fine


Elastic Stack API deletes resource group during deletion

:wave:

Service: Elastic
API Version: 2020-07-01
Method: DELETE
Endpoint: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Elastic/monitors/{name}

We've noticed an issue in the Elastic Stack API where deleting the Elastic Stack will also (with no confirmation) delete the Resource Group that it is contained within.
This differs from the behaviour of every other Azure API where the parent resource is left alone - and as such this is an API bug.
It may be the case that this API is only deleting the parent Resource Group if the Elastic Stack is the only thing in the Resource Group - but I haven't tested this.

As such - can this API be updated to not delete the parent Resource Group during deletion, like other Azure API's do?
Thanks!

@keryul this is not related to SQL - this is the Elastic RP (as in Elasticsearch)

@tombuildsstuff I tested the API but I was not able to replicate the bug. The API is working as expected i.e. resource group is not being deleted. Can you recheck the request you sent and retry to see if behavior is the same?


FIX #7357, decouple ColumnSchema::{db,php}Typecast

dbTypecast and phpTypecast do not call each other anymore, so value correction or encode/decode remain and is not reverted
#7357

https://github.com/yiisoft/yii2/commit/56dbd4cefa47616a2cc2fb07680e095ca35ac5d5
Thanks!


agolgeocoder crashing application when using invalid client and secret key instead of throwing exception

in the file node-geocoder/lib/geocoder/agolgeocoder.js:86:21,
when agolgeocoder returns an error, the variable result receives a json object format, as the example below:
{     error: {     code: 400,      error: 'invalid_client_id',      error_description: 'Invalid client_id',      message: 'Invalid client_id',      details: []    } }
but, the code tries to parse the object to json again, resulting on the following error:
`
/app/node_modules/bluebird/js/release/async.js:49
fn = function () { throw arg; };
^
SyntaxError: "[object Object]" is not valid JSON
at JSON.parse ()
at /app/node_modules/node-geocoder/lib/geocoder/agolgeocoder.js:86:21
at tryCatcher (/app/node_modules/bluebird/js/release/util.js:16:23)
at Promise.successAdapter [as _fulfillmentHandler0] (/app/node_modules/bluebird/js/release/nodeify.js:23:30)
at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:601:21)
at Promise._settlePromise0 (/app/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/app/node_modules/bluebird/js/release/promise.js:729:18)
at _drainQueueStep (/app/node_modules/bluebird/js/release/async.js:93:12)
at _drainQueue (/app/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/app/node_modules/bluebird/js/release/async.js:102:5)
at Async.drainQueues (/app/node_modules/bluebird/js/release/async.js:15:14)
at process.processImmediate (node:internal/timers:483:21)
`
a possible fix would be parse the response to json only if it's of the type string, something like:
if (typeof result === 'string') {   result = JSON.parse(result); }
what do you think?

Yes it will be better, do you think you could come with a PR to fix that?

I can verify this error, but it appears to occur with even valid credentials as the token results are already parsed.

Ok @nchaulet , I will create a new PR


Cherry pick support cast signed numeric to decimal to active_release

Automatic cherry-pick of 4b3d928

Originally appeared in https://github.com/apache/arrow-rs/pull/1044: support cast signed numeric to decimal

Codecov Report

Merging #1079 (36e87a3) into active_release (31911a4) will increase coverage by 0.02%.
 Consider uploading reports for the commit 2dcdf76 to get more accurate results

@@                Coverage Diff                 @@
##           active_release    #1079      +/-   ##
==================================================
+ Coverage           82.37%   82.39%   +0.02%
==================================================
  Files                 168      168
  Lines               49134    49213      +79
==================================================
+ Hits                40474    40549      +75
- Misses               8660     8664       +4

Impacted Files
Coverage Δ

arrow/src/compute/kernels/cast.rs
94.65% <97.53%> (+0.17%)
:arrow_up:

parquet_derive/src/parquet_field.rs
65.75% <0.00%> (-0.46%)
:arrow_down:

arrow/src/datatypes/field.rs
53.37% <0.00%> (-0.31%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 31911a4...2dcdf76. Read the comment docs.


Fix and test ForwardDiff type stability

DI extensions

ForwardDiff: Make hvp fall back on the SecondOrder implementation, which is optimized for correct inner preparation.

DI tests

Test type stability for ForwardDiff across all operators

Codecov Report
Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 5.38%. Comparing base (a1d9c26) to head (a53e4ec).

Files with missing lines
Patch %
Lines

...t/DifferentiationInterfaceForwardDiffExt/onearg.jl
0.00%
6 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (a1d9c26) and HEAD (a53e4ec). Click for more details.
HEAD has 54 uploads less than BASE

Flag
BASE (a1d9c26)
HEAD (a53e4ec)

DI
46
1

DIT
10
1

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #473       +/-   ##
==========================================
- Coverage   98.50%   5.38%   -93.13%
==========================================
  Files         107      90       -17
  Lines        4291    4089      -202
==========================================
- Hits         4227     220     -4007
- Misses         64    3869     +3805

Flag
Coverage Δ

DI
8.08% <0.00%> (-90.50%)
:arrow_down:

DIT
0.14% <ø> (-98.23%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


bazel: Allow to distdir all dependencies

To use --distdir option of Bazel (which allows to use previously
fetched tarballs instead of downloading dependencies during
build), all dependencies should use http instead of git and need
to have sha256 sums specified.
Signed-off-by: Michal Rostecki<EMAIL_ADDRESS>

@lizan ?

@qiwzhang done


Next Steps should point to Encryption, not to the same page.

The suggested Next Step is pointing to the same page you are already in. Following the ToC it should point to Encryption instead.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: baccedfe-c0c9-cfd4-0aa5-406deeae32f9
Version Independent ID: baccedfe-c0c9-cfd4-0aa5-406deeae32f9
Content: Data protection in Azure - Microsoft Azure Well-Architected Framework
Content Source: well-architected/security/design-storage.md
Service: architecture-center
Sub-service: well-architected
GitHub Login: @martinekuan
Microsoft Alias: martinek

Thank you for finding this and taking the time to let us know.  @avjolsakaj has provided a fix for this in #61.  This change will likely be live in about 48 hours after that merges.


Improve deleted file experience

Improve User Experience for Deleted Files.
Fixes: Issue #23739

Screenshots and screen captures:

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

@zulipbot claim

@alya It is ready for review. Thanks.

Thanks! Next, you'll need to fill out the PR template (including screenshots), clean up your commit history, and explain how and why your approach differs from the other open PRs for this issue.

Fixed commit history.

Thanks! Next, you'll need to fill out the PR template (including screenshots), clean up your commit history, and explain how and why your approach differs from the other open PRs for this issue.

I have updated the PR with screenshots. As for the differences between other commits and this one, this approach indicates whether the file was not found due to a server error or due to the file being deleted and returns an appropriate error message. I've add more description in the PR.

@alexmv Could you please take a look when you have time and see if we should continue with #24587, or work on this PR?

@zulipbot claim

Closing as this does not have much progress towards the main complexity of the issue (https://chat.zulip.org/#narrow/stream/3-backend/topic/404s.20for.20deleted.20files).


MSL: atomic writes to Texture3D issues

Hi, I've noticed a couple more issues with the workaround for atomic writes to textures in MSL - the first is that with argument buffers enabled for the following HLSL:
RWTexture3D<int> g_gridTexture : register(u0);

[numthreads(256, 1, 1)]
void gridUpdate(uint3 GTid : SV_GroupThreadID,
	uint3 gId : SV_GroupID,
	uint gIdx : SV_GroupIndex,
	uint3 dID : SV_DispatchThreadID)
{
    InterlockedAdd(g_gridTexture[gId], gIdx);
}

generates this MSL:
#pragma clang diagnostic ignored "-Wunused-variable"

#include <metal_stdlib>
#include <simd/simd.h>
#include <metal_atomic>

using namespace metal;

struct spvDescriptorSetBuffer0
{
    texture3d<int> g_gridTexture [[id(0)]];
    device atomic_uint* g_gridTexture_atomic [[id(1)]];
};

kernel void gridUpdate(constant spvDescriptorSetBuffer0& spvDescriptorSet0 [[buffer(0)]], uint3 gl_WorkGroupID [[threadgroup_position_in_grid]], uint gl_LocalInvocationIndex [[thread_index_in_threadgroup]])
{
    int _24 = atomic_fetch_add_explicit((device atomic_int*)&spvDescriptorSet0.g_gridTexture_atomic[gl_WorkGroupID], int(gl_LocalInvocationIndex), memory_order_relaxed);
}

The texture3d is shadowed by an atomic_uint*, which should obviously be an atomic_int* instead. The other more serious issue here is that it attempts to access that atomic_uint using an int3, which is obviously not going to compile. This is also problematic because it's not possible to alias a 3D texture from a buffer so there may not be too much that can be done about this anyway. Regardless, here is the source HLSL and the generated spirv and MSL:
Texture3D_atomic.zip

The texture3d is shadowed by an atomic_uint*, which should obviously be an atomic_int* instead.

If you notice, it cast the pointer to device atomic_int* in the actual atomic call. The real problem here is that it's taking the address of the pointer itself, instead of passing the pointer directly. This really needs to be addressed.

The other more serious issue here is that it attempts to access that atomic_uint using an int3, which is obviously not going to compile.

This is because we don't handle the texture3d case, at all, precisely because, as you yourself mentioned:

This is also problematic because it's not possible to alias a 3D texture from a buffer

For this case, we'd need a function which accepts the texel coordinates and returns the corresponding index into the buffer, which is how 2D textures are handled.

Would it be possible/worth having some kind of explicit error in the MSL compiler for the specific case of trying to use atomic operations with 3D textures? Because yeah, regardless of how we access into the buffer it's just not going to be possible to create that underlying buffer in the first place.

Would it be possible/worth having some kind of explicit error in the MSL compiler for the specific case of trying to use atomic operations with 3D textures?

Yeah, we probably ought to be throwing an error when trying to use atomics on an image that is not an image2D or a imageBuffer, or an image1D if we're treating 1D as 2D.


Plugin installed, logged in, but videos don't play

Bug report
I've finally gotten the netflix plugin installed under kodi 18.7, under a raspberry pi. Succeeded in getting logged in (this actually required running the authentication key on a separate machine and saving the key file-- but it worked). The netflix add on shows my user list, and each user's recently watched lists, which shows that the connection to Netflix is working. However, each time I try to run a video, the screen flashes for a second, and then... nothing. I've examined the kodi log, and see only a generic "cvideoplayer::openinputstream - error" associated with netflix. I'm at a loss. Help?
Your Environment

Netflix add-on version:  1.10.0
Operating system version/name:  Raspberry OS Linux pi-4 5.4.51-v7l+
Device model:

Used Operating system:

[ ] Android
[ ] iOS
[ ] Linux
[ ] OSX
[ ] Raspberry-Pi
[ ] Windows

Describe the bug

Expected behavior

Actual behavior

Steps to reproduce the behavior

Possible fix

Debug log

The debug log can be found from this link:
Additional context or screenshots (if appropriate)
Installation

[ ] I'm using other Netflix Repo
[ ] I'm using a different source

Other information

Screenshots

same issue happen with 1.9.0_20200912 in kodi 18.7 raspi4

same issue happen with 1.9.0_20200912 in kodi 18.7 raspi4

just happen to be working again after reinstalling widevine

I will assume that your flagging this with "Ignored rules" relates to my failure to include a debug log, although it isn't clear from your response.
To comply with your rules, please consult https://paste.kodi.tv/ocucuqokob

Whoever creates an Issue thread has the duty and responsibility to follow the instructions and not ignore them,
The instructions are in the Readme and also in the guide of the Issue thread creation.
OT Closed

ERROR: AddOnLog: InputStream Adaptive: Unable to load widevine shared library (/home/pi/.kodi/cdm/libwidevinecdm.so)

This is not a error of NF add-on but a bug of InputStream Adaptive,
you have to follow these instructions:
https://github.com/emilsvennesson/script.module.inputstreamhelper/issues/395

Got it. Reinstalled widevine for a fourth time; this time it works.

For anyone else coming here with this same problem, I just updated manually updated my Widevine using InputStream Helper.
The button is labelled "(Re)Install".
The widevine which doesn't work is "4.10.1679.0 (13310.91.0)"
I downgraded to 4.10.1610, and it gave me some weird other errors. Then I did the "(Re)Install" and it downloaded and installed version "4.10.1679.0 (13310.93.0)" <-- This one works!


Enhancement: Separate the 'Bulk' operations into a separate Solution and Nuget Package.

Available at RepoDb (>= v1.10.7) and RepoDb.SqlServer.BulkOperation (>= v1.0.0).


update jackson

I recently bumped into this bug in a project and had to exclude jackson dependency of ektorp because it's too old. Please update jackson version.

Did you try to use <dependencyManagement/> in maven pom.xml in order to use the version you like?

That's a good idea, I just used the <exclusion> mechanism actually...

Latest jackson 2.9.0 is causing the following exception with current ektorp 1.4.4:
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.SerializationConfig.withSerializationInclusion(Lcom/fasterxml/jackson/annotation/JsonInclude$Include;)Lcom/fasterxml/jackson/databind/SerializationConfig;
	at org.ektorp.impl.StdObjectMapperFactory.applyDefaultConfiguration(StdObjectMapperFactory.java:50)
	at org.ektorp.impl.StdObjectMapperFactory.createObjectMapper(StdObjectMapperFactory.java:26)
	at org.ektorp.impl.StdCouchDbInstance.<init>(StdCouchDbInstance.java:40)
	at org.ektorp.impl.StdCouchDbInstance.<init>(StdCouchDbInstance.java:32)

cc @cowtowncoder

@huxi This method was deprecated in 2.7, replaced with withPropertyInclusion(). In hindsight might have been better to wait for one more version (since next will very likely be breaking, 3.0).
But as it is, this needs to be updated.

@huxi thank you for the report, could you please fill a separate issue for this ? In the time it takes to change this, you can always write and inject your own class implementing the interface.

Please see PR #273
I bumped the dependency to the latest version of Jackson lib that does not break any test.
There is a feature in Ektorp (@DocumentReference) that does not work with Jackson 2.7.0 or later.
If you do not use this feature (I guess it is not widely used), you can upgrade Jackson to a later version in your own application.

Thanks a lot for this fix and the release of 1.5.0!
Small problem: the release and related changes aren't mentioned in changelog.md.


Shallow copy instead of Deep copy

Hello,
Sorry to open two posts at the same time :/ But I think I found a problem in the new internal filter process.
Since 5.0.0-rc2 no need to provide a copy of the list, FlexibleAdapter is doing it for us.
However there are a problem in the copy. A Shallow copy is performed instead of a Deep copy.
public void filterItems(@IntRange(from = 0) long delay) {
    if (mOriginalList == null) mOriginalList = mItems;
    filterItems(mOriginalList, delay);
}

So for example with the following code:
private synchronized void executeNotifications(Payload payloadChange) {
    Log.i("Performing %s notifications", mNotifications.size());
    mItems = mTempItems; //Update mItems in the UI Thread

Will also change the value of the objects stored in mOriginalList . So for example I have the following problem:

If a ExpandableHeaderItem was collapsed before filter process, its status (mExpanded value) isn't restored well if it is expanded during the filter process. Indeed, when we set mExpanded to true in mList to change the UI it also change the value in mOriginalList. So we don't restore the good value after the filter process.

Thanks

@LeMimit, expanded status are saved and restored after search is cleared out.
Original list and filtered list represent different status of the information, so it's ok to do a shallow copy as you name it.
Deep copy is not in the requirements, it will make the filter much slower too.


Fix broken link to former Ruby installation docs

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

I signed it!

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

nothing?

this is ok to test


Can you add more details in data collection implementation?

HI DexDeform team,
Thank you for your contribution, and I am astounded by the method you use to collect data in the absence of objects.
Is there any advice on how to build a data collection system, or will you release the code later? It's difficult to follow your work without these guidences. If you could provide some references or explanations in details, I would greatly appreciate it.

@Norweig1an I have added the leap motion tracking implementations. It shows how we convert the detected human hand poses into joint angles for the shadow robot hand. This repo is also a great resource.
After installing leap_motion/setup.py, leap_motion/tracking/main.py is a demo that creates a shadow hand environment in PyBullet, and allows you to teleoperate the hand via leap motion hand detection.
(Coming soon) I will add more documentation and support for controlling the shadow hand in our deformable simulation.

Hi, I am wondering, is your code depends on the windows system as this repo?

Leap Motion Library has undergone lots of changes. I am not sure about the latest version.
My implementation currently depends on the Windows system but should be easily adaptable to Linux if you build Leap SDK in Linux.


MeshDataTool.get_vertex_normal returns incorrect normals

Godot version
4.0.alpha16.official
System information
Windows 10
Issue description
Since v4.0.alpha15.official MeshDataTool.get_vertex_normal doesn't return correct normals for the mesh.
This happens with both imported meshes (left) and primitive meshes (right), see screenshot.
Might be related to the octahedral normal compression:
https://github.com/godotengine/godot/pull/60309/
Alpha 14:

Alpha 15 & 16:

Steps to reproduce
Run the attached project in either alpha 15 or alpha 16 and notice that the drawn mesh normals are not correct.
Minimal reproduction project
meshtool-normal-issue.zip

I didn't used MeshDataTool.get_vertex_normal, but since alpha15 I noticed that shadows appears broken. When I switched display mode to Normal Buffer, it looks like all normals was shuffled (whereas in the import preview meshes was displayed ok).
Moreover, in the scenes with the custom environment all meshes was rendered blury with a huge halo around its contours (both imported with materials and primitives without materials).
The problem was solved by deleting .godot/imported and .godot/shader_cashe from the project.

Fixed in v4.0-beta1 via commit ef4f968232d7bbf36ed8011f80c4e98dc883e7c2
Thanks for the awesome work!


chore: Adds airgapped handling of empty canvas hints

Description
Fixes airgapped condition for empty canvas state.
Fixes #Issue Number
or
Fixes Issue URL

[!WARNING]
If no issue exists, please create an issue first, and check with the maintainers if the issue is valid.

Automation
/ok-to-test<EMAIL_ADDRESS>:mag: Cypress test results

[!IMPORTANT]
🟣 🟣 🟣 Your tests are running.
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/9076942082
Commit: 71a84f22d3c2b652a47707a603bdb462dec24e42
Workflow: PR Automation test suite
Tags: @tag.Templates

Communication
Should the DevRel and Marketing teams inform users about this change?

[ ] Yes
[x] No

@rahulbarwal why are you not running "IDE" tag?


404 at /docker-cloud/builds/push-images/

I found a broken link : https://docs.docker.com/docker-cloud/builds/push-images/
I arrived on this page through: https://link.zhihu.com/?target=https%3A//docs.docker.com/v17.12/docker-cloud/builds/push-images/

Hey @irenyu
Thanks for reporting the broken link. Please report this issue on the source page where the link is located.


Move assets to a structure conducive of a third-party solution. (2/2)

a) You can just pull the changes, once committed. If you've got local changes, you'll want to stash them before you update your local repository.

b) Sure, it'll be updated soon.

a) You'll have to explain that a bit more. In my local github reposistry the files are in RoadCreator/Assets/Resourcources (or something similar). How should I suddenly make it ignore the first folder but magicly keep the files?
b) Is this ready to merge? Tell me when it is and I will.

a) That depends on whether or not you have any local uncommitted changes.
Do you have any of those?
b) It's ready to pull.
On Sat, Nov 10, 2018, 08:26 embeddedt<EMAIL_ADDRESS>wrote:

@MCrafterzz https://github.com/MCrafterzz From what I can tell
@djohns451 https://github.com/djohns451 moved everything out of Road
Creator in his repository. As soon as you download his copy by pulling in
GitHub Desktop, it will move the files for you as well.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MCrafterzz/roadcreator/pull/3#issuecomment-437583753,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQsPZzamXoOOQpKBhsO_YKmJ4wFscksaks5uttQegaJpZM4YUSxp
.

@embeddedt I think he's talking about remapping his changes on top of my
changes.
On Sat, Nov 10, 2018, 08:26 embeddedt<EMAIL_ADDRESS>wrote:

@MCrafterzz https://github.com/MCrafterzz From what I can tell
@djohns451 https://github.com/djohns451 moved everything out of Road
Creator in his repository. As soon as you download his copy by pulling in
GitHub Desktop, it will move the files for you as well.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MCrafterzz/roadcreator/pull/3#issuecomment-437583753,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQsPZzamXoOOQpKBhsO_YKmJ4wFscksaks5uttQegaJpZM4YUSxp
.

@djohns451 Ok. So he just merged your changes into proceduralintersections. Now he should pull proceduralintersections, and then push proceduralintersections.

When he pulls proceduralintersections it will merge his local changes with yours. Then git push would push the merged tree to GitHub.

@djohns451 @MCrafterzz But, looking at this specific case, there could be an issue because you have moved the files which he was working on. That might require some manual merging... I'm not sure if Git is smart enough to handle it in this case.

@MCrafterzz Do you have access to the command-line git client?

@djohns451 Should we make a Gitter room? This might get long...

@embeddedt Yeah, that's better.

@djohns451 Ok, I think you'll have to make it as you are a "Contributor".

See the README in my fork of the project -- I've got a room setup.

@djohns451 I'm there!

@MCrafterzz git push -b [whatever branch your commits are on] https://github.com/djohns451/roadcreator

(Or use SSH if you have that setup.)


Add support to reset components to the latest nightly build

This is an example of usage:
kubectl ks com reset --version nightly

/help

/help

/help

/help

/help

/help

/help

/help

/help

/help

/good-first-issue

/good-first-issue

/assign @LinuxSuRen

/assign @LinuxSuRen

/shrug

/shrug

/close

/close

/close

/close

/reopen

/reopen


fixed: setImmediate is not defined

When using module mqtt and webpack, variant setImmediate maybe defined.

https://github.com/jryans/timers-browserify/issues/25

@jryans Could you please merge this pr if appropriate?

Looks like whitespace changes are still an issue, please use 2 space indents and revert whitespace for unchanged lines.

I do need this very much, would you please spend some time on this pr @jryans

Thanks for the PR! I merged a slightly tweaked version just now.

I have published 2.0.3 which should contain the fix.

Thanks

Unfortunately, I had to revert this in 2.0.4 for breaking other Webpack users. We'll need better testing if we want to try this again.


[backport] Show pytest summary in flexCI

Backport of #8212

[automatic post] Jenkins, test this please.

Jenkins CI test (for commit 5eb160b600fff1b11f520667280f8130ba7300e0, target branch v6) succeeded!

@niboshi This pull-request is marked as st:test-and-merge, but there were no activities for the last 3 days. Could you check?

Jenkins, test this please

Jenkins CI test (for commit 5eb160b600fff1b11f520667280f8130ba7300e0, target branch v6) succeeded!

@niboshi This pull-request is marked as st:test-and-merge, but there were no activities for the last 3 days. Could you check?

Jenkins, test this please


Feature: Migrate from Axios to fetch

Current problem
Axios supports only Node.js  streams, and it doesn't work in browser properly.
Possible solution
Fetch API was added to Node.js, so we can use it on browser and client sides with streams.
Additional context
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

The plan was to migrate to the Fetch API in v1.0.0. But the API still has an experimental status in the current Node.js 18. . It makes sense to wait a bit.


Publish version 0.0.1 to maven central repository

This is in progress

etcd-java v0.0.1 has now been released to maven central.