Can't launch

I have added dependency to  commonMain but now when I launch iOS I get

Execution failed for task ':shared:compileKotlinIosSimulatorArm64'.
> Could not resolve all files for configuration ':shared:iosSimulatorArm64CompileKlibraries'.
   > Could not resolve lol.hostov.icons:straight:1.1.0.
     Required by:
         project :shared
      > No matching variant of lol.hostov.icons:straight:1.1.0 was found. The consumer was configured to find a library for use during 'kotlin-api', preferably optimized for non-jvm, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native', attribute 'org.jetbrains.kotlin.native.target' with value 'ios_simulator_arm64' but:
          - Variant 'desktopApiElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library for use during compile-time, preferably optimized for standard JVMs:
              - Incompatible because this component declares a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm' and the consumer needed a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'desktopRuntimeElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library for use during runtime, preferably optimized for standard JVMs:
              - Incompatible because this component declares a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm' and the consumer needed a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'desktopSourcesElements-published' capability lol.hostov.icons:straight:1.1.0 declares a component for use during runtime, preferably optimized for standard JVMs:
              - Incompatible because this component declares documentation, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm' and the consumer needed a library, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'metadataApiElements' capability lol.hostov.icons:straight:1.1.0 declares a library, preferably optimized for non-jvm:
              - Incompatible because this component declares a component for use during 'kotlin-metadata', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'common' and the consumer needed a component for use during 'kotlin-api', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'metadataSourcesElements' capability lol.hostov.icons:straight:1.1.0 declares a component, preferably optimized for non-jvm:
              - Incompatible because this component declares documentation for use during 'kotlin-runtime', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'common' and the consumer needed a library for use during 'kotlin-api', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'releaseApiElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library for use during compile-time, preferably optimized for Android:
              - Incompatible because this component declares a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm' and the consumer needed a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'releaseRuntimeElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library for use during runtime, preferably optimized for Android:
              - Incompatible because this component declares a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm' and the consumer needed a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'releaseSourcesElements-published' capability lol.hostov.icons:straight:1.1.0 declares a component for use during runtime, preferably optimized for Android:
              - Incompatible because this component declares documentation, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm' and the consumer needed a library, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'wasmJsApiElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library for use during 'kotlin-api', preferably optimized for non-jvm:
              - Incompatible because this component declares a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'wasm' and the consumer needed a component, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'wasmJsResourcesElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library, preferably optimized for non-jvm:
              - Incompatible because this component declares a component for use during 'kotlin-multiplatformresourcesjs', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'wasm' and the consumer needed a component for use during 'kotlin-api', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'wasmJsRuntimeElements-published' capability lol.hostov.icons:straight:1.1.0 declares a library, preferably optimized for non-jvm:
              - Incompatible because this component declares a component for use during 'kotlin-runtime', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'wasm' and the consumer needed a component for use during 'kotlin-api', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')
          - Variant 'wasmJsSourcesElements-published' capability lol.hostov.icons:straight:1.1.0 declares a component, preferably optimized for non-jvm:
              - Incompatible because this component declares documentation for use during 'kotlin-runtime', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'wasm' and the consumer needed a library for use during 'kotlin-api', as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'native'
              - Other compatible attribute:
                  - Doesn't say anything about org.jetbrains.kotlin.native.target (required 'ios_simulator_arm64')

hi, thanks for the report, but I don't have a macbook to add and test the iOS target. But you can send a pull request with this target


The built version of WP8 is wrong in many places

When building the wp8 platform, a fair number of differences arise in the https://github.com/phonegap/phonegap-app-developer/blob/master/platforms/wp8/Properties/WMAppManifest.xml file.
Here are the tags/settings that are incorrect:

<App Description="Adobe Phonegap Team"> should be  Description="The PhoneGap Developer app is a testing utility for web developers and designers using the PhoneGap framework. After installing this app on your device you will be able to connect it to the PhoneGap desktop app to instantly view and test your work."
<App ProductID>  has a different hash
<App Title="PhoneGap"> should be Title="PhoneGap Developer"
<PrimaryToken TokenID="PhoneGapToken"> should be TokenID="PhoneGap"
Show default phonegap app icon
Shows app default phonegap splash screen

It appears that <App ProductID /> having a different hash is ok. It's a GUID:
ProductID

The default value is the GUID for the project (128 bit). During the app submission process, a new product ID is inserted into the manifest file.

From: https://msdn.microsoft.com/en-us/library/windows/apps/ff769509(v=vs.105).aspx

I decided to keep the Product ID the same - may as well keep it the same.

Nice


Editor allows multiple consecutive spaces outside of indents

Currently, it's possible to put multiple spaces after (for example) a record field:

In general, I think it'll be nicest if we try to keep the code formatted the same way Roc's formatter would format it - which is to say, the only places multiple consecutive spaces would ever appear would be when indenting.
I think ideally the way this would work would be:

If I press spacebar, and there's already a space before the caret, and I'm not currently indenting something...
...we should not add a space
...but we should give the user some indication of having registered the keystroke and intentionally not added anything, e.g. displaying a brief explanation in a status bar or something.

We can figure out exactly how that indication would work separately from this issue (cc @popara - definitely a design question that fits into bigger designs!) but I've often found it confusing when I press a button in an application that normally does something, the application doesn't appear to do anything, and also doesn't tell me why...I generally think "something is broken!" and then, after pressing it a few times "...or maybe it's working as intended, and I'm not allowed to do that here? But if so, why not?" - and I think we can do better than that!

Editor is no longer being worked on, and has been removed from the repo. Closing.


Feature Request for signing up

Add 3 signup button in top-wrapper
(a) signup with google
(b) signup with facebook
(c) signup up with twitter
make use of icons from font-awesome
Style the buttons as per your wish.

Assigned, Please add screenshot/short video while commit. Thanks!


Clarify how react should be imported in an ES6 environment

The website carefully avoids mentioning how the "react" module should be imported, import React from … or import * as React from …. None of the code examples (that I was able to find) contain the import statement. Both import forms are used in the wild, but our tools disagree which one is correct:

The React source uses a default export, so the first form will become correct once React decides to provide the code as ES6 modules.
TypeScript declarations currently want you to use the second form, but you can enable a compiler option (allowSyntheticDefaultImports ) to allow it to accept the first form, for compatibility with CommonJS modules.
Flow recommends using the second form, but also allows the first form in some cases.

While I don't particularly care which form React decides on (though I dislike the default export because it makes it impossible to treeshake the module), it would be nice to document the "correct" way, so that we can already now prepare for the transition to React-as-true-ES-module.

import * as React from 'react' is the way to do it.
https://github.com/facebook/react/pull/18102


sap_hana_install: Add a comment to the top of files created with Ansible

As the result of the template processing is a file which will not be deleted after the role has finished (the file can be very useful for problem analysis or for documenting the installation steps), we should make sure that there is a comment which shows that this file has been created by the role.
See also https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template_module.html .

Solved in https://github.com/sap-linuxlab/community.sap_install/pull/69 .


proxyclient/i2c: experiment with the i2c and ti usb chip

unfortunately not enough to bring up usb in host mode...
Signed-off-by: Sven Peter<EMAIL_ADDRESS>

Can you reformat this with spaces? See the .editorconfig for reference.

sure, done.


first attempt to support awkward arrays

First draft at supporting awkward arrays.
As discussed with @ivirshup this would be useful for Squidpy @hspitzer (discussed also here #609 ) and potentially @Zethson EHR project.
Here's a walkthrough showcasing some basic functionality we could use for sub observation annotations (e.g. spatial coordinates of rna-molecules/segmentations).

Details
import numpy as np
import scanpy as sc
import squidpy as sq
import matplotlib.pyplot as plt
from numpy.random import default_rng
from sklearn.datasets import make_blobs
import awkward as ak

adata = sq.datasets.visium_hne_adata()
adata = adata[:10, :].copy()
adata.obsm["spatial"] = (
    adata.obsm["spatial"] - np.std(adata.obsm["spatial"], 0)
) / np.mean(adata.obsm["spatial"], 0)

# generate data
obs_list = []
rng = default_rng(42)
for idx in adata.obs_names.values:
    coord, _ = make_blobs(
        n_samples=rng.integers(5, 15),
        cluster_std=0.02,
        centers=adata[idx].obsm["spatial"],
        random_state=42,
    )
    obs_list.append(coord)

sub_obs = ak.Array(obs_list)

fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].set_aspect("equal")
ax[0].scatter(x=adata.obsm["spatial"][:, 0], y=adata.obsm["spatial"][:, 1])
for i in range(adata.shape[0]):
    ax[1].scatter(
        x=sub_obs[i, :, 0],
        y=sub_obs[i, :, 1],
    )
    ax[1].axis("equal")

adata.obsm["sub_obs"] = sub_obs  # sub_obs is an awkward array
adata_subset = adata[:5]  # let's subset, it also works for `.copy()`

fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].set_aspect("equal")
ax[0].scatter(
    x=adata_subset.obsm["spatial"][:, 0], y=adata_subset.obsm["spatial"][:, 1]
)
for i in range(adata_subset.shape[0]):
    ax[1].scatter(
        x=adata_subset.obsm["sub_obs"][i, :, 0],
        y=adata_subset.obsm["sub_obs"][i, :, 1],
    )
    ax[1].set_aspect("equal")

What fails atm is adata.concatenate() because of errors in reindexing of alternate axis
https://github.com/theislab/anndata/blob/286bc7f207863964cb861f17c96ab24fe0cf72ac/anndata/_core/merge.py#L478
in awkward arrays there is no shape attribute so when array.shape[0] is needed we can resort to len(awkward_array) but for array.shape[1] we should (probably) simply skip it and concatenate. In awkward it would be like this:
ak.concatenate([sub_obs, sub_obs])
>>> <Array [[[1.25, 0.833], ... [0.697, 1.22]]] type='20 * var * var * float64'>

TODO

[x] wait for #554 to get merged and work on IO
[x] switch to v2 api of awkward. v1 EOL is in four months and bugs are not getting fixed already now
[ ] address newly added TODOs in the code
[ ] Implement outer joins during concatenation. Either implement inner/outer join logic or raise a corresponding warning
[x] AwkwardArrayView with behavior

Tests

[x] test that assigning an awkward v1 array fails
[x] test the dim_len function
[ ] tests for concatenating awkward array with None or MissingVal
[x] test awkward support in .uns
[x] add specific test for more complex awkward arrays
[x] add element-wise IO tests (check different types e.g. union, optiontype, ...)
[x] Add tests for getting and setting awkward arrays (e.g. adata.obsm["x"] = awk)
[ ] Add tests for concatenating

[ ] Proper reindexing during concatenation
[ ] When a.obsm["x"] is a awkward array and b.obsm["x"] isn't (error?)
[ ] What's the right fill value for when join=outer and a.obsm["x"] exists but b.obsm["x"] doesn't
[ ] Are values being compared correctly when merge="equal"

Docs

[ ] add examples
[x] show which types of awkward arrays are supported (need to be "regular" in the aligned dimensions)

Really cool @giovp and everyone involved. I can already see a couple of use-cases for ehrapy and am eager to see this move forward. Do you expect the support for akward array to also require modifications for all/many of Scanpys algorithms is this mostly a drop-in replacement?
CC @imipenem

Do you expect the support for akward array to also require modifications for all/many of Scanpys algorithms is this mostly a drop-in replacement?

I doubt it, also can't think of any scanpy function that could use such representation out of the box. We'd have multiple use cases in squidpy though! Essentially being able to slice/subset/concatenate and copy anndata preserving the 0-axis of an akward array would cover most fo the cases I can think of right now.

@giovp, for concat, AFAICT you can't have a multi-dimensional awkward array, so there won't be an "alternative" axis. Also, I don't think there is a logical way to concatenate an awkward array with any other kind of array, so it should probably just error. I think this could be handled similar to the all dataframe case.
Could you modify gen_adata in anndata.tests.helpers to add awkward arrays into obsm and varm? I think that'll help catch a lot of bugs.

@giovp, for concat, AFAICT you can't have a multi-dimensional awkward array, so there won't be an "alternative" axis. Also, I don't think there is a logical way to concatenate an awkward array with any other kind of array, so it should probably just error. I think this could be handled similar to the all dataframe case.

yeah you can't have multi-dimensonal akward array, but I think it would still be good to concatenate them across axis=0, and so this should be supported and hence escape the current alterante_axes check?

Could you modify gen_adata in anndata.tests.helpers to add awkward arrays into obsm and varm? I think that'll help catch a lot of bugs.
I will do that and run test locally, sorry for that

yeah you can't have multi-dimensonal akward array, but I think it would still be good to concatenate them across axis=0, and so this should be supported and hence escape the current alterante_axes check?

Ahh yeah, this is what I meant. Basically have a case for all elements being awkward arrays.

Ok, at this stage concat works both for inner and outer on obsm, and subsetting works for varm.

Example
import numpy as np
import scanpy as sc
import squidpy as sq
import matplotlib.pyplot as plt
from numpy.random import default_rng
from sklearn.datasets import make_blobs
import awkward as ak
import pandas as pd
from cycler import cycler

sc.set_figure_params()

adata = sq.datasets.visium_hne_adata()
varm = adata.obsm["spatial"][15:30, :]
adata = adata[:10, :15].copy()
adata.obsm["spatial"] = (
    adata.obsm["spatial"] - np.std(adata.obsm["spatial"], 0)
) / np.mean(adata.obsm["spatial"], 0)

adata.varm["spatial"] = varm.copy()
adata.varm["spatial"] = (
    adata.varm["spatial"] - np.std(adata.varm["spatial"], 0)
) / np.mean(adata.varm["spatial"], 0)

obs_list = []
var_list = []
rng = default_rng(42)
for idx in adata.obs_names.values:
    coord, _ = make_blobs(
        n_samples=rng.integers(5, 15),
        cluster_std=0.02,
        centers=adata[idx].obsm["spatial"],
        random_state=42,
    )
    obs_list.append(coord)

for idx in adata.var_names.values:
    coord, _ = make_blobs(
        n_samples=rng.integers(5, 15),
        cluster_std=0.02,
        centers=adata[:, idx].varm["spatial"],
        random_state=42,
    )
    var_list.append(coord)

sub_obs = ak.Array(obs_list)
sub_var = ak.Array(var_list)

def plot_points(adata, main: np.ndarray, sub: ak.Array, axis: int, cmap_name):

    fig, ax = plt.subplots(1, 2, figsize=(10, 5))
    ax[1].set_prop_cycle(
        cycler("color", plt.get_cmap(cmap_name)(np.linspace(0, 1, len(main))))
    )
    ax[0].axis("equal")
    ax[0].scatter(
        x=main[:, 0], y=main[:, 1], c="grey", edgecolors="black", s=100, linewidths=1
    )
    for i in range(adata.shape[axis]):
        ax[1].scatter(
            x=sub[i, :, 0],
            y=sub[i, :, 1],
            edgecolors="black",
            alpha=0.7,
        )
        ax[1].axis("equal")

    return

plot_points(adata, adata.obsm["spatial"], sub_obs, 0, "winter")
plot_points(adata, adata.varm["spatial"], sub_var, 1, "cool")

adata.obsm

adata.varm

Slicing and copying also works
adata.obsm["sub_obs"] = sub_obs  # sub_obs is an awkward array
adata.varm["sub_var"] = sub_var
adata_subset = adata[:5, :7]  # let's subset

If I run test locally with pytest anndata/tests no tests fails, not sure, where shall I start looking?

I think you just need to add awkward array to the testing dependencies

Also, for gen_adata to return an object with awkward arrays in it, you've got to generate them and place them here:
https://github.com/theislab/anndata/blob/286bc7f207863964cb861f17c96ab24fe0cf72ac/anndata/tests/helpers.py#L111-L121

Also, for gen_adata to return an object with awkward arrays in it, you've got to generate them and place them here:

🤦 🤦 🤦
So I'm at stage where I could fix+add tests for awkward array for ages and am happy to do so but would like to get a sense of how welcomed this PR is. As it stands, we'd have to include awkward array as a (optional) dependency and it would be a non-negligible addition to the code base (although I expected worse, with singledispatch made it quite slim).
A major thing that could be impactful is that there is no .view() or .copy() notion in awkward array, see here and here.
Overall, I think we could use it right away in Squidpy, mostly as a way to enable anndata slicing and indexing while retaining sub-obs or sub-var info.  We would not really use it to do any arithmetics or stuff like that, although it would be cool to come up with function ideas and also the fact that it support numba and jax jitting is cool.
So, to summarize, shall I go ahead? @ivirshup @hspitzer @Zethson @michalk8 ?

I'm all for it. Having more record like data has been requested a number of times.
I would note that there are some things in the awkward array api that will change (https://github.com/scikit-hep/awkward-1.0/discussions/1151), but that's mostly down the line stuff.

A major thing that could be impactful is that there is no .view() or .copy() notion in awkward array

I believe it does have copy, just not as a method. We might be able to get them to add that?

TO DO:

wait for #554 to get merged and work on IO
add specific test for more complex awkward arrays

@grst, if you wanted to try having variable length values per cell, this PR is starting to look pretty ready. Would appreciate any early feedback!

I'm just catching up to all of this, but I can answer a few questions now.

I believe it does have copy, just not as a method. We might be able to get them to add that?

There's also __copy__. We want to minimize the number of methods on ak.Array to avoid conflicts with methods that downstream libraries add through the ak.behavior mechanism.
Do you want it to deeply copy (__deepcopy__)? __copy__ only makes a new Python object wrapping the tree of data buffers; it doesn't recurse down that tree or replace any of the buffers.
For what reason do you need to copy? That would be relevant if you're going to change a buffer in place—to avoid changes in other arrays that share the same buffers. (Awkward Array operations freely share buffers as views because all of the operations treat arrays as immutable.) I don't know of other reasons to copy without changing anything.

I'd like to know more about this, though it's a lot to take in at once. Would you like to talk sometime so that I can learn more about anndata and you can ask questions about Awkward?

I would note that there are some things in the awkward array api that will change (scikit-hep/awkward-1.0#1151), but that's mostly down the line stuff.

In the three levels of abstraction for Awkward:

high-level, what data analysts see, ak.Array, ak.some_function, ak.some_other_function, slices, applying NumPy ufuncs, high-level Types, etc.
mid-level, what projects like this one building on top of Awkward's public API see, such as .layout and Forms (finer granularity than Types),
low-level, Awkward internals,

The high-level API is almost entirely unchanged, and the mid-level has some minor renamings for consistency and laziness (VirtualArray and PartitionedArray) are being removed in favor of dask-awkward. As long as you don't depend on lazy arrays, the transition will be easy or automatic.
You'll probably want to pin this implementation to awkward>=1,<2 so that if there are any method name changes needed for v2, anndata won't accidentally get an incompatible version of Awkward. After updating anndata, it can then be pinned to awkward>=2,<3. An alternative would be to try to support both with one version of anndata, which wouldn't be crazy, given the small scope of renamed functions. (We do similar things in our dependence on NumPy: it changes slowly enough that we can detect when names change like tostring → tobytes.)
Another alternative would be to just aim for version 2 now:
import awkward._v2 as ak

The last time I checked (by counting number of lines of code expected), the version 2 submodule that we're shipping inside Awkward 1.8.0 is 95% complete.

I started playing around with this and it looks really neat! :rocket:
I was wondering if it is planned to support obs with only some "awkward" columns?
adata = AnnData(
    obs=pd.DataFrame(index=["cell1", "cell2"]).assign(
        awkward_col=[ak.Array([1, 2, 3]), ak.Array([4, 5])],
        normal_col=list("AB")
    )
)

> adata.obs
      awkward_col normal_col
cell1   [1, 2, 3]          A
cell2      [4, 5]          B

I can create this object, but saving fails with

Stacktrace
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/projects/2022/anndata/anndata/_io/utils.py:214, in report_write_key_on_error.<locals>.func_wrapper(elem, key, val, *args, **kwargs)
    213 try:
--> 214     return func(elem, key, val, *args, **kwargs)
    215 except Exception as e:

File ~/projects/2022/anndata/anndata/_io/specs/registry.py:171, in write_elem(f, k, elem, modifiers, *args, **kwargs)
    167 if (
    168     hasattr(elem, "dtype")
    169     and (dest_type, (t, elem.dtype.kind), modifiers) in _REGISTRY.write
    170 ):
--> 171     _REGISTRY.get_writer(dest_type, (t, elem.dtype.kind), modifiers)(
    172         f, k, elem, *args, **kwargs
    173     )
    174 else:

File ~/projects/2022/anndata/anndata/_io/specs/registry.py:24, in write_spec.<locals>.decorator.<locals>.wrapper(g, k, *args, **kwargs)
     22 @wraps(func)
     23 def wrapper(g, k, *args, **kwargs):
---> 24     result = func(g, k, *args, **kwargs)
     25     g[k].attrs.setdefault("encoding-type", spec.encoding_type)

File ~/projects/2022/anndata/anndata/_io/specs/methods.py:347, in write_vlen_string_array(f, k, elem, dataset_kwargs)
    346 str_dtype = h5py.special_dtype(vlen=str)
--> 347 f.create_dataset(k, data=elem.astype(str_dtype), dtype=str_dtype, **dataset_kwargs)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/h5py/_hl/group.py:161, in Group.create_dataset(self, name, shape, dtype, data, **kwds)
    159         group = self.require_group(parent_path)
--> 161 dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
    162 dset = dataset.Dataset(dsid)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/h5py/_hl/dataset.py:159, in make_new_dset(parent, shape, dtype, data, name, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times, external, track_order, dcpl, dapl, efile_prefix, virtual_prefix, allow_unknown_filter)
    158 if (data is not None) and (not isinstance(data, Empty)):
--> 159     dset_id.write(h5s.ALL, h5s.ALL, data)
    161 return dset_id

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py/h5d.pyx:232, in h5py.h5d.DatasetID.write()

File h5py/_proxy.pyx:145, in h5py._proxy.dset_rw()

File h5py/_conv.pyx:444, in h5py._conv.str2vlen()

File h5py/_conv.pyx:95, in h5py._conv.generic_converter()

File h5py/_conv.pyx:249, in h5py._conv.conv_str2vlen()

TypeError: Can't implicitly convert non-string objects to strings

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Input In [56], in <cell line: 1>()
----> 1 adata.write_h5ad("test.h5ad")

File ~/projects/2022/anndata/anndata/_core/anndata.py:1918, in AnnData.write_h5ad(self, filename, compression, compression_opts, force_dense, as_dense)
   1915 if filename is None:
   1916     filename = self.filename
-> 1918 _write_h5ad(
   1919     Path(filename),
   1920     self,
   1921     compression=compression,
   1922     compression_opts=compression_opts,
   1923     force_dense=force_dense,
   1924     as_dense=as_dense,
   1925 )
   1927 if self.isbacked:
   1928     self.file.filename = filename

File ~/projects/2022/anndata/anndata/_io/h5ad.py:98, in write_h5ad(filepath, adata, force_dense, as_dense, dataset_kwargs, **kwargs)
     96 elif adata.raw is not None:
     97     write_elem(f, "raw", adata.raw, dataset_kwargs=dataset_kwargs)
---> 98 write_elem(f, "obs", adata.obs, dataset_kwargs=dataset_kwargs)
     99 write_elem(f, "var", adata.var, dataset_kwargs=dataset_kwargs)
    100 write_elem(f, "obsm", dict(adata.obsm), dataset_kwargs=dataset_kwargs)

File ~/projects/2022/anndata/anndata/_io/utils.py:214, in report_write_key_on_error.<locals>.func_wrapper(elem, key, val, *args, **kwargs)
    211 @wraps(func)
    212 def func_wrapper(elem, key, val, *args, **kwargs):
    213     try:
--> 214         return func(elem, key, val, *args, **kwargs)
    215     except Exception as e:
    216         if "Above error raised while writing key" in format(e):

File ~/projects/2022/anndata/anndata/_io/specs/registry.py:175, in write_elem(f, k, elem, modifiers, *args, **kwargs)
    171     _REGISTRY.get_writer(dest_type, (t, elem.dtype.kind), modifiers)(
    172         f, k, elem, *args, **kwargs
    173     )
    174 else:
--> 175     _REGISTRY.get_writer(dest_type, t, modifiers)(f, k, elem, *args, **kwargs)

File ~/projects/2022/anndata/anndata/_io/specs/registry.py:24, in write_spec.<locals>.decorator.<locals>.wrapper(g, k, *args, **kwargs)
     22 @wraps(func)
     23 def wrapper(g, k, *args, **kwargs):
---> 24     result = func(g, k, *args, **kwargs)
     25     g[k].attrs.setdefault("encoding-type", spec.encoding_type)
     26     g[k].attrs.setdefault("encoding-version", spec.encoding_version)

File ~/projects/2022/anndata/anndata/_io/specs/methods.py:544, in write_dataframe(f, key, df, dataset_kwargs)
    541 write_elem(group, index_name, df.index._values, dataset_kwargs=dataset_kwargs)
    542 for colname, series in df.items():
    543     # TODO: this should write the "true" representation of the series (i.e. the underlying array or ndarray depending)
--> 544     write_elem(group, colname, series._values, dataset_kwargs=dataset_kwargs)

File ~/projects/2022/anndata/anndata/_io/utils.py:220, in report_write_key_on_error.<locals>.func_wrapper(elem, key, val, *args, **kwargs)
    218 else:
    219     parent = _get_parent(elem)
--> 220     raise type(e)(
    221         f"{e}\n\n"
    222         f"Above error raised while writing key {key!r} of {type(elem)} "
    223         f"to {parent}"
    224     ) from e

TypeError: Can't implicitly convert non-string objects to strings

Above error raised while writing key 'awkward_col' of <class 'h5py._hl.group.Group'> to /

Using an awkward array instead of a dataframe for obs doesn't work either:
adata = AnnData(
    obs=ak.Array({"awkward_col": [[1, 2, 3], [4, 5]], "normal_col": list("AB")})
)

Stacktrace
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [58], in <cell line: 1>()
----> 1 adata = AnnData(
      2     obs=ak.Array({"awkward_col": [[1,2,3], [4,5]], "normal_col": list("AB")})
      3 )

File ~/projects/2022/anndata/anndata/_core/anndata.py:291, in AnnData.__init__(self, X, obs, var, uns, obsm, varm, layers, raw, dtype, shape, filename, filemode, asview, obsp, varp, oidx, vidx)
    289     self._init_as_view(X, oidx, vidx)
    290 else:
--> 291     self._init_as_actual(
    292         X=X,
    293         obs=obs,
    294         var=var,
    295         uns=uns,
    296         obsm=obsm,
    297         varm=varm,
    298         raw=raw,
    299         layers=layers,
    300         dtype=dtype,
    301         shape=shape,
    302         obsp=obsp,
    303         varp=varp,
    304         filename=filename,
    305         filemode=filemode,
    306     )

File ~/projects/2022/anndata/anndata/_core/anndata.py:497, in AnnData._init_as_actual(self, X, obs, var, uns, obsm, varm, varp, obsp, raw, layers, dtype, shape, filename, filemode)
    494                 raise ValueError("`shape` is inconsistent with `var`")
    496 # annotations
--> 497 self._obs = _gen_dataframe(obs, self._n_obs, ["obs_names", "row_names"])
    498 self._var = _gen_dataframe(var, self._n_vars, ["var_names", "col_names"])
    500 # now we can verify if indices match!

File ~/anaconda3/envs/test_awkward/lib/python3.9/functools.py:888, in singledispatch.<locals>.wrapper(*args, **kw)
    884 if not args:
    885     raise TypeError(f'{funcname} requires at least '
    886                     '1 positional argument')
--> 888 return dispatch(args[0].__class__)(*args, **kw)

File ~/projects/2022/anndata/anndata/_core/anndata.py:114, in _gen_dataframe(anno, length, index_names)
    108     if index_name in anno:
    109         return pd.DataFrame(
    110             anno,
    111             index=anno[index_name],
    112             columns=[k for k in anno.keys() if k != index_name],
    113         )
--> 114 return pd.DataFrame(anno, index=pd.RangeIndex(0, length, name=None).astype(str))

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/pandas/core/frame.py:708, in DataFrame.__init__(self, data, index, columns, dtype, copy)
    705 if not isinstance(data, (abc.Sequence, ExtensionArray)):
    706     if hasattr(data, "__array__"):
    707         # GH#44616 big perf improvement for e.g. pytorch tensor
--> 708         data = np.asarray(data)
    709     else:
    710         data = list(data)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/highlevel.py:1351, in Array.__array__(self, *args, **kwargs)
   1326 def __array__(self, *args, **kwargs):
   1327     """
   1328     Intercepts attempts to convert this Array into a NumPy array and
   1329     either performs a zero-copy conversion or raises an error.
   (...)
   1349     cannot be sliced as dimensions.
   1350     """
-> 1351     return ak._connect._numpy.convert_to_array(self.layout, args, kwargs)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/_connect/_numpy.py:13, in convert_to_array(layout, args, kwargs)
     12 def convert_to_array(layout, args, kwargs):
---> 13     out = ak.operations.convert.to_numpy(layout, allow_missing=False)
     14     if args == () and kwargs == {}:
     15         return out

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/operations/convert.py:329, in to_numpy(array, allow_missing)
    327 if array.numfields == 0:
    328     return numpy.empty(len(array), dtype=[])
--> 329 contents = [
    330     to_numpy(array.field(i), allow_missing=allow_missing)
    331     for i in range(array.numfields)
    332 ]
    333 if any(len(x.shape) != 1 for x in contents):
    334     raise ValueError(
    335         f"cannot convert {array} into np.ndarray"
    336         + ak._util.exception_suffix(__file__)
    337     )

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/operations/convert.py:330, in <listcomp>(.0)
    327 if array.numfields == 0:
    328     return numpy.empty(len(array), dtype=[])
    329 contents = [
--> 330     to_numpy(array.field(i), allow_missing=allow_missing)
    331     for i in range(array.numfields)
    332 ]
    333 if any(len(x.shape) != 1 for x in contents):
    334     raise ValueError(
    335         f"cannot convert {array} into np.ndarray"
    336         + ak._util.exception_suffix(__file__)
    337     )

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/operations/convert.py:324, in to_numpy(array, allow_missing)
    321     return out[: shape[0] * array.size].reshape(shape)
    323 elif isinstance(array, ak._util.listtypes):
--> 324     return to_numpy(array.toRegularArray(), allow_missing=allow_missing)
    326 elif isinstance(array, ak._util.recordtypes):
    327     if array.numfields == 0:

ValueError: in ListOffsetArray64, cannot convert to RegularArray because subarray lengths are not regular

(https://github.com/scikit-hep/awkward-1.0/blob/1.8.0/src/cpu-kernels/awkward_ListOffsetArray_toRegularArray.cpp#L22)

The same works flawlessly for .obsm, also with saving
adata = AnnData(
    obs=pd.DataFrame(index=["cell1", "cell2"]),
    obsm={"awkward": ak.Array({"awkward_col": [[1, 2, 3], [4, 5]], "normal_col": list("AB")})}
)
adata.write_h5ad("test.h5ad")
adata = anndata.read_h5ad("test.h5ad")
adata.obsm["awkward"]

<Array [{awkward_col: [1, ... normal_col: 'B'}] type='2 * {"awkward_col": var * ...'>

This is very interesting! We had been planning to reintroduce "Awkward Array as a Pandas column" by creating a new library that is external to awkward itself, so that

awkward would not require pandas as a dependency
the wrapper can extend pandas.api.extensions.ExtensionArray
the many methods and properties ExtensionArray requires would be in the wrapper, not ak.Array itself (ak.Array shouldn't, for example, have a shape)

I knew that you were planning to add Awkward Arrays as an array type to AnnData, but I didn't realize that it would be through Pandas (something that I didn't know about AnnData). Maybe this is how we can satisfy this requirement, and promote AnnData as the way to get Awkward Arrays in Pandas?
Regarding the usages above,
awkward_col=[ak.Array([1, 2, 3]), ak.Array([4, 5])]

is not scalable: this length-2 list represents something that scales with the size of the dataset, and ak.Array has a large per-object overhead. (The tree describing the data structures in the array scales with the complexity of the data type, but not with the number of entries in the data. So if you have a lot of ak.Array objects, that type description is duplicated, but if you have one ak.Array object with lots of entries, it's not. The same could be said of NumPy arrays, which have shape, strides, flags, etc. for every array object, but ak.Arrays are more complex than np.ndarrays.)
obs=ak.Array({"awkward_col": [[1, 2, 3], [4, 5]], "normal_col": list("AB")})

or
awkward_col=ak.Array([[1, 2, 3], [4, 5]]), normal_col=list("AB")

would be the right way to do it. In the first of these, "obs" is the only column[^1] and the "awkward_col"/"normal_col" structure is inside the ak.Array; in the second, "awkward_col" and "normal_col" are columns of the AnnData. For ak.Array, whether the internal structure is lists of numbers or records whose fields are lists of numbers and letters is an internal matter.
The first stacktrace seems to be in AnnData; its attempt to find an h5py writer for ak.Array fails when—maybe—some identifying code thinks it's a string and then the writing code realizes it's not a string? Awkward Arrays can be written to HDF5, but they have to be explicitly deconstructed and reconstructed, as described here. Or they could be pickled, and Awkward version 2 arrays (import awkward._v2 as ak, which I recommend developing on because version 1 will be dropped in December) supports Python 3.8+'s pickle protocol 5 with out-of-band data (a feature inherited through NumPy—they did all the work). ak.Array.__setstate__/__getstate__ (pickle) is using the method described in the "how to save to HDF5" documentation, linked above.
The second stacktrace comes from awkward itself; it's an indication that ak.Array.__array__ is being called on data that can't be converted to NumPy. The array that @grst created,
>>> array = ak.Array({"awkward_col": [[1, 2, 3], [4, 5]], "normal_col": list("AB")})
>>> array
<Array [{awkward_col: [1, ... normal_col: 'B'}] type='2 * {"awkward_col": var * ...'>

is perfectly valid and is a better pattern than [ak.Array([1, 2, 3]), ak.Array([4, 5])]. However,
>>> np.asarray(array)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/mambaforge/lib/python3.9/site-packages/awkward/highlevel.py", line 1351, in __array__
    return ak._connect._numpy.convert_to_array(self.layout, args, kwargs)
  File "/home/jpivarski/mambaforge/lib/python3.9/site-packages/awkward/_connect/_numpy.py", line 13, in convert_to_array
    out = ak.operations.convert.to_numpy(layout, allow_missing=False)
  File "/home/jpivarski/mambaforge/lib/python3.9/site-packages/awkward/operations/convert.py", line 329, in to_numpy
    contents = [
  File "/home/jpivarski/mambaforge/lib/python3.9/site-packages/awkward/operations/convert.py", line 330, in <listcomp>
    to_numpy(array.field(i), allow_missing=allow_missing)
  File "/home/jpivarski/mambaforge/lib/python3.9/site-packages/awkward/operations/convert.py", line 324, in to_numpy
    return to_numpy(array.toRegularArray(), allow_missing=allow_missing)
ValueError: in ListOffsetArray64, cannot convert to RegularArray because subarray lengths are not regular

The subarray lengths are not regular: the first entry of "awkward_col" has length 3 and the second has length 2.
>>> array["awkward_col", 0]
<Array [1, 2, 3] type='3 * int64'>
>>> array["awkward_col", 1]
<Array [4, 5] type='2 * int64'>

By contrast, it would work if the ak.Array happened to be regular:
>>> regular_array = ak._v2.Array([[1, 2], [4, 5]])
>>> np.asarray(regular_array)
array([[1, 2],
       [4, 5]])

but those are only the cases in which you could be using NumPy instead of Awkward Array, anyway.
It's interesting that .obsm (mapping) does work. It must be avoiding the cast-as-NumPy that .obs is hitting. Maybe it's pickling the values?
[^1]: I'm reading up on AnnData, having just realized that obs (and obsm) are special names with special meanings, not arbitrary column names.

Thanks for the detailed write up, as always, @jpivarski.
It would be super great if columns in dataframes could be ragged arrays via awkward array. I wonder how exactly you would want to manage this though. Would you want "nested columns"? Or should those be exposed at separate columns in the dataframe?

It's interesting that .obsm (mapping) does work. It must be avoiding the cast-as-NumPy that .obs is hitting. Maybe it's pickling the values?

We dispatch our writers on the type of the element passed. Since we can store awkward arrays in obsm, we can dispatch the writers. When I try to put an awkward array in a dataframe, it is converted to a object array. The only kind of object  dtype array we know how to write is arrays of strings, so an error is thrown.
If there was an pandas extension type for awkward arrays, this wouldn't be an issue.
One big use case for ragged columns in dataframes for us would be lists of genomic ranges, like bioconductors GRangesList object.

This is an example of how an Awkward Array can be wrapped as a Pandas column:
https://github.com/martindurant/awkward_extras/tree/main/awkward_pandas
(See also AwkwardSeries and tests.) Pandas ExtensionArrays can only be one-dimensional, so we would give it the ak.Array as an opaque object. Pandas doesn't get to see if it has any record fields—what might be considered "nested columns."
Actually, what would be best for you—to internalize the Awkward-Pandas wrapping into AnnData or to have it be a separate project that you can use? (The latter option would be three libraries: awkward, awkward-pandas, and anndata.)
@martindurant's code would need a little modernizing—it's from Oct 2020—and it can take advantage of @douglasdavis's https://github.com/ContinuumIO/dask-awkward/ .

That looks quite promising! Thanks for pointing it out.
I think it would make more sense externally. I can imagine use cases where people would want ragged arrays in their dataframe, but not AnnData.

There are obviously some thing that don't translate too well from pandas to an awkward extension array (groupby and any sorting spring to mind). Also, we ought to make an effort that any selection on an awkward column that results in one value per row should probably become a Series, and any operation resulting in a non-nested record per row could optionally become a dataframe. A direct .to_dataframe for awkward array of- or a single record may already exist.

The pandas extension sounds great! It also sounds like a lot of work though :/
How hard would it be to implement awkward array support for .X? In https://github.com/scverse/scirpy/issues/327#issuecomment-1172977130 it was suggested to store AIRR data in .X. When combining this with mudata, this would actually make a lot of sense IMO.

Hi @jpivarski,
thanks for another extensive explanation. This actually goes one step further in the discussion than I had expected, but this helps a lot for the implementation of the AIRR datastructure. This looks pretty much like what I need!

Following this comment, I gather that what you mean by ".X"

With .X I was merely referring to the "main data matrix" slot in AnnData. Currently, this can only be a two dimensional numpy array (variables x observations). This is most typically a gene expression matrix, but may contain other "omics" data as we are moving to multimodal data. My question was more directed to @giovp or @ivirshup how hard it would be to support an awkward array instead of a numpy array in this slot. In my case, the awkward array would indeed have two fixed and one variable dimension, s.t. it could be aligned to an .obs and a .var data frame and have the usual .var_names and .obs_names indices.

For reference, here's an overview of the AnnData data structure:

obs/var are pandas data frames aligned to the two axes of the .X array
obsm/varm are numpy arrays with one dimension aligned with the two axes of the .X array
.var_names and .obs_names are aliases for .var.index and .obs.index, respectively

The current draft implementation in this PR only supports awkward arrays in .obsm.

@jpivarski, one more thing I haven't fully understood is how an ak.Array compares to an ak.Record.
E.g. AIRR Rearrangement data is essentially a list of chain-dictionaries for each cell. Every cell can have 0-n chains, every chain 1-n fields, but every chain has the same fields. Every field can have different data types.
dict_cell_chain_field = {
    "cell1": [
        {"locus": "TRA", "v_call": "TRAV1", "cdr3_length": 15}, # <-- represents one chain
        {"locus": "TRB", "v_call": "TRBV1", "cdr3_length": 12},
    ],
    "cell2": [{"locus": "TRA", "v_call": "TRAV1", "cdr3_length": 13}],
    "cell3": [],
}

Now I could represent this as a record
r = ak.Record(dict_cell_chain_field)

which works and preserves the field names.
However, while I can make the first dimension regular
r = ak.to_regular(r, 0)

I can't make the third dimension regular (although all dicts have the same fields)
# AttributeError: 'awkward._ext.Record' object has no attribute 'axis_wrap_if_negative'
r = ak.to_regular(r, 2)

Stacktrace
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [127], in <cell line: 1>()
----> 1 ak.to_regular(r_cell_field_chain, 1)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/operations/structure.py:800, in to_regular(array, axis, highlevel, behavior)
    798 out = ak.operations.convert.to_layout(array)
    799 if axis != 0:
--> 800     out = ak._util.recursively_apply(
    801         out,
    802         getfunction,
    803         pass_depth=True,
    804         pass_user=True,
    805         user=axis,
    806         numpy_to_regular=True,
    807     )
    809 return ak._util.maybe_wrap_like(out, array, behavior, highlevel)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/_util.py:1236, in recursively_apply(layout, getfunction, pass_depth, pass_user, user, keep_parameters, numpy_to_regular)
   1231     else:
   1232         return transform_child_layouts(
   1233             transform, layout, depth, user=custom, keep_parameters=keep_parameters
   1234         )
-> 1236 return transform(layout, 1, user)

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/_util.py:1227, in recursively_apply.<locals>.transform(layout, depth, user)
   1224 if pass_user:
   1225     args = args + (user,)
-> 1227 custom = getfunction(layout, *args)
   1228 if callable(custom):
   1229     return custom()

File ~/anaconda3/envs/test_awkward/lib/python3.9/site-packages/awkward/operations/structure.py:786, in to_regular.<locals>.getfunction(layout, depth, posaxis)
    785 def getfunction(layout, depth, posaxis):
--> 786     posaxis = layout.axis_wrap_if_negative(posaxis)
    787     if posaxis == depth and isinstance(layout, ak.layout.RegularArray):
    788         return lambda: layout

AttributeError: 'awkward._ext.Record' object has no attribute 'axis_wrap_if_negative'

Alternatively, I could represent the same data as ak.Array
list_cell_chain_field = [
    [["TRA", "TRAV1", 15], ["TRB", "TRBV1", 12]],
    [["TRA", "TRAV1", 13]],
    [],
]

a = ak.Array(list_cell_chain_field)
# this works
a = ak.to_regular(a, 0)
a = ak.to_regular(a, 2)

(Eventually, I would probably want to switch the second and third dimension here, s.t. the first two dimensions are fixed)

Are there any benefits of one over the other?
Should AnnData support both records and arrays? I mean, named records are nice, but then again the names in the first (or first two) dimensions are handled through obs and var metadata anyway.

Tests currently fail (among other things) because https://github.com/scikit-hep/awkward/issues/1557 leads to inconsistent dimensions during concatenation. Need to update to v2 API of awkward and wait for this to be fixed.
I collected the TODOs scattered over this thread in the top-level comment.

@giovp, I switched to the V2 api of awkward, fixed a couple of issues and added more tests.
Would be great if you could look into IO at some point.

@grst @giovp, I'm a little apprehensive about having awkward arrays in X and layers. Do you think they are necessary? If they are, are they necessary in this PR – or could they happen in a separate one?

My preferred way of storing AIRR data for the new scirpy datastructure would indeed be to put them in an awkward adata.X as laid out in https://github.com/scverse/scirpy/issues/327#issuecomment-1187284292.
In fact, I already started a draft PR on the scirpy side based on this anndata branch.
What are your concerns in particular?

What are your concerns in particular?

Mostly that it's significantly more complicated.
I think this was fairly close to merging without the X and layers support, and am not sure when we include those. I would like to have a better handle of the 1d case before figuring out the 2d. Especially since there is a lot of ugly code around X that this PR would need to start touching.
From the AIRR data structure thread, I see the usefulness of having an awkward array with two regular dimensions in X. But could it happen in a second PR?

Granted for the ugly code around .X, but I don't think obsm or layers are fundamentally different -- both is handled by the aligned mapping code. @giovp's initial draft didn't verify anything when setting .obsm (for instance that the length of the awkward array matches adata.shape[0]). So a lot of the code I added for the 2d case is also required for the 1d case to work properly.

I agree these probably share a lot of code, but I think cases like this:
https://github.com/scverse/anndata/blob/fceab1b9fe4c1346f5f3bb5c795a6bdb960326ae/anndata/tests/test_base.py#L615-L626
Significantly expand the scope of this PR.
Also my model has been that awkward arrays are inherently one dimensional. I think taking time to understand the differences between the ArrayType and RegularType will be important for understanding how awkward arrays in X or layers work.

Whats the release plan for this? Are we requiring awkward 1.9.0 or awkward 2.0 for these features?
Are we marking this as experimental?

The v1 API of awkward is basically dead already (issues that break our tests not getting fixed) and v2 will be officially released in a few months. So we will require a 1.9.0rc version for now. I would mark it as experimental (at least) until the official awkward v2 is out.

If awkward v2 is still namespaces as _v2 when this releases, how will we handle this? Will we end up doing conversions to _v2, and will v1 code work with those?

In the awkward gitter, the following was suggested to be compatible with both 1.9rc and 2.x versions of awkward. We can include it in compat.py.
from importlib.metadata import version
from packaging import Version

if Version(version("awkward")).major < 2:
    import awkward._v2 as ak
else:
    import awkward as ak

File format docs need to be written. Would you like me to do that?

yes, that would be great! Maybe you could also look into the element-wise IO tests?

Maybe you could also look into the element-wise IO tests?

I would prefer for you todo this one, since I'd like to see what kinds of values you're expecting to see in awkward arrays.

@ivirshup, technically, using the current version it is possible to store awkward arrays in adata.layers (with the first 2 dims being regular) already now (thanks to the AlignedMapping being generic). Just not in .X (which we probably shouldn't implement anyway, and rather go for the solution proposed in #706).
I'm wondering if we should add back the tests for layers in that case, or if you would prefer to explicitly disallow layers being awkward, or we just leave it as an undocumented feature.

I'm conflicted on this. If it works, that's great, would be good if it were tested. Does make concatenation more complicated, and we don't quite have a solution there yet.
Also, I think it would be valuable to have a fixed length per observation or row, but I don't think we can quite do that.
But also we don't allow dataframes in layers, though I think we probably should. Dataframes are much more normal than these.
What do you think?

ok, let's talk again when concatenation works for the obsm/varm case.

Also, I think it would be valuable to have a fixed length per observation or row, but I don't think we can quite do that.

Why not? If an awkward is of regular type in the given dimension (which we can check with our dim_len implementation), it is fixed-length.

I was thinking like dim0: Regular * dim2: var * dim1: Regular. So per column, the number of entries is fixed.

Just updated to the stable v2 of awkward. A few things broke but I already fixed all of them except for one IO test-case (see https://github.com/scikit-hep/awkward/issues/2058).
Apart from that only the concatenation tests are still failing as they did before.

Seems I still need to follow up on this conversation about ak.with_name. I can do that next week.
@ivirshup, what else do we need need to get this over the finish line?

I think you wanted to have a look at why the concatenation tests are still failing.
anything to do in terms of documentation?

I'm making progress with the new scirpy data structure, so it would be good to have this released soonish.

Concatenation
Yeah, I wanted to get concatenation working. It's going meh.
I think I'll only target a very simple form of awkward arrays for testing. Basically nothing with static dimensions or nesting records.
I'm not sure I'm going to be able to get outer joins working in a sane way. I'm running into a new set of problems, and getting different results from older awkward array versions.
I'm assuming some form of outer concatenation is critical, right?
Docs
Docs would be great. Maybe a small tutorial or something with a quick demonstration of how awkward arrays can be used, and indicating that they are experimental.
Warnings
I think we should have some loud warnings saying this is experimental. Could happen on assignment? I'm concerned that some of the types that come out of whatever outer concatenation should throw errors in later versions.

I'm not sure I'm going to be able to get outer joins working in a sane way. I'm running into a new set of problems, and getting different results from older awkward array versions.

Since 2.0.6 more union types are being merged into something useful: https://github.com/scikit-hep/awkward/issues/2058
At this point, I don't think we should bother about anything but the latest awkward version.

I'm assuming some form of outer concatenation is critical, right?

The minimal use case I would like to have working concatenating two RecordTypes in .obsm. This should actually be covered by ak.concatenate without any reindexing.
I agree on docs+warnings, and also on only focusing on simple, common use-cases for now.

Since 2.0.6 more union types are being merged into something useful: https://github.com/scikit-hep/awkward/issues/2058

Perhaps the cause of: https://github.com/scikit-hep/awkward/issues/2173?

The minimal use case I would like to have working concatenating two RecordTypes in .obsm. This should actually be covered by ak.concatenate without any reindexing.

So, with matched record fields?

So, with matched record fields?

>>> a1 = ak.Array([{'a': 'aaa', 'b': 'bbb'}])
>>> a2 = ak.Array([{'b': 'bbb', 'c': 'ccc'}])

>>> ak.concatenate([a1, a2])
[{a: 'aaa', b: 'bbb'},
 {b: 'bbb', c: 'ccc'}]
----------------------
type: 2 * union[
    {
        a: string,
        b: string
    },
    {
        b: string,
        c: string
    }
]

this currently creates a union type. I'd rather like this to be an option type, but maybe this should also be solved on the awkward side.
That said, I'd be fine with that behavior for now. I'm not sure we should customize what awkward does too much.

this currently creates a union type. I'd rather like this to be an option type, but maybe this should also be solved on the awkward side.

My issue is that once you have a union field, you can't even access it anymore.

 Trying to access a union field
In [2]: import awkward as ak
   ...:
   ...: a1 = ak.Array([{'a': 'aaa', 'b': 'bbb'}])
   ...: a2 = ak.Array([{'b': 'bbb', 'c': 'ccc'}])
   ...:
   ...: result = ak.concatenate([a1, a2])
   ...:
   ...: result["c"]

---------------------------------------------------------------------------
FieldNotFoundError                        Traceback (most recent call last)

But the real issue is when you concat along obs with an outer join, and need to insert values in varm.

But the real issue is when you concat along obs with an outer join, and need to insert values in varm.

You mean, if both objects have an awkward array in varm, and one of them has more fields in a record type than the other?  I can see how this makes sense for data frames, but how is this even defined for regular 2D arrays?
Can we get away with throwing an error if they are not the same?

My issue is that once you have a union field, you can't even access it anymore.

:confused:
Horribly inefficient workaround :sweat_smile:
ak.Array(ak.to_list(a3))

Actually, I would find it intuitive if
ak.Array(ak.to_list(concatenated)) == concatenated

were always True. Shall I open an issue to discuss merging types during concatenation?

You mean, if both objects have an awkward array in varm, and one of them has more fields in a record type than the other?

Sorta but not quite.
I mean when one or more objects have an awkward array in varm, but one object has a longer set of var_names so we need to expand the awkward array.

 Here's an example with dense arrays and dataframes
In [37]: import anndata as ad, numpy as np, pandas as pd
    ...: from scipy import sparse
    ...:
    ...: a = ad.AnnData(
    ...:     X=np.ones((1, 3)),
    ...:     obs=pd.DataFrame(index=["a"]),
    ...:     var=pd.DataFrame(index=["a", "b", "c"]),
    ...: )
    ...: b = ad.AnnData(
    ...:     X=np.zeros((1, 2)),
    ...:     obs=pd.DataFrame(index=["b"]),
    ...:     var=pd.DataFrame({"col": [1, 3]}, index=["a", "c"]),
    ...: )
    ...:
    ...: c = ad.concat([a, b], join="outer", merge="only")
    ...:
    ...: display(c.X)
    ...: display(c.var)
array([[ 1.,  1.,  1.],
       [ 0., nan,  0.]])
   col
a  1.0
b  NaN
c  3.0

With your example, I think we can largely use the same logic as dataframes for merge (assuming a simple awkward array structure).

Shall I open an issue to discuss merging types during concatenation?

Here, or on the awkward array repo?

Here, or on the awkward array repo?

awkward

I mean when one or more objects have an awkward array in varm, but one object has a longer set of var_names so we need to expand the awkward array.

But currently, if there is a data frame in .varm it is just discarded? Am I missing something?

Example
a = ad.AnnData(
    X=np.ones((1, 3)),
    var=pd.DataFrame(index=["a", "b", "c"]),
    varm={"test": pd.DataFrame(index=["a", "b", "c"])},
)
b = ad.AnnData(
    X=np.zeros((1, 2)),
    var=pd.DataFrame(index=["a", "c"]),
    varm={"test": pd.DataFrame({"col": [1, 3]}, index=["a", "c"])},
)
c = ad.concat([a, b], join="outer", merge="only")
c.varm["test"]

KeyError                                  Traceback (most recent call last)
Input In [36], in <cell line: 12>()
      6 b = ad.AnnData(
      7     X=np.zeros((1, 2)),
      8     var=pd.DataFrame(index=["a", "c"]),
      9     varm={"test": pd.DataFrame({"col": [1, 3]}, index=["a", "c"])},
     10 )
     11 c = ad.concat([a, b], join="outer", merge="only")
---> 12 c.varm["test"]

File ~/projects/2022/anndata/anndata/_core/aligned_mapping.py:162, in AlignedActualMixin.__getitem__(self, key)
    161 def __getitem__(self, key: str) -> V:
--> 162     return self._data[key]

KeyError: 'test'

Here, or on the awkward array repo?
awkward

:+1:

But currently, if there is a data frame in .varm it is just discarded? Am I missing something?

It depends on the merge strategy whether it comes up:

 Example of it coming up
import anndata as ad, numpy as np, pandas as pd

a = ad.AnnData(
    X=np.ones((1, 3)),
    obs=pd.DataFrame(index=["a"]),
    var=pd.DataFrame(index=["a", "b", "c"]),
)
b = ad.AnnData(
    X=np.zeros((1, 2)),
    obs=pd.DataFrame(index=["b"]),
    var=pd.DataFrame(index=["a", "c"]),
    varm={"df": pd.DataFrame({"col": [1, 3]}, index=["a", "c"])},
)

c = ad.concat([a, b], join="outer", merge="only")

display(c.X)
display(c.var)
display(c.varm["df"])

array([[ 1.,  1.,  1.],
       [ 0., nan,  0.]])
Empty DataFrame
Columns: []
Index: [a, b, c]
   col
a  1.0
b  NaN
c  3.0

I think we can error on this for now, but will need a solution in the near future, and expect it may be difficult to come up with. This will also come up with aligning AnnDatas.

Draft PR for tutorial: https://github.com/scverse/anndata-tutorials/pull/15

@grst tests are passing!
I think we need to open some issues on behavior we want to change, like the unions from outer concatenation. I would also like to take a look over the coverage, and see what we're missing.
How's the tutorial going?

How's the tutorial going?

I can finish that beginning of next week

Congratulations!!! This is not an ordinary PR. :)


Anko doesn't have BottomAppBar

Exactly as the title says. Anko currently does not provide a dsl function to create BottomAppBar.

+1

+1 @developers please at least confirm that this is not just a matter of adding a dependency and indeed needs some work on the dsl; thanks


Installing cmapR requires R >= 4.0

Hello,
I'm trying to install cmapR on R 3.6.2 and I get this error:

devtools::install_github("cmap/cmapR")
Downloading GitHub repo cmap/cmapR@master
✓  checking for file ‘/tmp/RtmpUSPwSG/remotes10de422e46bd/cmap-cmapR-f4d3879/DESCRIPTION’ ...
─  preparing ‘cmapR’:
✓  checking DESCRIPTION meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  looking to see if a ‘data/datalist’ file should be added
─  building ‘cmapR_0.99.18.tar.gz’
Warning: invalid uid value replaced by that for user 'nobody'
Warning: invalid gid value replaced by that for user 'nobody'
Installing package into ‘/home/ferreen2/R/x86_64-pc-linux-gnu-library/3.6’
(as ‘lib’ is unspecified)
ERROR: this R is version 3.6.2, package 'cmapR' requires R >= 4.0
Error: Failed to install 'cmapR' from GitHub:
(converted from warning) installation of package ‘/tmp/RtmpUSPwSG/file10de472bd25dd/cmapR_0.99.18.tar.gz’ had non-zero exit status

As of today, R 4.0 has not even be released yet.
Can you please let me know how can I install the package on R 3.6.2?
Thank you,
Enrico

Hi Enrico,
Sorry for the confusion about this. We're in the process of submitting
cmapR to Bioconductor, which requires R 4.0, so we needed to temporarily
update the requirements to get it to build on their system. I've changed
the requirements in the source code now to require R >= 3.5. Could you
please try again to install?
Thanks a lot,
Ted
On Fri, Mar 27, 2020 at 5:53 AM Enrico Ferrero<EMAIL_ADDRESS>wrote:

Hello,
I'm trying to install cmapR on R 3.6.2 and I get this error:
devtools::install_github("cmap/cmapR")
Downloading GitHub repo cmap/cmapR@master
✓ checking for file
‘/tmp/RtmpUSPwSG/remotes10de422e46bd/cmap-cmapR-f4d3879/DESCRIPTION’ ...
─ preparing ‘cmapR’:
✓ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ looking to see if a ‘data/datalist’ file should be added
─ building ‘cmapR_0.99.18.tar.gz’
Warning: invalid uid value replaced by that for user 'nobody'
Warning: invalid gid value replaced by that for user 'nobody'
Installing package into ‘/home/ferreen2/R/x86_64-pc-linux-gnu-library/3.6’
(as ‘lib’ is unspecified)
ERROR: this R is version 3.6.2, package 'cmapR' requires R >= 4.0
Error: Failed to install 'cmapR' from GitHub:
(converted from warning) installation of package
‘/tmp/RtmpUSPwSG/file10de472bd25dd/cmapR_0.99.18.tar.gz’ had non-zero exit
status
As of today, R 4.0 has not even be released yet.
Can you please let me know how can I install the package on R 3.6.2?
Thank you,
Enrico
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cmap/cmapR/issues/51, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAO6F3ZJZAPXHHRJALVYJQTRJRZP5ANCNFSM4LU4A3WQ
.

Thank you Ted - it works now!

It seems to me that I have the same issue again.

Hi @hsiaoyi0504,
Sorry you are having trouble with package. Can you please let me know how you are trying to install it? Are you installing from Bioconductor (recommended) or directly from github? Are you using R >= 4.0?
Thanks a lot,
Ted

@tnat1031 Sorry for the late reply. I was using R 3.6 and renv.
It seems to me that I successfully installed the package later using the R3.6 branch, but is this documented at any place?
Probably I didn't find it.

Hi @hsiaoyi0504,
No worries. I don't think that's documented anywhere, but it will work if you need to use R < 4.0. If possible, it might be advisable to use R > 4.0 to that you have access to the latest package versions, updates, etc.
Thanks a lot,
Ted


New Rule: disallowNodeTypes

Fixes #1389
Need some help here.
Esprima node types: https://github.com/jquery/esprima/blob/758196a1c5dd20c3ead6300283a1112428bc7045/esprima.js#L108-L169
Acorn/babel node types:
https://github.com/babel/babel/blob/65a44a1e131c8bc336b218991bae7548176de283/src/babel/types/alias-keys.json#L2-L121
Espree node types: https://github.com/eslint/espree/blob/a23b65db10648daf7d4b0761b0ce09eea864550a/lib/ast-node-types.js#L41-L115
Do we want to check if the node types given in the array are supported by the parser or just let it error?

Coverage increased (+0.0%) to 99.02% when pulling 7e3177d2bc59404422748e1be8dfe6c5b0af0184 on hzoo:i-1389 into 6284a69137885257b50def57bdb2fb95248c1de7 on jscs-dev:master.

/cc @ljharb, @mikesherov

( ﾟ◡ﾟ)/ 👍

This feels like a JSHint rule.

Yeah that's true. However, we do have disallowIdentifiers, and disallowKeywords that are used with the intent to disallow parts as well (with, certain methods like eval or something) so this rule is created with a similar purpose?

Sure then. DisallowKeywords was likely discussed at length and accepted so
this is fine by me.
On May 23, 2015 9:49 AM, "Henry Zhu"<EMAIL_ADDRESS>wrote:

Yeah that's true. However, we do have disallowIdentifiers, and
disallowKeywords that are used with the intent to disallow parts as well (
with, certain methods like eval or something) so this rule is created
with a similar purpose?
—
Reply to this email directly or view it on GitHub
https://github.com/jscs-dev/node-jscs/pull/1398#issuecomment-104900146.

mm, where broad rule, couldn't hurt though

@markelog was from https://github.com/jscs-dev/node-jscs/issues/1385#issuecomment-102881542 - originally to disallow LabeledStatements

Yeah, i read it, there might be use-cases for that, sure, it feels as advanced rule though

Yeah that's definitely true, since you have to know what node types are/esprima/ASTs.


Document behavior on DDL commands

See https://mendix.slack.com/archives/C5SASDZBN/p1630679438064000
Hopefully this will improved, but for now we should document the way it currently works.

I see I only did the Mendix 9 document. The Mendix 8 and Mendix 7 docs should be updated as well, I think?

We normally only do the current version but, since it is you asking, I'll do 7 and 8 as well.


How do you get this to work in webstorm?

Node version:
npm version:
Operating system:
Command line used:
Steps to reproduce:

@greghawk, I haven't tried debugging react-slingshot in WebStorm, but this looks like a good starting point, https://blog.jetbrains.com/webstorm/2015/09/debugging-webpack-applications-in-webstorm

How do you mean get this to work in WebStorm? I exclusively use WebStorm and I know it is Cory's main IDE.
I have not attempted browser debugging it in, but I was able to use the mocha runner to debug tests.
Could you please supply more information on what you are attempting?

I got it to work.  Had another question though.  How can we get css modules to work in this kit? I am diving in to see what i can do but the npm start version seems like it would be tricky

I had a PR that added this functionality #236 which may be able to help. Slingshot does not have a stance on CSS implementation at this point.

@greghawk in terms of debugging in WebStorm, did the link I send help you or did you figure it out elsewhere? I only ask, because it would be good to get a PR in the docs here, https://github.com/coryhouse/react-slingshot/blob/master/docs/FAQ.md#how-do-i-debug, explaining how to debug in WebStorm. Would you be willing to submit a PR for that?

@nickytonline I kind of gave up temporarily and just used debugger; but I would like this to work. If I figured out I can submit a PR

@kwelch That is awesome!! I was stumbling through the webpack config to do this.  I agree it may be out of scope for this project but as a fork into a css enabled one it would be good.  thoughts? @coryhouse

I'm unclear what question you're asking me @greghawk

@coryhouse What I was thinking was that we forked your starter to have one that does include usage of css modules.  It can be named appropriately as the starter/slingshot with css modules as experimental or the like. I know you didn't want to include it in this project as it hasn't exactly landed yet.

Ah, gotcha. Of course, anyone is free to fork at will :)

@coryhouse By the way I love your pluralsight stuff.  I am at a place mired in .net mvc, old angular and the like and am trying to make a case for moving to the react/redux etc.. model.  Your slingshot gets me a lot of the way there for a poc on dev. thanks!

@greghawk Great to hear! :)


Release early, release often

I suggest to make luigi release and release luigi more often in future. For example 1 time per month.

I think we need to merge #1182 first and iterate on the documentation once first. Also  I'm curious about #1243.
Can you review #1182 @toidi?

Ok. I will take a look this evening.

Thanks @toidi !!


Fix run that doesn't end on a word boundary

There was a discussion on StackOverflow about this run and how it is wrong, see it below for more detailed information on what this commit fixes.
The StackOverflow answers show proof that the padding byte is necessary, along with a link to nearly identical documentation that does include the padding byte.
I've also chosen to reword each run must be aligned on a word boundary to each run must be zero-padded to end on a 16-bit word boundary, as per these comments on that StackOverflow page:

Presumably, this means that runs with an odd-numbered length must be zero-padded (although not specified, I assume words are 16 bits long in this context).

Don't they mean that it has to start on a word-aligned address (in uncompressed form)? It's certainly vague, but that would be true in the example.

https://stackoverflow.com/questions/46349718/align-with-word-boundary-in-rle-bitmap-contradiction-in-microsoft-documentation

@MyNameIsTrez : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Build for Dotty

Add support for Dotty compilation ( https://github.com/lampepfl/dotty-example-project ).
Closes #370

Amazing 🎉 Can you also update the travis.yml so that compilation and tests are run against Dotty too?

I expect there will be compilation issues, they can all be fixed with type ascriptions usually.

Closes #370

Scala 3 is out, needs rethink.


UnicodeDecodeError

Running the updater from cron the update errors with the following error.
Traceback (most recent call last):
  File "updateHostsFile.py", line 1276, in <module>
    main()
  File "updateHostsFile.py", line 152, in main
    update_all_sources(source_data_filename, settings["hostfilename"])
  File "updateHostsFile.py", line 552, in update_all_sources
    update_data = json.load(update_file)
  File "/usr/lib/python3.4/json/__init__.py", line 265, in load
    return loads(fp.read(),
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 27: ordinal not in range(128)

The cron line is
/usr/bin/python3 updateHostsFile.py -a -s

relevant for
commit 3d027fd436579318152120369ca1818142047db0

Thanks @kisst.
This line works fine here – Works Mn My Machine Certification – using MacOS and both Python 2.7.10 and ython 3.5.1.
Can you tell me a little more about your rig?
I don't understand the reference to commit 3d027fd?  Is this your first issue submit?

The reference to the commit is just to tell that I'm few commits behind but not something super old stuff, and also as it used to work before I pulled.
About the rig
$ facter virtual lsbdistdescription architecture pythonversion
architecture => i386
lsbdistdescription => Debian GNU/Linux 8.9 (jessie)
virtual => physical
pythonversion => 3.4.2

in the meantime I validated and on an interactive shell ( cli on ssh ) it works fine, but on cron it is crashing with the error above.
on cli
$ env | grep "LANG\|SHELL"
SHELL=/bin/bash
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en

on cron
LANGUAGE=en_GB:en
SHELL=/bin/sh

@StevenBlack do you need more info about the setup, or can I help in any other way to debug this ?

@kisst thanks for the offer.  I have no way to debug this here.
Is anyone else able to reproduce this issue?

As I received a daily reminder from the cron daemon to get this fixed I went after it, the issue was the missing LANG  environment variable, so for now I patched the cron line to be
export LANG=en_GB.UTF-8  ; /usr/bin/python3 updateHostsFile.py -a -s
I hope this help with the debugging, managed to reproduce the error on an active shell,
$ /git/hosts$unset LANG
$ /git/hosts$env | grep LANG
LANGUAGE=en_GB:en
$ /git/hosts$/usr/bin/python3 updateHostsFile.py -a -s
Updating source ./data/StevenBlack from https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
Updating source ./data/SpotifyAds from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/SpotifyAds/hosts
Traceback (most recent call last):
  File "updateHostsFile.py", line 1276, in <module>
    main()
  File "updateHostsFile.py", line 152, in main
    update_all_sources(source_data_filename, settings["hostfilename"])
  File "updateHostsFile.py", line 552, in update_all_sources
    update_data = json.load(update_file)
  File "/usr/lib/python3.4/json/__init__.py", line 265, in load
    return loads(fp.read(),
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 27: ordinal not in range(128)

Not a common use-case I know, but I guess cron can be nice to have.

I also encountered this issue on my PC. Since the default system encoding of my Win10 x64(simplified Chinese) is not utf-8, but GBK.

F:\myd_backup\hosts>updateHostsWindows.bat
Updating source data\adaway.org from https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Updating source data\add.2o7Net from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
Updating source data\add.Dead from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts
Updating source data\add.Risk from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
Updating source data\add.Spam from https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
Updating source data\Badd-Boyz-Hosts from https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts
Updating source data\KADhosts from https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt
Updating source data\malwaredomainlist.com from http://www.malwaredomainlist.com/hostslist/hosts.txt
Updating source data\mvps.org from http://winhelp2002.mvps.org/hosts.txt
Traceback (most recent call last):
File "updateHostsFile.py", line 1276, in
main()
File "updateHostsFile.py", line 152, in main
update_all_sources(source_data_filename, settings["hostfilename"])
File "updateHostsFile.py", line 552, in update_all_sources
update_data = json.load(update_file)
File "e:\ProgramData\Anaconda3\lib\json_init_.py", line 296, in load
return loads(fp.read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0x93 in position 30: illegal multibyte sequence

Can a non-volatile PR be suggested for this that would solve the issue for those affected and not cause issues for those not affected, and then we can take a look at it and go from there? It seems like explicitly specifying encoding would keep the environment more uniform from system to system and doesn't sound like it should cause problems, but I'm not the resident expert on these things.

Same error here with https://hosts-file.net/psh.txt line 168754 containing: <IP_ADDRESS>   www.hualañe.cl
(macOS 10.13.3, Python 2.7.10)

This can now be (also) closed @StevenBlack :+1:


Pause() after Resume() doesn't work

I am playing with this plugin now, playing a flv file locally via http. It's not a real live stream.
The bug I encountered is this:

Play()
Pause()
Resume()
Pause() <-- gives error that says cannot pause if the stream is not playing.

It feels like a bug but is it one that would be moot for use on live stream video?
My purpose ultimately is to use this plugin for stream RSTP video. I'm simply testing with a local file for style and other dev purpose.

I've noticed this issue as well. Seems like a pretty huge issue.
Here's a concrete reproduction of the issue: (you just need to throw this HTML on a server alongside locomote.min.js and Player.swf).

    <title>Locomote</title>

@drowzy I'd love to work on a fix here - any pointers?


DSVD{UnitfulMatrix} units not correct

matrix left divide not translating units in DSVD for UnitfulMatrix
check that matrix left divide is correct, especially with regards to exact flag
within check function should also check for units so that errors do not escape detection

Solved by tracking down issue with matrix left divide.
Bugs in unitdomain/unitrange matching to dims(first)/ dims(last).


Multi-database functionality ?

How can I provide User login and User registration on one application instance, until each user is stored in separate database??? (Databases have the same table structure). I see no possible solution. Thank you

Solved - it is possible with current implementation.


Den dev

the work on last commit is still in progress. Adding buttons to edit delete main topic (thread)

did you manage to format the time?
i'm not really sure where and how to do it, can you check and give some suggestions?

time format example
info.Created = thread.Created.Format("2006-01-02 15:04:05")

finished. Description is in commit comment.
issues remained:

I could not make so edit/delete button is visible only for creator of post/topic. Due to the way our template is parsed inside the html. Cannot have 2 variables "userIDCreatedPost" and "loggedUserID", to compare them with {{if eq ... }} statement.
We discussed - when post is created, by restarting the page we duplicate it. EVERY POST request must be redirected" at the end. This work requires like additional at least one day of work. Can be done during next "dop" for forum.


Button helper not destroying

I have a few buttons on a game. When I destroy the game in buttons disabled state and start again, the mouse pointers are flickering. Looks like two button helpers are created for that button. How to completely remove button helpers when destroying the game?

Please use StackOverflow for support questions. If you believe this is a bug, please provide an isolated test for it in jsfiddle or codepen.
However, to answer your immediate question: Unless you are retaining references elsewhere, ButtonHelper should be freed for garbage collection when its target is removed. The only references a ButtonHelper instance creates to itself is in the target's event listener queues.


JBEHAVE-1281 Enable dependencies caching on Travis to speed up builds

The changes are added to master branch: 5568a9159ce611b6e449dec5c5c7f51cef75c093


repo: remove dev installation instructions.

I think it'd be helpful to have some kind of guidance for someone looking to set up a dev box. They should just set up frontend and backend separately, right?
CC @smoyth

Yes I still wouldn't know how to set up a dev environment based on this readme. I've been told to use the production instructions but it's not clear from this readme that that's what one is supposed to do. I'm also curious now based on what @lognaturel has said if that is in fact the correct course of action.

sorry, yes, i am 10000% slammed right now and not able to write new instructions. i wanted mostly to eliminate the confusing ones based on the conversation we had. for now @smoyth please check out the dev instructions for the central-frontend repo and they should help you get there.

I certainly know that feeling! Thanks.

Makes sense, thanks, @issa-tseng! I should have noted I knew you were slammed and wasn't expecting an answer right away. Let's merge this and follow up later!


Add support for custom docs layout with document

This pr is a successor of https://github.com/facebook/Docusaurus/pull/128, with an additional default config. Also added documents requested in https://github.com/facebook/Docusaurus/issues/737.

@yangshun Fixed. But why the cicleci is on hold?

Closing this in favor of v2 implementation


Error while using Laravel Octane

Reproduction steps:

https://github.com/laravel/framework 9.24.0
https://github.com/laravel/octane 1.3.0 (SWOOLE)
https://github.com/opcodesio/log-viewer 1.0.4

Screenshot

Hey @lazerg , this should now be fixed in the latest v1.0.5 release.
Let me know if there's still an issue!


chore(DatasourceEditor): Create Datasource Legacy Editor Feature Flag

SUMMARY

This creates a new feature flag for the legacy datasource editor that we are going to depreciate in superset 2.0. Currently it is set to False, which will show the legacy datasource editor, but it gives users the option to not have the option available anymore.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

LGTM! CI might require some love though

Thank you! Just fixed CI issues (crossing fingers)

Approved 🚀


libass 0.14.0 -> 0.15.0

Motivation for this change
Build ffmpeg with newer libass
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@ajs124

Please retarget this pull request against the staging branch.

done so here #101983 @mweinelt


Fluent UI is incompatible with Flutter Material

This package is great but it doesn't work in a single codebase with material.
As an example I want to build a custom class called AdaptiveCheckbox. This class decides which Checkbox should be used:
if(Platform.isWindows) return Checkbox(); //Fluent UI

if(Platform.isAndroid) return Checkbox(); //Material

Errors occurre at ThemeData, TextButton, Tab, Divider, Scrollbar, Tooltip, showDialog TextStyle or Colors.
The error message is:
The name 'ThemeData' is defined in the libraries 'package:fluent_ui/src/styles/theme.dart (via package:fluent_ui/fluent_ui.dart)' and 'package:flutter/src/material/theme_data.dart (via package:flutter/material.dart)'. (ambiguous_import at [demo] lib/src/constants/theme.dart:113)
or
'TextButton' isn't a function. (invocation_of_non_function at [demo] lib/src/widgets/buttons/blank_button.dart:136)
Solution
Could you please rename overlapping class names to something like FluentName or FlName.
That would be so helpful.

Did you try using library prefix? https://dart.dev/guides/language/language-tour#using-libraries
import 'package:lib2/lib2.dart' as lib2;

@jolleekin No, I didn't.
It may work but the linter doesn't fully support it (regarding suggestions). Or is there a option to enable this.

Hello! You can use the as keyword when importing the package:
import 'package:fluent_ui/fluent_ui.dart' as fluent_ui;
import 'package:flutter/material.dart';

And use it using the prefix fluent_ui:
if(Platform.isWindows) return fluent_ui.Checkbox(); //Fluent UI

if(Platform.isAndroid) return Checkbox(); //Material


@doc.pruduces not working

@dataset_svc.get('/datasets', strict_slashes=True)
@doc.summary('list all datasets')
@doc.produces([{"name": str, "id": str, "type": str}])
async def list_datasets(request):
    dataset_class = get_dataset_class('CSV')
   try:
        datasets = dataset_class.list_datasets()
        return response.json(datasets, status=200)
    except Exception:
        return response.json({}, status=500)
there is nothing produced, what could be the reason?

it is a duplicated issue of https://github.com/huge-success/sanic-openapi/issues/50
@doc.route() is the workaround

@doc.produces() is worked after merge #58, close this issue.


Identify adequate stability of likelihood values

In e.g. #112 and #97 we have seen precomputed likelihood values change on O(0.01%) depending on the CAMB version. At the moment likelihood value stability is checked using np.isclose which has default settings on accuracy of rtol=1.e05, atol=1.e-8.
We should identify what rtol and atol values are actually appropriate for the SO likelihood codes.

TAC advice is that third decimal place accuracy should be adequate. I would propose that tests using allclose should have tolerances updated to reflect this, with the values being some package-wide parameter which can be changed if desired.


fix: use Astro.site property to generate default canonical urls

People are able to define a site URL in their Astro config, which should be preferred as the base URL for the canonical URL if it exists. This is the same URL that Astro itself uses for canonicalizing  URLs and generating sitemaps. They strongly recommend setting this property in their config file.
Discrepancies can arise between Astro.url and Astro.site so it is important to use Astro.site if it exists. An example of this is preview websites on Vercel where you don't want the site to be indexed to avoid duplicated content penalties. Astro.url will be set to the preview URL and Astro.site will be set to the canonicalized URL.
See:

https://docs.astro.build/en/reference/configuration-reference/#site
https://docs.astro.build/en/reference/api-reference/#astrosite
https://docs.astro.build/en/guides/upgrade-to/v1/#deprecated-astrocanonicalurl

Thank you for your contribution @jaredLunde! This PR went live with https://github.com/jonasmerlin/astro-seo/releases/tag/v0.8.4


connect from android

I'm trying to connect to laravel-echo-server from android. I tried sending subscribe with a channel name but getting _taI9TT5pexmVYHMAAAB left channel: undefined on laravel-echo-server.
Is that possible?

That means you're channel name is empty probably.


error saving file while using tf.data.experimental.TFRecordWriter at https://www.tensorflow.org/beta/tutorials/load_data/tf_records#writing_a_tfrecord_file

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
python notebook using python v3.7.

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): 2.0.0-beta
tfrecord-example-with-tfdata.ipynb.zip

Python version: 3.7

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory: Deep learning VM available on GCP

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
I am following the tutorial at https://www.tensorflow.org/beta/tutorials/load_data/tf_records#writing_a_tfrecord_file
Describe the expected behavior
I should be able to save the file.
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
See attached python notebook
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Was able to reproduce the issue with Tensorflow 2.0.0.beta1 on my system. Thanks!

When you write (let's say, in eager mode, since that's simpler), you need to iterate through the dataset. In the relevant cell of your notebook:
writer = tf.compat.v1.data.experimental.TFRecordWriter(filename)
for features in serialized_features_dataset:
  writer.write(features)

This doesn't look like a bug in TensorFlow to me (of course I may be missing something). Please open a new issue if I am mistaken.


Fix typo

I just found the typo in the comment, so I fixed it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Exception when using StreamBridge to send partition message with nativeEncoding

Describe the issue
When using StreamBridge to send a partitioned message with nativeEncoding, StreamBridge will throw a type cast exception.
To Reproduce
Steps to reproduce the behavior:

useNativeEncoding set to true
Send Message with partition header

Version of the framework
3.0.10.RELEASE
Expected behavior
Message is sent correctly
Screenshots
java.lang.ClassCastException: org.springframework.cloud.stream.function.StreamBridge$$Lambda$1350/1117349678 cannot be cast to org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper
	at org.springframework.cloud.stream.function.StreamBridge.send(StreamBridge.java:153)
	at org.springframework.cloud.stream.function.StreamBridge.send(StreamBridge.java:124)

Additional context
I take a look at the code of StreamBridge and this issue was caused by this line
		Function<Object, Object> functionToInvoke = skipConversion
				? v -> v instanceof Message ? v :  MessageBuilder.withPayload(v).build()
						: this.functionCatalog.lookup(STREAM_BRIDGE_FUNC_NAME, outputContentType.toString());
		if (producerProperties != null && producerProperties.isPartitioned()) {
			functionToInvoke = new PartitionAwareFunctionWrapper((FunctionInvocationWrapper) functionToInvoke, this.applicationContext, producerProperties);
		}

So in the case of skipConversion and partitioned message, a Function object will be cast to FunctionInvocationWrapper

I switched to use Spring Kafka instead of Spring Cloud Stream already, but I still want to know if this bug is really a bug or just my mistake :(

No, you are correct. Fixing it shortly

Hi, I'm experiencing the same issue. Is there a workaround to make this work with version 3.1.3.

We are desperate and now trying to workaround this issue using jitpack.io. However, a different solution would be very appreciated. Is a release in sight?

@bonndan Sorry about the delay. The next Spring Cloud release which will pick up 3.1.4 of stream is set for September 21.
I just updated the Milestone


it's not possible to run the tests for a single challenge in RWD-22

In 2022/responsive-web-design there is no way to test for a single challenge.
As there is a step 1 for each practice project, running
cd curriculum
npm run test -- -g 'Step 1'

as instructed runs a lot of tests
><EMAIL_ADDRESS>test
> ts-node ../node_modules/mocha/bin/mocha --delay --exit --reporter progress --bail "-g" "Step 1"

⠦ Populate tests.

  2814 passing (34s)
  3 pending

I wonder if we can updated it to take the file name instead. 🤔

Here's the area of code where the challenges get filtered. If someone wants to add more options to test a single challenge, I'm fine with it.

it could make sense

file name
project-dashed-name/step-#

Throwing a help wanted label in case someone wants to give it a try - seems like an addition with no downside.

I'm having trouble replicating OP's beginning result. It just reports no test files found. My belief is that the run method needs overridden as well.

See: https://contribute.freecodecamp.org/#/how-to-work-on-coding-challenges?id=testing-challenges
Does this help answer your query?

Yes it does.


Arch/ARM OSX Migrators v2 (includes/supersedes #12 and #13)

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I'm not 100% sure the cross-compiled builds will run on the new Apple silicon. I guess it's better to try and fix it than not to give an option.


Static Code Analyzer : System Information Leak

these are all Console.Writeline calls, should be removed or marked for DEBUG only
#if DEBUG
#endif

Addressed in PR #502 and can be verified with #456 (see: https://github.com/Esri/distance-direction-addin-dotnet/issues/456#issuecomment-366808136).

This has been verified.  There are no longer any references to this build warning on the latest build.


[BUG] Greyed "Revert" option on a snapshot is not disabled

Describe the bug
Greyed "Revert" option on a snapshot is not disabled
To Reproduce

Create a volume, attach it on the node.
Go to the volume details page
Click on Create Snapshot
A snapshot gets created. Click on the snapshot. Revert option is Greyed.
Click on revert.
The below pop up is displayed

Click on Okay.  Revert call is sent, and an error is seen on the UI

Expected behavior
The "Revert" is greyed, the option should NOT be clickable
Environment:

Longhorn version: 1.0.0-rc3

Now the design is changed to be clickable, but the pop-up window becomes as shown in the figure. Clicking ok will not request and will only close the pop-up window. @yasker @sowmyav27

@yasker @smallteeths Do we need Revert option to be enabled? Because the tool tip on Revert shows this message.
Do we need an additional message on pop up ?

The image has been updated to wangsiye/longhorn-ui:6771633 . @sowmyav27 @yasker .
Modify to only disable without pop-up window.

Validated using UI tag: wangsiye/longhorn-ui:6771633
Validation: PASSED

Revert is Greyed out.
No pop up seen or no requests are made when clicked on Revert

Validated on latest master ui version : 23c813d longhorn: v1.0.0-rc3
Validation: PASSED

Revert is Greyed out.
No pop up seen or no requests are made when clicked on Revert