Showing Empty Panel on Android and No Canvas is showing to capture Signature

Hi,
I have installed SignaturePad.Forms nuget on PCL/ Platform specific projects, and SignaturePad nuget on Android/ iOS projects. However, SignaturePad is working fine on iOS, but not on Android as its not showing the Canvas panel it self and showing empty panel. When I was going through the references added in Android project, SignaturePad assembly is pointing to the assembly which is installed in ..\packages\Xamarin.Controls.SignaturePad.2.1.0\lib\Xamarin.iOS and not the MonoDroid assmebly.
This problem happens only when you want to use/install SignaturePad control on both iOS and Android platforms. And its only working on one platform and on others its just showing empty panel.
Please let me know the exact issue to correct it or fix it if its a nuget package issue.
Thanks,
Ramesh Majji

Ramesh,
Is this a Xamarin forms or Xamarin native project with a pcl? Also how are you causing the view to appear ( push or Modal)?

Its a Xamarin.Forms application and defined the SignaturePad control in Xaml page itself.. Following is the code.
<signature:SignaturePadView x:Name="PadView"
HeightRequest="120" WidthRequest="520"
BackgroundColor="White"
CaptionText="Please Sign here" CaptionTextColor="Black"
ClearText="" ClearTextColor="White"
PromptText="✍" PromptTextColor="Black"
SignatureLineColor="Maroon" StrokeColor="Black" StrokeWidth="4" />
Its displaying the SignaturePad control in UWP and iOS fine, but on Android its displaying a white panel which doesn't have any Canvas/ Caption Text/ etc.. attributes.

Sorry how are you getting to that xaml page pushaync or pushmodalasync?

I get to the xaml page using navigation module (custom navigation). Is there any restriction for that

And I created a overlay popup for the Signature pad and added two buttons (Save, Clear) along the actual SignaturePad control. This pop-up will be displayed on top of my current Xaml page when I click the button which is on main Xaml page.

Ramesh,
I am asking if it's push or pushmodal ( or which is it more similar to) as I also have had issues like yours and there is a bug in forms where one method the view does not render things properly. Its happening with the zxing barcode scanner plugin as well. Im also asking so I can compare to the code I have working to see if it's the same method or no ( I'm not at my PC or I'd just tell you which it is)
The reason it's not displaying may also be with how your doing your overlay so I am trying to rule those factors out

Another thing that you can check is to make sure that both the SignaturePad NuGets are installed in the Android app.

MobileRez,
Following is the Settings Page Xaml code which includes the SignaturePad overlay (content view). On Settings page I have other controls and also a button which triggers the flag for SignaturePad overlay.. SettingsPage is called fr4om a Dashboard page using Navigation module. And I have installed both SignaturePad, SignaturePad.Forms nugets on Android project. Ideally, SignaturePad control should display the canvas and PromptText etc.. on Overlay irrespective of the how its being invoked.

<AbsoluteLayout>
    <!-- Normal Page Content -->
    <StackLayout AbsoluteLayout.LayoutBounds="0, 0, 1, 1"
             AbsoluteLayout.LayoutFlags="All">
        <Grid Margin="0" RowSpacing="0">
            <!-- Main Page coding -->
	<!-- Here, I have a button to display the following Signature overlay popup -->
        </Grid>
    </StackLayout>

    <!-- SignatureControlOverlay -->
    <ContentView x:Name="SignatureControlOverlay"
             AbsoluteLayout.LayoutBounds="0, 0, 1, 1"
             AbsoluteLayout.LayoutFlags="All"
             IsVisible="False"
             BackgroundColor="#C0808080"
             Margin="0">
        <ContentView.Triggers>
            <DataTrigger TargetType="ContentView"
                Binding="{Binding ShowSignatureOverlay}" Value ="True">
                <Setter Property="IsVisible" Value="True" />
            </DataTrigger>
        </ContentView.Triggers>
        <StackLayout Style="{StaticResource ContentStackLayoutStyle}"
               Orientation="Vertical"
               BackgroundColor="White"
               WidthRequest="520"
               HorizontalOptions="Center"
               VerticalOptions="Center"
               Margin="15" Padding="0">
            <StackLayout Margin="15" Orientation="Vertical" WidthRequest="520" VerticalOptions="StartAndExpand" HorizontalOptions="CenterAndExpand">
                <Grid>
                    <Grid.RowDefinitions>
                        <RowDefinition Height="150" />
                        <RowDefinition Height="50" />
                    </Grid.RowDefinitions>
                    <Frame Grid.Row="0" OutlineColor="{StaticResource Kenny.MediumGray}" HasShadow="False" HorizontalOptions="FillAndExpand" BackgroundColor="{StaticResource Kenny.White}">
                        <signature:SignaturePadView x:Name="PadView"
                            HeightRequest="120" WidthRequest="520"
                            BackgroundColor="White"
                            CaptionText="Please Sign here" CaptionTextColor="Black"
                            ClearText="" ClearTextColor="White"
                            PromptText="✍" PromptTextColor="Black"
                            SignatureLineColor="Maroon" StrokeColor="Black" StrokeWidth="4" />
                    </Frame>
                    <Grid Grid.Row="1">
                        <Grid.ColumnDefinitions>
                            <ColumnDefinition Width="auto" />
                            <ColumnDefinition Width="*" />
                        </Grid.ColumnDefinitions>

                        <!-- Left Side -->
                        <StackLayout Grid.Column="0" Orientation="Horizontal" HorizontalOptions="StartAndExpand">
                            <Button WidthRequest="120" Text="Save" Clicked="OnSave" />
                        </StackLayout>

                        <!-- Right Side -->
                        <StackLayout Grid.Column="1" Orientation="Horizontal" HorizontalOptions="EndAndExpand">
                            <Button WidthRequest="120" Text="Clear" Clicked="OnClear" />
                        </StackLayout>
                    </Grid>
                </Grid>
            </StackLayout>
        </StackLayout>
    </ContentView>
</AbsoluteLayout>

mattelibow,
Yes, I have included both SignaturePad & SignaturePad.Forms nugets on Android/ iOS projects. The things is its perfectly working fine on iOS, but giving trouble on Android.

Ramesh,
thanks for sending this page, but it still does not answer my question. the navigation modal you are using eventually has to do either pushmodalasync to display the page or pushasync to display the page. it looks like in the code behind your just setting the signature page on top of the existing layout via the absolute layout settings. am I correct in this?

MobileRez,
I got your question now, and am using PushAsync to show the view in NavService. Let me know if it causes a problem.
await XamarinFormsNav.PushAsync(view, true);

@RameshMajji
so the signature page that i am using is a ContentPage not a content view, and I am pushing the page from a list view cell with PushAsync.
can you change it from a content view to a content page and see if that changes anything? though given the way your doing it, can you try displaying it not in a content view?
@mattleibow is there any reason with how the content view or in my case what I had tried earlier in my project psushmodalasync not cause the canvas to render? My first though is its something within the content views that forms is having trouble rendering but I cant nail down why that would be (and create a bugzilla report based on said findings)

@MobileRez
I even tried on a normal page (Content Page) and its giving the same problem on Android. SignaturePad control is not showing the Canvas and other elements (CaptionText, ClearText at all) and ended up seeing a blank white panel.

@RameshMajji I had a look at this, and even using your XAML, I could get the popup to show and the pad visible. There should be no difference between the native renderers and SignaturePad renderers other than the fact that they come from another assembly. Usually when nothing loads at all, it is the case that the renderers were never registered.
You mentioned this:

pointing to the assembly which is installed in ..\packages\Xamarin.Controls.SignaturePad.2.1.0\lib\Xamarin.iOS and not the MonoDroid assembly

I would suggest uninstalling both SignaturePad packages from the Android project - and then make sure that there is no reference to the SignaturePad dlls in the references section. If there are, just remove them. Once the packages are removed, go ahead and re-install them from NuGet.
Here is one of my samples that uses a signature and navigation: https://github.com/mattleibow/SignaturePadWizardDemo.
What I can have a look at is your project files. I don't need the source, but if you can gist the csproj files, I can have a look and see if something pops out.

@mattleibow  I uninstalled the nugets in all projects and installed correctly. Again I am getting the same problem with Android. But, I don't see issues on UWP and iOS and the SignaturePad overlay popup is working fine, but not on Android.
I have attached the .cs proj files for PCL, iOS, and Droid in a zip file.. Please have a look at them and see if you find any issues..
Thanks for your help!
cs proj files.zip

@mattleibow @MobileRez I figured out the issue. This issue is happening if we use 2015 Visual Studio. Currently I am using 2015 Visual Studio and got the renderer issues.. But after I upgrade the Visual Studio to 2017, I am not getting the renderer issues for SignaturePad on Android machines.

Good to hear that you managed to get it working. It is a bit strange since I did most of my dev in VS2015, and only for the last few commits use VS2017.
Maybe you could try again in VS2015 now that things are working. If it doesn't work still, then there is something else the matter as the IDE shouldn't matter this much. In the case of it not working, you could attach the verbose/diagnostic build log (private email if it sensitive) and I can get to the bottom of this.

Closing this issue as it is working. Will re-open if more info indicates a bug.


Fix #449

[Current coverage][cc-pull] is 89.94%

Merging [#464][cc-pull] into [master][cc-base-branch] will not change coverage

@@             master       #464   diff @@
==========================================
  Files            18         18
  Lines          1283       1283
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1154       1154
  Misses          129        129
  Partials          0          0

[][cc-pull]

Powered by Codecov. Last updated by [6b2d87b...67401d0][cc-compare]
[cc-base-branch]: https://codecov.io/gh/bigchaindb/bigchaindb/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/bigchaindb/bigchaindb/compare/6b2d87b53f0ffe798c55d2a2847fb32bb12fd8e8...67401d0f86cde50e2f847983075d7066d0b9c6f0?src=pr
[cc-pull]: https://codecov.io/gh/bigchaindb/bigchaindb/pull/464?src=pr


Adopt CommonMark?

@tmcw writes:

Right now we're using marked as a Markdown processor, and we simply document the syntax as Markdown. marked is modeled after the markdown.pl processor, rather than the CommonMark spec.
I think it'd be good for the future™ to commit to CommonMark syntax, as there are multiple commonmark-compliant parsers (commonmark.js, remarkable, markdown-it) with compatible syntax and output, and pointing to CommonMark as 'the syntax we support' is more rigorous than referring to the rather loose original Markdown description.

I think the cat may already be out of the bag on this one. I expect quite a few notebooks are already using GFM features.

Yeah, at this stage we can't break all those Markdown tables. A fix would be a bug at this point.


Fix unwanted behavior with parent filters

Saw some unwanted behavior when we have some parent with specific filters like:
[{
  category: "my-app",
  sinks: ["a"],
}, {
  category: ["my-app", "with_level_with_filters"],
  level: "info",
  filters: ["someSpecificFilter"],
}, {
  category: ["my-app", "with_level_with_filters", "no_level_no_filters"],
}]

before this PR if we log to ["my-app", "with_level_with_filters", "no_level_no_filters"] then someSpecificFilter will be applied but it seems unobvious since if I just add any level to last config then someSpecificFilter won't be applied, e.g.
{
  category: ["my-app", "with_level_with_filters", "no_level_no_filters"],
  level: "info",
}

Some unwanted commits seem included.  Could you rebase it?


When creating an actuator-, sensor- or signal-mast, show a pulldown menu of already existing JMRI objects.

Is your feature request related to a problem? Please describe.
When creating an actuator-, sensor- or signal-mast, show a pulldown menu of already existing JMRI objects.
Describe the solution you'd like
Describe alternatives you've considered
Additional context

Fixed


Mirror certs+ELBs in GovCloud

SSL certs need to be pulled out of E / W and installed into GovCloud

[ ] Identify certs / locate original certs
Notify users of the cut-off date for ELB/cert change requests in the E/W environment
Install certs in the GovCloud environment
Setup ELBs in the GovCloud environment

Let's go with getting people to use the CDN broker instead.


Unable to update indices settings

Unable to update indices settings when trying to update all indices:

https://www.elastic.co/guide/en/elasticsearch/reference/7.2/indices-update-settings.html#indices-update-settings

You can refer to this for updating .opendistro_security
https://github.com/opendistro-for-elasticsearch/security/issues/107#issuecomment-519775491

This can be achieved by admin certificate with security enabled
curl -k --cert key.pem --key kirk.key.pem -X PUT -H "Content-Type: application/json" -d '{"index.blocks.read_only_allow_delete": null }' https://localhost:9200/.opendistro_security/_settings
I am closing this ticket, please feel free to open in case of any more doubts.


no sd card after install

Hello all,
First of all thanks for your great stuff.
I followed the tutorial with a new mjsxj03hl and UART install and it worked.
Some details in the tutorial is nto up to date but minor things (names of files from archive for example is wrong)
Anyway all is working except that I cannot see anymore the sdcard after boot. (the same that I used for flash during install).
I tried fat32, exfat... nothing.
I see in the main page the disclaimer : "2023.08.16 Attention, in some 03HL revisions there is a problem with SD card access in the bootloader - no power supply. Try to execute the commands described below and perform scheduled actions on SD card."
I suppose this is close to my case, but playing with gpio 54 does nothing and the command "mmcinfo" seems not installed.
Anyone for a tip please ?
Thanks

I made an alternative version of the installation routine. You might want to check it. https://github.com/themactep/device-mjsxj03hl/blob/master/installation.md
There is a weird mmc lock when it cannot recognize a thing. Cold reboot helps with it. See Troubleshooting at the bottom of the page.

Hi @themactep
The cold reboot did nothing today.
Is it possible to reinstall without re-disassemble the entire camera ? (it is almost hard to disassemble to connect UART...)
I have full access to ssh and web interface (but no sdcard...)
What does the GPIO 54 do exactly ?
Thanks

If you have access to ssh then you can update system with sysupgrade either from github or from local files uploaded to /tmp.
GPIO 54 supposedly enables MMC where card reader resides.
I soldered an external connector for UART to keep the camera assembled but still hackable.

I did all the updates to today, and nothing help for sdcard.
I dont know what to do next.
Also I realize that there is no MQTT client with this procedure ? I dont know yet why because I have already installed OpenIPC lite on other cam with MQTT client inside.
Mesage :

MQTT client is not a part of your firmware.
Firmware

Version
<IP_ADDRESS>-lite
Build
master+cd92d00, 2023-08-26
Majestic
master+c79acf8, 2023-08-23
Web UI
dev+788bf0b, 2023-08-19
U-Boot
U-Boot 2013.07-g2762b69-dirty (Apr 19 2023 - 22:04:12)

Any suggestion ?
Thanks

you can check t31_lite_defconfig to see if mqtt is included in the lite build. ultimate does have mqtt. if you connect uart you can install ultimate. just don't forget to run setnor16m afterwards.

Hi @themactep
I have done some tests today.
I completly reinstalled all the stuff according to your doc and scripts.
Some remarks :

on your picture here : https://github.com/themactep/device-mjsxj03hl/raw/master/images/uart.png TX and RX is reversed
in your scripts you use "fdisk -x" which is deprecated (so error)
beware that now on OpenIPC when we download files some file extensions have changed (t31n > t31) : rootfs.squashfs.t31n is now rootfs.squashfs.t31
But after all, same situation as before :
as soon as i reboot with the WIFI config, the system does nkt detect SD Card anymore (even after cold boot)
still no MQTT client  (weird because other lite version have it...)

When you say "check t31_lite_defconfig" what do you mean please ?
Last, i cannot select "ultimate" in the tool https://openipc.org/cameras/vendors/ingenic/socs/t31n it is greyed/unselectable (I hesitate to change the NOR xM option as I dont know at all what it means).
If you have another tips I can help =)
Thanks

UART is a cross connection, RX on the board connects to TX on adapter and vice versa. So my picture is correct, the one in the upstream instruction is not.
I don't know what kind of fdisk you use but fdisk from debian sid does have -x.
My script does not use separate kernel nor rootfs. Are you sure you used correct instruction?
You cannot select ultimate version with 8mb because it requires a 16mb flash, but since these cameras come with 16mb flash, you can change size selector to 16mb and select ultimate.
Defconfig file is the configuration of firmware for a particular build. Configurations for Ingenic you can find here https://github.com/OpenIPC/firmware/tree/master/br-ext-chip-ingenic/configs

OK I understand.
Some new tests, we progress ...
I try ultimate version with 16M NOR and client MQTT is present : this problem fixed.
Now remains the initial problem : as soon as I boot with the wifi configuration, the SDCard support disappears.
Not before obviously, because when I use your script sdcard2.sh, it actually reads the sdcard to initialize wifi.
And after reboot, SDcard unavailable.
If I interrupt boot and do the "gpio clear 54", the sdcard comes back, but disappears as soon as I reset to OpenIPC.
I understand that the "configuration wifi phase" breaks the sdcard support but I dont know how/why.
Is it possible to activate "gpio clear 54" at the boot of OpenIPC to test if it fix the problem ?

update :
Today I resintalled a new firmware with same procedure (sdcard2.sh), and now I can see sdcard after reboot... did change nothing else in procedure. Perhaps the firmware has been update meanwhile ?
Anyway now SD Card it is working.
I have another problem but I will open a new issue for that.
Thanks

Hello, back again,
Today I updated the firmware, and the sdcard is non-working anymore.
Message now changed and is "ERROR: the mmc is Permanent write protection !!!!!!!!!!!!!!" (any sdcard)
And I am stuck.
When I go to bootloader and set "gpio clear 54" it is working after for next commands, but non working as soon as I reset/reboot.
Is it possible to 'load' the command "gpio clear 54" at the boot of camera ? How to do ?
I am not a developper and I really dont know how to call that correctly.
Thanks

Maybe the same issue.
I created SD card using https://github.com/themactep/device-mjsxj03hl/blob/master/sdcards/sdcard2.sh
RUN firstboot ,but nothing in /mnt
[    6.382951] mmc0: error -145 whilst initialising SD card

Remove the card, unplug both power and uart, give the camera to rest for a few minutes, connect power, connect uart, insert the card, check if it is mounted.

I can confirm @duchengyao issues on one of my Mi 2k camera.
So currently I've two cameras one with SD card issue, second is ok.
This with SD card issue has chipset T31 :
070602608-BRN20
this without
190012603-BRN01
So on this with SD card issue only executing gpio clear 54 ; mmcinfo before kernel installation helps.
I noticed that I need to execute it twice. After first, mmcinfo outputs zeros, second run properly initialize SD card.

Hi @themactep
I tried it so many times : "Remove the card, unplug both power and uart, give the camera to rest for a few minutes, connect power, connect uart, insert the card, check if it is mounted." in any order.
Now I am not even able to load wifi config after firmware install.
Very strange.
Is it possible to continue bootload after setting "gpio clear 54" without do a reset ? (which also reset the sdcard ...)

@jedrus2000 , on your second camera when you restart after "gpio clear 54" you are able to see SDCard in system ? In my case, setting "gpio clear 54" is good for bootloader, but as soon as I reboot/reset/enter the firmware, no sdcard again. That's why I would "set gpio at boot time" to fix this issue, but perhaps it is completly wrong.

Now I am not even able to load wifi config after firmware install. Very strange.

@darkfuncat
Workaround is :

place SD card in slot
connect UART, start console, and power camera
during boot CTRL+C
execute this gpio clear 54 and then mmc rescan and fatls mmc 0:1 to see if its work
execute cmd: boot to boot kernel
I was able to see SD card when logging in, and setup WIFI (I did it manually copying files)

Ho great @jedrus2000 !
I was missing the 'boot' command.
Now I can continue my tests.
The idea now is to find a way to activate "gpio clear 54" automatically at boot time ... but I dont know how.

I had the same problem but I managed to mount sd card with these commands:
gpio clear 54

echo "INSERT" > /sys/devices/platform/jzmmc_v1.2.0/present

after this the card should be visible under /mnt dir.

gpio clear 54

echo "INSERT" > /sys/devices/platform/jzmmc_v1.2.0/present

Added this to /etc/rc.local  before exit 1, works well. Thank you for the temporary solution @Slusko88 !


Add features in the singa web page

Rework on the singa web page to add nine features, e.g., Easy installation, Model zoo, Distributed training, Automatic gradient calculation, Memory optimization, Parameter optimization, Interoperability, Time profiling, Half precision.

This has been resolved by https://github.com/apache/singa-doc/commit/74666e04e904bd821b327707896fedb84c509ab0


Anndata not properly garbage collected

Description
When an anndata object is deleted in the current scope the underlying memory is not (reliably) freed  as it is for numpy and others. The memory is kept allocated until the process exits. This lead to huge memory consumption if several anndata files are read sequentially in the same process.
Version:
Python 3.7.6 and Python 3.8.2
anndata==0.7.1
scanpy==1.4.6
Code to Reproduce:
import scanpy as sc
import numpy as np
import os
import tracemalloc

def display_top(snapshot, key_type='lineno'):
    snapshot = snapshot.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    top_stats = snapshot.statistics(key_type)
    total = sum(stat.size for stat in top_stats)
    return total

def trace_function(func, arg, n):
    total = np.zeros(n)
    data = func(arg).copy()
    tracemalloc.start()
    for i in range(n):
        data = func(arg).copy()
        snapshot = tracemalloc.take_snapshot()
        total[i] = display_top(snapshot)
    tracemalloc.stop()
    return total

_NP_FILENAME = 'numpydata.npy'
_ANNDATA_FILENAME = 'testfile.h5ad'
_RUNS = 5

data = sc.read(_ANNDATA_FILENAME)
np.save(_NP_FILENAME, np.random.rand(*data.shape))

total_np = trace_function(np.load, _NP_FILENAME, _RUNS)
total = trace_function(sc.read, _ANNDATA_FILENAME, _RUNS)

display(total_np)

array([1.74600576e+09, 1.74602235e+09, 1.74602685e+09, 1.74602814e+09,
       1.74602672e+09])

display(total)

array([1.74998379e+09, 2.62655985e+09, 3.50312986e+09, 2.62657418e+09,
       3.50315218e+09])

import matplotlib.pyplot as plt

plt.plot(np.arange(_RUNS), total_np, label='numpy')
plt.plot(np.arange(_RUNS), total, label='anndata')
plt.legend()
plt.show()

I think I've noticed this before but haven't been able to figure out what's going on. If it were a simple memory leak where the memory use just kept increasing, that would make sense to me, and would obviously be a leak. But the memory sometimes get collected.
Do you have any idea what could be going on?

Thank you for your quick reply.
Yes you're right, it's not a normal memory leak.
I did a small test today with pympler and it looks like the complete object (and not only parts of it) are kept in memory for some time. However I have no idea why this is the case, but I'll let you know when I have investigated it in more detail.
---- SUMMARY ------------------------------------------------------------------
Before juggling with tracked obj...      active      0     B      average   pct
  anndata._core.anndata.AnnData               0      0     B      0     B    0%
First init                               active      0     B      average   pct
  anndata._core.anndata.AnnData               1      1.65 GB      1.65 GB   73%
0. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               2      3.30 GB      1.65 GB  107%
1. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               3      4.95 GB      1.65 GB  127%
2. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               4      6.61 GB      1.65 GB  139%
3. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               5      8.26 GB      1.65 GB  148%
4. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               6      9.91 GB      1.65 GB  155%
5. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               7     11.56 GB      1.65 GB  160%
6. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               2      3.30 GB      1.65 GB  105%
7. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               3      4.95 GB      1.65 GB  125%
8. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               4      6.61 GB      1.65 GB  138%
9. iteration                             active      0     B      average   pct
  anndata._core.anndata.AnnData               5      8.26 GB      1.65 GB  147%
-------------------------------------------------------------------------------

One point I would like to know: can your machine run out of memory due to objects which could be collected? Is it possible that collection of these objects is triggered based on need?

Yes It can. Not in the single-processed example I showed above, but when I do it multiprocessed my machine gets out-of-memory. See the code below for example. This fills 32 GB of RAM quite fast with a anndata file which takes 1.65 GB in RAM usually.
_RUNS = 3
def trace_function():
    data = anndata.read_h5ad(_ANNDATA_FILENAME)
    for i in range(_RUNS):
        data = anndata.read_h5ad(_ANNDATA_FILENAME)
    return 0
import multiprocessing as mp
with mp.Pool(3) as pool:
    results = pool.starmap_async(trace_function, [() for _ in range(100)])
    print(results.get())

There is a workaround by setting mp.Pool(3,maxtasksperchild=1), meaning when A a (sub)process exits the memory is garbage collected as expected but if the processes are reused the anndata objects still accumulate.

The issue seems to be circular references, which cannot be resolved with the standard reference counter garbage collector. They can only be resolved when with the full garbage collector  (gc.collect()).
This somehow explains the weird behaviour, because in the single-process example the full garbage collector is called, when the allocated memory reaches some limits. However in the multi-processed example, for each process the limit is not reached but in total all RAM is allocated.
The issue seem to be _parent in anndata._core.aligned_mapping.Layers, adata in anndata._core.file_backing.AnnDataFileManager and probably some more, which you can show by:
import gc```
gc.get_referrers(data)
In total I get 6 circular and 1 actual reference for the anndata object.
The best thing, in my opinion, would be store only weak references here (or drop them completely if possible), but weak references seems prevent pickling of the object.

Glad to see you open a PR for this!
Just a general point about this as an issue though. Unless this memory usage is being triggered by non-pathological code, I'm not sure it justifies huge changes in APIs or intended behavior to fix it.
In practice, I think the gc runs frequently enough that these objects get collected. All the example code we have here is pretty weird, just allocating large amounts of memory and doing nothing with it. From what I can see, making a plot or running a pca seems to reliably deallocate the memory.

Thank you again for your quick responses and willingness to help.

Just a general point about this as an issue though. Unless this memory usage is being triggered by non-pathological code, I'm not sure it justifies huge changes in APIs or intended behavior to fix it.

I can understand that you don't want any changes in APIs or similar and I'm happy to take your ideas, such that we can avoid this.

In practice, I think the gc runs frequently enough that these objects get collected. All the example code we have here is pretty weird, just allocating large amounts of memory and doing nothing with it.

Sure these code snippets are very pathological, but we see this behaviour also in non-pathological situations/ pipelines where we process or copy anndata objects a lot. However these codes are obviously to large to show here.

From what I can see, making a plot or running a pca seems to reliably deallocate the memory.

Additionally I cannot agree, that running pca is reliably deallocating the memory.
If you modify the above snippet, this still gradually fills memory. And I think doing multithreaded 100 PCA's is not as pathological.
import os
import anndata
import scanpy as sc
import multiprocessing as mp

def do_pca(filename):
    data = anndata.read_h5ad(filename)
    sc.tl.pca(data)
    data.write('tmpfile.h5ad')
    os.remove('tmpfile.h5ad')
    return 0

with mp.Pool(3) as pool:
    results = pool.starmap_async(do_pca, [(_ANNDATA_FILENAME,) for _ in range(100)])
    print(results.get())

Those examples sound more reasonable. I just gave this case a try with your PR and the current master. I wasn't able to see any consistent improvement in memory usage on your PR branch though. I'm not completely sure what to make of all this though, since I've had generally negative experiences with memory usage and multiprocessing. Here are the results I recorded:

 The script I used
Data was generated with:
from scipy import sparse
import scanpy as sc

(
    sc.AnnData(sparse.random(50000, 10000, format="csr"))
    .write_h5ad("test_sparse.h5ad", compression="lzf")
)

I was also using scanpy master for the efficient sparse pca implementation.
import os
os.environ["OMP_NUM_THREADS"] = "4"

import anndata
import scanpy as sc
import multiprocessing as mp
import gc

_ANNDATA_FILENAME = "./test_sparse.h5ad"

def do_pca(filename):
    data = anndata.read_h5ad(filename)
    data.layers["dense"] = data.X.toarray()  # To increase memory usage
    sc.tl.pca(data)
    return 0

with mp.Pool(2) as pool:
    results = pool.starmap_async(do_pca, [(_ANNDATA_FILENAME,) for _ in range(30)])
    print(results.get())

Of course, YMMV. Have you seen memory usage improvements for your workflows using your PR?

Yes I got improved memory usage. I now tested with the master branch of scanpy and got the same results as you. I guess it's not completely fixed and pca may stores somewhere circular references in the anndata object ? Strangely I didn't saw this behaviour with stable scanpy.
I went back to a toy example and replaced sc.tl.pca with time.sleep(1) and got the following results.

I never checked our pipeline with the PR yet, however maxtasksperchild in multiprocessing.Pool was solving the issue for us in several cases.

I went back to a toy example and replaced sc.tl.pca with time.sleep(1) and got the following results.

I figured this one out, and am now able to see improved memory usage. Basically accessing uns creates circular references, where one of those object holds a reference to the anndata object. It's a very good use case for weakref, since one of the objects should only ever be referred to by the other. I'll make a quick PR for this, since it's a two line change, and you can merge that into your branch.

however maxtasksperchild in multiprocessing.Pool was solving the issue for us in several cases.

Good to hear. As an aside, I've generally had much better experiences with resource handling through dask than multiprocessing. This is especially true if task rely on each other, since dask is good about optimizing for data locality.

Thanks for fixing and this hint about dask!

I just ran in a similar issue with the current anndata version 0.7.5.
Unfortunately I know not enough about pythons garbage collection to fix it myself, but maybe a really trivial example might help you guys fix it:
def do_stuff(adata): # %mprun magic works only if this is defined somewhere else in a file
    copy0 = adata.copy()
    del copy0
    copy1 = adata.copy()
    del copy1
    copy2 = adata.copy()
    del copy2

adata = ad.AnnData(np.ones((10000,10000)))

# copy numpy array
%mprun -f do_stuff do_stuff(adata.X)
# copy anndata
%mprun -f do_stuff do_stuff(adata)

Output for the numpy part:
Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
   957    581.3 MiB    581.3 MiB           1   def do_stuff(adata):
   958    962.7 MiB    381.5 MiB           1       copy0 = adata.copy()
   959    581.3 MiB   -381.5 MiB           1       del copy0
   960    962.6 MiB    381.3 MiB           1       copy1 = adata.copy()
   961    581.3 MiB   -381.3 MiB           1       del copy1
   962    962.6 MiB    381.3 MiB           1       copy2 = adata.copy()
   963    581.3 MiB   -381.3 MiB           1       del copy2

Output for the anndata part:
Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
   957    581.3 MiB    581.3 MiB           1   def do_stuff(adata):
   958    963.1 MiB    381.8 MiB           1       copy0 = adata.copy()
   959    963.1 MiB      0.0 MiB           1       del copy0
   960   1345.1 MiB    382.1 MiB           1       copy1 = adata.copy()
   961   1345.1 MiB      0.0 MiB           1       del copy1
   962   1727.0 MiB    381.8 MiB           1       copy2 = adata.copy()
   963   1727.0 MiB      0.0 MiB           1       del copy2

With some luck, in some runs some partial garbage collection takes place randomly, but over time some net leakage remains.
As it was mentioned that the issues would not occur in real life, some words about my use case. I have a couple GB adata and run a method for O(10) different parameter sets. Every single call of the method should not change the original adata, and therefore uses a local copy to do its thing. All (many) of these local copies accumulate in memory eventually crashing the program.
The gc.collect() call (thanks @fhausmann ) fixes my real life scenario as does replacing .X in the copies with an empty sparse matrix, but having to do this every time one uses a temporary anndata is certainly not ideal...

For others that end up on this tread - I had a similar memory leak in version 0.8.0 and handled it to converting my data to a sparse pandas data frame.

I am having the same issue in 0.9.1 when I loop over a few anndata objects. Even when I explicitly del the object every loop iteration, the problem persists. Per @fhausmann, using gc.collect() solves the problem.

I am having the same issue in 0.9.1 when I loop over a few anndata objects. Even when I explicitly del the object every loop iteration, the problem persists. Per @fhausmann, using gc.collect() solves the problem.

This works for me. I tried both adata = None then gc.collect() and del adata then gc.collect(), and only the latter one works.

Issue seems to persist in anndata  0.10.6.
Running a function that creates AnnData object, runs PCA, Louvain & UMAP (with scanpy interface), returns the AnnData object.
I then delete the object with del and call gc.collect().
At each iteration the amount of used memory increases regardless.
This I would say is typical usage of running a pipeline using multiple parameters in order to evaluate the differences.

Feel free to check cap-anndata solution.
It was specially designed for minimal RAM consumption.

Feel free to check cap-anndata solution. It was specially designed for minimal RAM consumption.

Thanks, will try to integrate it in pipeline to replace AnnData  -  the interface should be compatible I suppose?
Quick question: does it support / integrate with multiomics (ie ATAC + RNA), like Muon?

@jeanmonet

Thanks, will try to integrate it in pipeline to replace AnnData - the interface should be compatible I suppose?
Yep

Quick question: does it support / integrate with multiomics (ie ATAC + RNA), like Muon?
I don't think it's supported right now, but you could create issue for it.


Latest changes to nginx ingress controller is not pushed as a docker image

Trying to deploy the latest rc.yaml file on master branch fails the liveness/readiness probe. Looks like the /healthz port was changed 12 days ago but the docker image is couple months old

@dhawal55 the current code is the correct one.
(we rollback a PR that used references to code in the next version)

@aledbf I'm trying to deploy the current code and readiness/liveness check fails. The ports specified on current rc.yaml does not match the ones the image version gcr.io/google_containers/nginx-ingress-controller:0.8.3 is listening on.

@dhawal55 please post the output of kubectl describe pod <ing pod> and kubectl logs <ing pod>

duh!! Nginx was failing to load the ingresses:
failed to execute nginx -s reload: 2016/11/02 22:20:45 [emerg] 24#24: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64

Increasing server-names-hash-bucket-size fixed it. Thanks for your help. Closing this issue.

@dhawal55 in the next version this change is not needed https://github.com/kubernetes/contrib/pull/1596
Sorry for the trouble

@aledbf In which version is the bucket-size fix not needed?  0.8.3 seems to be the latest tag pushed.  I'm experiencing the bucket-size issue with that tag.  Maybe a new tag needs to be pushed?
https://console.cloud.google.com/kubernetes/images/tags/nginx-ingress-controller?location=GLOBAL&project=google-containers

@josdotso we are in the process to release a new version. Please check https://github.com/kubernetes/ingress/pull/155


Updating multiple database entries at once

Hi,
This is my function to update multiple database values at once
Menuitem.update([{id: 1},{position: 3}], [{id: 2},{position: 3}]).exec(function(err, updatedRecords) {
                if (err) {
                    return res.send({message: 'Could not update the records', err: err}, 500);
                }

                if (updatedRecords) {
                    return res.send({ records: updatedRecords }, 200);
                } else {
                    return res.notFound('Records not found');
                }
            });

Checked the docs at https://github.com/balderdashy/waterline-docs/blob/master/query-methods.md#update-search-criteria--values--callback-
But could not find how to update multiple records at once.
http://stackoverflow.com/questions/26952705/update-multiple-records-at-once-using-sailsjs

According to the doc, the 2nd parameter of update() accepts [{}]. What does it mean?

@sgress454
I reached this issue while trying to find out how to batch update, so I'm also very curious what that [{}] in the docs is referring to. I'm using Mongo as my database and it's throwing an error when I provide an array of objects as the second parameter of .update()

@sgress454 , if your answer is to check the documentation, you should really ensure that it addresses the issue.  As the previous two posts show, the docs seem to indicate that this should work.

@carpiediem That is very fair.  I made the mistake of assuming the docs were correct.  They usually are!  I've updated them in the docs repo, and will see to it that the site is re-deployed soon.
For future reference:
The find criteria (first argument) for .update() can either be a dictionary (same kind as you'd use for .find()) or, as a shortcut, a primary key value.  To update multiple records by primary key, use an "in" query, e.g. User.update({ id: {in: [1, 2, 3]} } ... or the shortcut User.update({ id: [1, 2, 3]}...
The values (second argument) for .update() must be a dictionary of values to set.  Waterline doesn't support using an array as the second argument to set different values for different records.
The Sails 1.0 docs are correct, at least 🙃

Cheers, thanks.


Match Darwin implementation of NSError.localizedDescription

Fixes the difference between what NSError on Darwin prints and what we are currently printing.

@compnerd please test

@swift-ci please test

@compnerd Passed CI, ready to merge.

I would like @millenomi to take a look at this due to the change in the conformance.

Since this is an internal class, I think we have that flexibility.


Add a wait process in localhost to wait for the database to start before starting the Django app

We've usually either used the wait-for or nc to implement a wait. It's possible that the database will take a little longer to be available.
https://github.com/tl-its-umich-edu/canvas-course-manager-next/pull/125

Also should add the migrate file to the start script when we do this.


[Chore]: Configure a linter and run it in CI

Background
We should have a linter configured for all our repositories. We've been using golangci-lint in eirini for a while now and although not perfect is a good option, as it incorporates many different linters. We can steal the configuration from here
Actions to take

add a configuration for golangci-lint in this repository and cf-k8s-controllers
add a target in the Makefile
run the linter in CI

Impact
It'll help us identify and remove problems in our codebase.

Work in progress in the wip-lints branch.


Fail to recognize Code

failed to recognize program code. that is not able to convert the code in HTML to Markdown ```.

Would you be able to provide an example? Thanks

Closing due to inactivity


Update notify prefix

Breaking change

This change fixes the notify scripts to all have the prefix "keymaster". This ensures notifications can be defined easily on a per lock basis and makes it easy to identify the integration to which the script block belongs.
Proposed change

Standardize the naming of the notification names.
Fix for #201
Type of change

[ ] Dependency upgrade
[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (which adds functionality)
[x] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #201
This PR is related to issue: #201

I skipped updating the tests for now as they appear to be a few revisions behind the current code base.

this is fine, but since script.LOCKNAME_manual_notify is created by the user, the user's workflow will break if they regenerate package files without making the requisite change to their script name. We will need to mention this in big bold letters in the release. We will also need to update https://github.com/FutureTense/keymaster/wiki/Notification-Example-Script when this is merged

The breaking-change flag should put it under the breaking changes release note item so that should be fine.
We'll need to include a quip about adding the prefix to their notify script.

Is action needed if you haven't setup any notification scripts?

No but you will get the warning message stating it cannot find the script to run. (safe to ignore it)


Strange link between model and template data, only when isolated: false and iterating over list

Description:
I'm creating an editor component that has a bunch of tab sub-components in it.  Each tab has a file and title.  The tabs are also associated with instances of a Tab class and I am linking them up in such a way that setting tab.title = "test" will automatically update the associated component's data.  To make this work I'm using Object.defineProperty.  The problem is that in certain circumstances (detailed below), setting the template's title/file data property causes the associated property on the tab object to be set, which invokes the defineProperty setter, which updates the template, and so on, until Maximum call stack size exceeded.  I can't see any logical reason for the data/objects to be tied together in this way.  Here is an example:
<!DOCTYPE html>

		<title>Ractive test</title>

There are two conditions that must be met in order to reproduce the issue (otherwise it all works OK):

isolated: false must be set
the tabs must be rendered in a list section (the un-commented template+data above)

Here is some of the stack trace from when it goes wrong:

function

line

set
@
setters.html:29

applyValue
@
ractive@0.9.2:1904

set
@
ractive@0.9.2:2053

set
@
ractive@0.9.2:1608

set
@
ractive@0.9.2:2361

Ractive$set
@
ractive@0.9.2:16060

set
@
setters.html:30

applyValue
@
ractive@0.9.2:1904

set
@
ractive@0.9.2:2053

set
@
ractive@0.9.2:1608

set
@
ractive@0.9.2:2361

Ractive$set
@
ractive@0.9.2:16060

set
@
setters.html:30

applyValue
@
ractive@0.9.2:1904

set
@
ractive@0.9.2:2053

set
@
ractive@0.9.2:1608

set
@
ractive@0.9.2:2361

Ractive$set
@
ractive@0.9.2:16060

set
@
setters.html:30

applyValue
@
ractive@0.9.2:1904

set
@
ractive@0.9.2:2053

set
@
ractive@0.9.2:1608

set
@
ractive@0.9.2:2361

Ractive$set
@
ractive@0.9.2:16060

set
@
setters.html:30

Here's some of the stack trace from the actual app with proper file references etc (HasComponents is my own code, where the ractive.set is being called from a property setter):

function

line

set
@
HasComponents.js:40

applyValue
@
Model.js:126

set
@
Model.js:275

set
@
LinkModel.js:166

set
@
set.js:42

Ractive$set
@
set.js:8

(anonymous function)
@
HasComponents.js:42

set
@
HasComponents.js:41

applyValue
@
Model.js:126

set
@
Model.js:275

set
@
LinkModel.js:166

set
@
set.js:42

Ractive$set
@
set.js:8

(anonymous function)
@
HasComponents.js:42

set
@
HasComponents.js:41

applyValue
@
Model.js:126

set
@
Model.js:275

set
@
LinkModel.js:166

set
@
set.js:42

Ractive$set
@
set.js:8

(anonymous function)
@
HasComponents.js:42

set
@
HasComponents.js:41

applyValue
@
Model.js:126

set
@
Model.js:275

set
@
LinkModel.js:166

I thought maybe Ractive was detecting that the props object was nested within the tab object during the initial this.set(obj.props), and setting up some kind of link between them (although it did seem like a stretch), but I see the same problem if the initial setting is done with for(var p in obj.props) this.set(p, obj[p]); instead.
Versions affected:
0.9.2

To clarify, this is what seems to be happening and what doesn't make sense to me:

a new Tab "t" is created with t.props = {file: "filename", title: "title"}
a tab component "tc" is created with obj = "t"
the tab object's .tpl is set to the component t.tpl = tc
tc.set t.props, initialising the tab's properties at the root of the component's data; the component now has {obj: t, title: "title", file: "filename"}
title changed t.title = "test"
setter invoked to automatically update template
t.tpl.set("title", "test")
tc somehow "knows" that the title property should be associated with the tab object, and sets t.title = "test"

In case there is a better solution, the reason I want references to plain domain objects representing the tabs etc is so I can have things like this.currentTab = tab; in the editor code, for e.g. delivering keyboard events to the currently selected tab.  But I also want to be able to render a tab using a component, AND to have a clean way of updating the template when the data changes.

I haven't quite wrapped my head around your example fully, but rather than creating a special domain object and pushing it around in ractive from outside, why not have a method on your outer instance to create a new tab with a domain object handler?
const Tab = Ractive.extend({ ... });
const Editor = Ractive.extend({
  components: { Tab },
  tab(file, title) {
    const props = { file, title };
    const idx = () => this.get('tabs').indexOf(props);
    this.push('tabs', props);
    const handle = {
      close() { this.splice('tabs', idx, 1); },
      get title() { return this.get(`tabs.${idx()}.title`); }
      set title(t) { return this.set(`tabs.#{idx()}.title`, t); },
      ...
    };
    return handle;
  }
});
const app = new Ractive({
  components: { Editor },
  on: {
    render() { this.editor = this.findComponent('Editor'); }
  },
  ...
});

const tab = app.editor.tab('text.txt', 'Test');

tab.title = 'OK';

You could also try adding a lock around the set to make sure the setter that is calling set doesn't somehow end up calling itself recursively. I'd bet that's what's happening.

Ahh, yes, a lock is what I need.  Would still love to know what's going on though.  The problem with returning a handle with setters is that the object itself will have methods that update the template, so ideally this.title = 123; would update the template.
Thanks!


"Invert if" should not be offered for if-case

With the following Dart 3.0 code:
f(num n) {
  if (n case int()) {
    print("it's an int");
  } else {
    print("it's not an int");
  }
}

Place the cursor on the if token and request quick assists.
Expected result: the quick assist "invert if" should not be offered.
Observed result: the quick assist "invert if" is offered, and its behaviour doesn't preserve semantics.  (It exchanges the if and else blocks, but doesn't change the if statement in any other way.)
Reproduced in 60fec9c72046c7556a93736ed55388a676c4d890.

https://dart-review.googlesource.com/c/sdk/+/297880


[Ext adv] Add support for periodic advertising

This task is split to few subtasks:

LL implementation for advertising   -   https://github.com/apache/mynewt-nimble/pull/354
LL implementation for scanner
Host implementation  -   https://github.com/apache/mynewt-nimble/pull/283


 (Than the previous version)

@joaomoreno changed this, and i actually like the change. Thus closing as designed as we do not plan to fix


Only 1 backup is created per game

Apparently, the max-backups per game settings don't work. No matter in which folder I look, I find only 1 backup and the mapping.yaml.
I activated the zip compression and changed the default path. Settings see screenshot.
In fact I now fiddled around a bit and the backup process seems bugged somehow. Not sure about the details.

Hi! Could you tell me which version of Ludusavi you're using? That particular setting requires Ludusavi 0.14.0 or newer.
Also, it will only create additional backups if the data for the game has changed, so make sure you're updating your save between tests.
When you say the backup seems bugged, what other specific issues are you seeing?

I was using Ludusavi-Playnite v0.11.0 and Ludusavi v0.13.1. Didn't expect Ludusavi getting an update. Maybe you can implement a version check with a warning.
Didn't get around to test extensively, but I'm having 2 backups in one folder now.
Regarding the bugginess: Backups sometimes only got deleted, so I either had 0 or 1 backup per game.
Running ludusavi --version in the terminal returns the version, if that's of any help.

Also sorry for the late response.

There's actually a notification in Playnite for that :) It only triggers once per recommended update, though.

Please let me know if you see that issue anymore with the latest version. For the games with 0 backups, do they have a mapping.yaml file? If so, would you mind uploading one of them here?

Just going through all folders I can find, and, at least for Cities Skylines, which seems to be the only one remaining, there's no files at all, so no mapping.yaml.
However, it's version related. Just didn't see the notification, I guess.
Would a mapping.yaml from a working game help?

Just going through all folders I can find, and, at least for Cities Skylines, which seems to be the only one remaining, there's no files at all, so no mapping.yaml.
However, it's version related. Just didn't see the notification, I guess.
Would a mapping.yaml from a working game help?

If you delete the empty folder for Cities Skylines and back it up again, what happens? If it shows up empty again, then we can get some logs to narrow down the issue. If it ends up with save files, then I think we can consider this fixed :)

Would a mapping.yaml from a working game help?

Nah, that wouldn't narrow it down.

I'm going to close this for now, but let me know if anything changes or if the issue still occurs for new backups of Cities: Skylines.


Outermost Block is Not good

When it comes to blockman I think It should not cover the outermost scope by default at all. The DX would be much better if it only covered inner scopes. Having it cover the outermost scope is simply just disturbing. I wanted to disable it for vue and for astro but I decided to bring this issue up to you instead.
This is what i'm talking about

This comes from a file that is using Astro This is a single file component format. The entire page is covered in this block. It's better to simply have the block associated with  tags, data structures and functions.

Hello, you can disable ground block.
Go to VS Code settings, search:
blockman depth0
Type none in N05 and N20.
The idea of ground block is that it helps us to quickly have a general sense of the entire file. I mean, it shows us what is the width of the widest line in the entire file.

Hello, you can disable ground block.
Go to VS Code settings, search: blockman depth0
Type none in N05 and N20.
The idea of ground block is that it helps us to quickly have a general sense of the entire file. I mean, it shows us what is the width of the widest line in the entire file.

Thanks for your help. I don't think having a background color around the outermost depth should be the default the padding should be enough.  Maybe in the future you could find a way to make it so that the colors change in relation to the theme color.

The problem with blockman is that it's always black. If it understood the theme color and adjusted itself based on that then it would be good.

If it understood the theme color and adjusted itself based on that

There is already a conversation about it. It seems that VS Code has such limitation that it does not seem any way to detect the theme colors of VS Code from an extension.
https://github.com/leodevbro/vscode-blockman/issues/14

If it understood the theme color and adjusted itself based on that

There is already a conversation about it. It seems that VS Code has such limitation that it does not seem any way to detect the theme colors of VS Code from an extension. #14

I didn't know. That sucks there has to be a push for this having to change the color means the developer will just use a few set of themes or have to change the settings for each color-related extension every time they change their themes. Until then please consider making closing the outermost block by default. Trust me most people will not turn it on.


Investigate where raw data files will be stored (file path)

How are raw data files saved? On Fly.io, it makes sense that they are saved on their servers. But what about for self-hosted installs? Where should they be saved? How does that work and how can users at least see them (but not edit)?

In my current (very simple) implementation, the files are stored in the folder called 'persistent_storage'.
This works well with:

fly.io, because we can just mount a fly volume
docker, because we can also just mount a volume
locally, because it is just stored in the persistent_storage folder

How to download?
However, we cannot just serve these assets in the statics folder. We need to have more control.
Therefore, we also persist some metadata for every file upload. Every file is tied to a table and the table defines who have access to its data.


Error 404: The requested URL returned error: Operation timed out.

Hi guys!
https://images.weserv.nl/?url=ssl:vendendo.nl%2Fwp-content%2Fuploads%2F2018%2F10%2F8718475577027_a_en_hd_1-2.jpg&w=75&h=65 is returning the following error: Error 404: Server couldn't parse the ?url= that you were looking for, error it got: The requested URL returned error: Operation timed out.
I've tried https://www.geoscreenshot.com, so it seems your IP isn't being blocked.
Do you see more than we do?
Thanks and enjoy the weekend!
Best,
Daan

Will investigate! (Need to clear some other tasks first, so may take a while).

The IP-adress of vendendo.nl (<IP_ADDRESS>) is unreachable from our network, I'm investigating whether this is a routing issue or otherwise..

Looks like we're blocked, as other IP's in that segment (eg. .57 and .63) are reachable. Packets to .58 are being dropped, packets to other ampro#.fcomet.com servers are also dropped (ampro1, ampro2, ampro3, etc).

Fastcomet (hoster of vendendo.nl) is trigger happy with regards to blocking IP's, quote from their FAQ:

We are using a complex Firewall system on all our shared hosting servers. If you are still not able to access your account, your IP address could be blocked on the server due to many reasons.

There's not much I can do to help you further, I'm sorry..

Thanks for checking Andries!
At least we can inform the user correctly now.

@andrieslouw Fastcomet requested the range of IPs Weserv uses. Could you provide these? Thanks!

All outgoing traffic is fixed to <IP_ADDRESS> (in range <IP_ADDRESS>/16) nowadays.

Seems to work fine nowadays, let us know if there are still any issues!


libxml2: 2.11.5 → 2.12.3

Description of changes
https://discourse.gnome.org/t/libxml2-2-12-3-released/18513
https://discourse.gnome.org/t/libxml2-2-12-2-released/18363
https://discourse.gnome.org/t/libxml2-2-12-1-released/18159
https://discourse.gnome.org/t/libxml2-2-12-0-released/18063
https://gitlab.gnome.org/GNOME/libxml2/-/compare/v2.11.5...v2.12.3

Changes in includes in public headers.
Python 3.12 support.
Lot of cleanups.

Supersedes: https://github.com/NixOS/nixpkgs/pull/269060
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Yeah, looks like with 2.12.3, the patches from #269060 are no longer necessary. (Built gnome-builder and inkscape successfully.)

I'm not sure but I think it broke at least apacheHttpd and inkscape:
apache-httpd> mod_xml2enc.c: In function 'sniff_encoding':
apache-httpd> mod_xml2enc.c:212:53: error: unknown type name 'xmlChar'
apache-httpd>   212 |         ctx->xml2enc = xmlDetectCharEncoding((const xmlChar*)ctx->buf,
apache-httpd>       |                                                     ^~~~~~~

inkscape> /build/inkscape-1.3.2_2023-11-25_091e20ef0f/src/object/uri.cpp: In constructor 'Inkscape::URI::URI(const gchar*, const char*)':
inkscape> /build/inkscape-1.3.2_2023-11-25_091e20ef0f/src/object/uri.cpp:86:9: error: 'xmlFree' was not declared in this scope; did you mean 'xmlFreeURI'?
inkscape>    86 |         xmlFree(full);
inkscape>       |         ^~~~~~~
inkscape>       |         xmlFreeURI

Weird. I built Inkscape on this branch successfully. In the worst case, we can backport the patches from https://github.com/NixOS/nixpkgs/pull/269060

Pushed the patches to staging-next.

apacheHttpd and inkscape build successfully on staging-next now. Thank you!


webview

@amrital123 checkout this webview example. This acts as webview on native and iframe on web. Let us know if there are any additional properties you need.


ELF: make everything inlinable

Motivation:
ELFs have loads of generic types and that basically screams allocations
and swift_getGenericMetdata everywhere because nothing was inlinable.
Modifications:
make everything inlinable
Result:
better performance working with ELFs

ping @Lukasa


analysis downloads use label istead of id

Fixes #609
Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

LGTM, thanks for making the change. Feel free to merge this one.


Committing changes to test infrastructure as a whole to allow Tempest

@richbrowne
Problem:
Implement Tempest plugin for agent code
Analysis:
We have added a Tempest plugin as well as a basic example test. Along
with this, we also moved all tests from top-level into the agent code
distribution. Also, I added a docker file to run the tempest tests.
This is a large change, only due to the fact that we moved all tests into a directory under f5_openstack_agent/tests

Flake 8?  Hmmmm, I am tempted to accept the pull request and resolve the flake8 stuff later.

I'll take care of the errors.

Closing as we have decided to not implement tempest tests in the agent, but in the driver only.


Observations on crouton and Crostini

I bought an HP Chromebook 4GB RAM (double the Dell). The native Chromebook terminal seems to run noticeably slower than the 2GB RAM Dell, but that may due to it being a VM and not a sandboxed part of the filesystem itself. They are both running in CLI mode.

Virtualization doesn't need to be slow.. something something.. --some random guy probably on a google form somewhere
it's limited by hardware acceleration
when crostini gets hardware acceleration it will be fast

Virtualization doesn't need to be slow.. something something.. --some random guy probably on a google form somewhere
it's limited by hardware acceleration
when crostini gets hardware acceleration it will be fast

I honestly doubt it. I'm on a nami based chromebook and I've tested crostini hardware acceleration (with the cros-gpu-alpha packages) and the performance was utter dogshit in comparison to a dual boot or chroot. I was utterly shocked by just how much worse it was. I figured 5% or so was fair, but I straight up only 12fps in dolphin on the vulkan backend (if it wasn't working, it wouldn't have even opened at all)

What's the general consensus are we sticking with crouton over Google's native VM? I wasn't able to open greater than one crostini window, so I have been using crostini and crouton since yesterday which sounds nuts.

Stick to crouton until google fixes it’s vm. If google optimizes enough
Virtualization could be fast enough to not need crouton mostly. Again,
virtualization Doesn’t have to be slow. it just is right now
On Fri, May 31, 2019 at 4:56 AM Adrian Lewis<EMAIL_ADDRESS>wrote:

What's the general consensus are we sticking with crouton over Google's
native VM? I wasn't able to open greater than one crostini window, so I
have been using crostini and crouton since yesterday which sounds nuts.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dnschneid/crouton/issues/4073?email_source=notifications&email_token=AG7C3GK7JJDQO6DWC55SBUTPYDR4DA5CNFSM4HRAIUQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWUUNRA#issuecomment-497632964,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG7C3GMXX2ZUJPPS4WD4WOTPYDR4DANCNFSM4HRAIUQA
.

I'm sticking with both.

I used to think it would be handy to chroot via crouton the filesystem crostini uses, for the all-too-often times and instances when crostini doesn't work well. But that doesn't help, like in the case of the recent problem (https://bugs.chromium.org/p/chromium/issues/detail?id=969127) where the whole VM wouldn't boot because of a typo https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1635832/2/vm_tools/concierge/service.cc#b1616
So I think with sommelier (https://github.com/dnschneid/crouton/wiki/Sommelier-(A-more-native-alternative-to-xiwi)) its back to crouton for me

Managed to completely break the VM and had to powerwash.  Dog slow as has been previously mentioned. Moved back to crouton. Thanks for the all the work @dnschneid. It would be an idea for Google to work with you.

Crostini on my new Pixelbook 2019 is  over 3x SLOWER than crouton on my Chromebook Pixel 2015.
Crostini feels like my Apple //c from decades ago. It's just not ready for real work.
Heading back to crouton. Thank you @dnschneid


Make sure don't miss the provider(module)

Thanks to @levou , he found out a potential bug in collector startup sequence. In the prev codes, some provider may be miss, and trigger an unexpected CycleDependencyException.

Coverage increased (+0.01%) to 22.98% when pulling 1f4e8717cd2cd9b7096571abaa128ebe48a189fb on fix/potential-unable-startup into e6bb08edc51326dd00a3871bb178e9ae2fd3579f on master.

Coverage increased (+0.01%) to 22.98% when pulling 0bde5ed12df228cc10fc5776536d7142066e63bc on fix/potential-unable-startup into 80aed97ffd2e50e3171305cbe01168e9e75799e0 on master.


How to adapt the code for BERT?

Hello, sorry for bothering you.
I am using "bmdal_reg" for a regression problem with BERT(https://arxiv.org/abs/1810.04805)
The error happen in this code:
`    def pop_feature_data(self) -> FeatureData:
# remove the hooks
self._input_hook.remove()
self._grad_output_hook.remove()
# compute the adjusted input \tilde{x} from the paper
# TODO: Clean this!
print("line 175 in layer_features.py: ",self.weight_factor, self._input_data.shape)
print("line 176 in layer_features.py: ",self.bias_factor, torch.ones((self._input_data.shape[0], 1, 1), device=self._input_data.device).shape)
print("line 177 in layer_features.py: ",self._grad_output_data.shape,"\n")
fd=None
inp = torch.cat([self.weight_factor * self._input_data,
self.bias_factor * torch.ones(self._input_data.shape[0], 1, device=self._input_data.device)],
dim=1)
# feature data for the two IdentityFeatureMaps in the ProductFeatureMap, given by inputs and grad_outputs
fd = ListFeatureData([TensorFeatureData(inp), TensorFeatureData(self._grad_output_data)])
    # allow to release memory earlier
    self._input_data = None
    self._grad_output_data = None

    return fd`

I don't why do we need to use:
        inp = torch.cat([self.weight_factor * self._input_data,                          self.bias_factor * torch.ones(self._input_data.shape[0], 1, device=self._input_data.device)],                         dim=1)
Also, in the demo of bmdal, the input data shape is like [batchsize,2]. The output of cat process is like [100,3]
However, why I use BERT, the input data shape is like batchsize, max_padding_length, word_vector. How can I motify the code in bmdal?

I adapt the code to
`    def pop_feature_data(self) -> FeatureData:
# remove the hooks
self._input_hook.remove()
self._grad_output_hook.remove()
# compute the adjusted input \tilde{x} from the paper
# TODO: Clean this!
print("line 175 in layer_features.py: ",self.weight_factor, self._input_data.shape)
print("line 176 in layer_features.py: ",self.bias_factor, torch.ones((self._input_data.shape[0],self._input_data.shape[1], 1), device=self._input_data.device).shape)
print("line 177 in layer_features.py: ",self._grad_output_data.shape)
fd=None
inp = torch.cat([self.weight_factor * self._input_data,
self.bias_factor * torch.ones((self._input_data.shape[0],self._input_data.shape[1], 1), device=self._input_data.device)],
dim=2)
print("line 182 in layer_features.py: ",inp.shape,"\n")
# feature data for the two IdentityFeatureMaps in the ProductFeatureMap, given by inputs and grad_outputs
fd = ListFeatureData([TensorFeatureData(inp), TensorFeatureData(self._grad_output_data)])
    # allow to release memory earlier
    self._input_data = None
    self._grad_output_data = None

    return fd`

It works.
However, I still don't know why

The reasoning behind the torch.cat() part is in Equations (4) - (6) in the paper, the idea is that the torch.ones() part comes from the gradients wrt the biases, while the rest comes from the gradients wrt the weights. I don't know if your code is correct since I don't know exactly how the respective BERT layer works - is it like a vectorized linear layer? With the same weight matrix for the max_padding_length dimension?

Thanks. If you can find a specific equation / implementation of the 3D linear layer I can maybe comment on the situation.


JSON parsing error on websocket_api.program_subscribe()

Hello,
I am reporting this while using solana-py==0.30.2 and solders==0.18.1
The reason I report this in solders and not solana-py is that the issue seem related directly to the JSON parsing, hence, solders.
I've taken the canonical example given on the documentation, and changed the log_subscribe call with program_subscribe.
Before any further investigation, you should check that the ID I provided is indeed supposed to be taken as a parameter for the program_subscribe method, as I am still a newbie with solana, and I may have taken it for a program address while it is not (but I believe it is, according to solscan).
Here is a full working example to reproduce :
import asyncio
from solders.pubkey import Pubkey
from solana.rpc.websocket_api import connect

LIQUIDITY_POOL_PROGRAM_ID_V4 = '675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8'

async def main():
    program_id = Pubkey.from_string(LIQUIDITY_POOL_PROGRAM_ID_V4)

    async with connect("wss://api.mainnet-beta.solana.com") as websocket:

        await websocket.program_subscribe(program_id)
        first_resp = await websocket.recv()
        subscription_id = first_resp[0].result
        print("First rep is : ")
        print(first_resp)
        print()
        response = await websocket.recv()
        print(response)

        await websocket.program_unsubscribe(subscription_id)

asyncio.run(main())

When running this, I get a stacktrace.
In order to provide something that may help, I put a few print in _process_rpc_response, basically printing the raw parameter
The output of the program looks like that, stacktrace included :
Print in websocket_api, by phylante, for debug :
{"jsonrpc":"2.0","result":230891,"id":1}

First rep is :
[SubscriptionResult {
    jsonrpc: TwoPointOh,
    id: 1,
    result: 230891,
}]

Print in websocket_api, by phylante, for debug :
{"jsonrpc":"2.0","method":"programNotification","params":{"result":{"context":{"slot":236633032},"value":{"pubkey":"7wZpAKYM1uygtosoF42V4a5tVLsrzpSN6Uedaxc6vGrQ","account":{"lamports":16258560,"data":"error: data too large for bs58 encoding","owner":"675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8","executable":false,"rentEpoch":0,"space":2208}}},"subscription":230891}}

Traceback (most recent call last):
  File "/home/phylante/code/solana_new_listings/src/listen_for_account.py", line 23, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/phylante/code/solana_new_listings/src/listen_for_account.py", line 18, in main
    response = await websocket.recv()
  File "/home/phylante/code/solana_new_listings/venv/lib/python3.10/site-packages/solana/rpc/websocket_api.py", line 108, in recv
    return self._process_rpc_response(cast(str, data))
  File "/home/phylante/code/solana_new_listings/venv/lib/python3.10/site-packages/solana/rpc/websocket_api.py", line 350, in _process_rpc_response
    parsed = parse_websocket_message(raw)
solders.SerdeJSONError: a list or single element

Running this code with the program ID "11111111111111111111111111111111" gives me some notifications, so I assume this is a bug with this particular message.
Thanks in advance for looking into this.

Looks like the account encoding is defaulting to base58, which is bad. Hence why the response says "data":"error: data too large for bs58 encoding".
I'll fix the default in solana-py, but in the meantime you should be able to avoid this issue with await websocket.program_subscribe(program_id, encoding="base64")

It's been a while, but just tested this, and it does the trick. Thank you very much


error while installing udacimak in windows

It's working on my end.
Are you using VPN or proxy? Can you give it a try on another machine?

I've got the same problem I can't solve


Improvements to pathType, pathIsDirectory and pathIsRegularFile

Description of changes
This is related to the path library effort.
Improves the lib.sources.pathType, lib.sources.pathIsDirectory and lib.sources.pathIsRegularFile functions, each of which is a single commit:

Moves them to lib.filesystem, since they only work with the filesystem, they don't import any sources. The functions are still available in lib.sources, but will be deprecated after the next release.
Add test cases
Minor refactor
Fix them for when the filesystem root / is passed. Previously they threw an unintuitive error:error: attribute '' missing

     at nixpkgs/lib/filesystem.nix:24:20:

         23|   */
         24|   pathType = path: (readDir (dirOf path)).${baseNameOf path};
           |                    ^
         25|

Improve the pathType error message for invalid paths. Previously this was the error:error: attribute 'nonexistent' missing

     at nixpkgs/lib/filesystem.nix:29:10:

         28|     if dirOf path == path then "directory"
         29|     else (readDir (dirOf path)).${baseNameOf path};
           |          ^
         30|

Improve documentation, adding a better description, a type and examples.
Indicate that pathType can get replaced with the new builtins.readFileType once Nix 2.14 is released

This work is sponsored by Antithesis :sparkles:
Things done

Wrote docs, built and checked the manual (nix-build doc)
Wrote and ran the tests

The new script lib/tests/filesystem.sh on its own
And the integration into nix-build lib/tests/release.nix

Something I don't want to go into for this PR, but maybe in the future: The semantics of pathIsDirectory and pathIsRegularFile are not obvious:

They return false when the path doesn't exist, which is often not what you want.
They always return false for symlinks without checking where the symlink points (it could point to a directory or a regular file)

We should consider having alternate functions with cleaner semantics in the future:

isDirectory :: Path -> Bool: Returns true iff the path is a directory, following symlinks. An error is thrown if the path doesn't exist.
isFile :: Path -> Bool: Returns true iff the path is a file, following symlinks. An error is thrown if the path doesn't exist.

These laws would then hold:

If isDirectory path succeeds and is true, then builtins.readDir path succeeds
If isFile path succeeds and is true, then builtins.readFile path succeeds
isDirectory path != isFile path

Unfortunately the implementation of these functions would have to be a bit hacky (throwing for non-existent paths not implemented here):
{
  isDirectory = path: builtins.pathExists "${toString path}/";
  isFile = path: ! builtins.pathExists "${toString path}/"
}

This uses the fact that:

toString can turn paths to strings, which would not be the case with lazy trees, and would violate the minimal assumption of the path library
builtins.pathExists works not only on paths, but also on strings, with the important difference that strings aren't normalised, which would remove trailing /'s

To implement these I'd propose adding more builtins, either:

builtins.readSymlink: return the target of a symlink
builtins.isDirectory: A builtins version of isDirectory

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/tweag-nix-dev-update-47/27387/1

We should consider having alternate functions with cleaner semantics in the future:

isDirectory :: Path -> Bool: Returns true iff the path is a directory, following symlinks. An error is thrown if the path doesn't exist.
isFile :: Path -> Bool: Returns true iff the path is a file, following symlinks. An error is thrown if the path doesn't exist.

That's good. How about this?

isDirectory :: Path -> Bool: Returns true iff the path is a directory, following symlinks. An error is thrown if the path doesn't exist.
existsDirectory :: Path -> Bool: Returns pathExists f && isDirectory f
same for *File

existsDirectory is just a rename of pathIsDirectory, but less ambiguous.

@roberth I'd probably go for directoryExists instead of existsDirectory, this then also mimics pathExists


PR: Add support for eslint-plugin-flowtype-errors

This plugin allows flow to be run as an eslint plugin 🎉 Would a PR for this be welcome?

Is this hindering a development without flow? If no then go for it, please :)

Closing, as the PR for this was just made


Fix git commit blog article link.

The trailing forward slash produces a 404 error instead of redirecting to the correct URL.

Hello @mortonfox,
thank for very much for the correction of the link in the README :+1:
Best regards,
Digitalcraftsman


Can you provide convert OPT-xx weights into Alpa formats?

I find convert OPT-175B weights into Alpa formats on the official website, but it can't convert OPT-(125m~66b) ，Can you provide convert OPT-(125m~66b) weights into Alpa formats?

@zhisbug I want to convert opt 125-66b pytorch format to alpa format , Instead of the converted opt format。Can you provide the corresponding conversion code？

Best wishes!

If you run these commands with opt 125M-66B, Alpa will automatically download and convert the weights


Python2 compatibility

Dear Cory, this is Leopold from the group of Berend Smit.
We would like to use pyIAST as part of an AiiDA workflow and the version of AiiDA we've been using is only python 2.7.
This PR makes pyIAST compatible with both python2 and python3. I hope this is ok?
It also adds a travis check that, so far, just installs pyIAST on python2 and python3 and imports the module (see here for the build log and here for how to enable travis for your repository - only steps 1-3 apply).
I saw you already have a few more tests in the test/ directory that are more of the "visual" type. You might consider at some point to add some continuous integration tests whose result can be checked automatically.
I've also added an auto-formatter (yapf).
In order to apply the autoformatter (+ automatic check that the code remains python2/3 compatible), do pip install -e .[pre-commit]; pre-commit install. This will run the "hooks" on every commit.
If you don't like the autoformatter, let me know and I can remove it again.

@ltalirz hi, nice to e-meet you. Great, thank you for contributing to pyIAST!
Could you possibly re-make this commit but without all of the formatting changes, then make the formatting changes later in a separate PR? The reason is that it is difficult for me to go through the many changes and see what has been changed. Most of this is formatting changes.
I agree that it would be great to have unit tests to run on Travis instead of 'visual' ones I originally had. I welcome a PR for this (converting my visual tests into unit tests)!. I will enable the Travis CI.
Thanks.

@ltalirz so now this PR is stale, can you make another PR with the formatting changes?
Thanks, I appreciate it.
Another issue: in the tests, I get the following warning from pandas. Is it okay if we change these to .idxmin or will this interfere with Python 2 compatibility? I know there is some way to ensure that the users have version x or higher pandas in the setup.py?
Then, I will release a new version so that pip sees the Python 2 compatible code. I suppose we should update the documentation now too.
/home/cokes/pyIAST/pyiast/isotherms.py:60: FutureWarning: 'argmin' is deprecated, use 'idxmin' instead. The behavior of 'argmin'
will be corrected to return the positional minimum in the future.
Use 'series.values.argmin' to get the position of the minimum now.
  idx_min = df_nonzero[loading_key].argmin()

can you make another PR with the formatting changes?

Done in #7

Another issue: in the tests, I get the following warning from pandas. Is it okay if we change these to .idxmin or will this interfere with Python 2 compatibility? I know there is some way to ensure that the users have version x or higher pandas in the setup.py?

Done in #8

I suppose we should update the documentation now too.

Done in #8

Then, I will release a new version so that pip sees the Python 2 compatible code

Yes, one the PRs are merged, please do!

@ltalirz thank you! I released a new version and updated PyPI. Please try a pip install on your machines to make sure it works as intended!
You're right that taking in PR's would be much easier if I had written unit tests instead of visual ones...


missing base file

Hi,
The file in the jupyter notebook:
!wget https://ndownloader.figshare.com/files/20150468 -O ./example_data/processed_R2478.h5
is not available

The file is online for me, you can check if the command without specifying the output:
!wget https://ndownloader.figshare.com/files/20150468
The cell above this one creates the folder /example_data/ which is needed for the wget command to work

yes working now I must have done something wrong before with the creation of the folder. Thanks for the quick reply


fix: missing i8 type

Motivation
pilota build failed
struct Number {
    1: optional i8 a = 0;
}

 thread 'main' panicked at 'unexpected literal Int(0) with ty I8', .../rsproxy.cn-8f6827c7555bfaf8/pilota-build-0.7.7/src/middle/context.rs:678:18

Solution
code gen i8 type

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Seems that there's no i8 type in the thrift types doc: https://thrift.apache.org/docs/types


Use zig allocator

Zig has a nice allocator system. The platform could use it instead of c-alloc.
Here is an example, how this is possible: https://github.com/ostcar/roc-aoc-platform/blob/43064f864b7a9c1c8db452c85b108bbbe917d3c3/host/host.zig#L142-L200
Currently, libc is also needed for the inline-expect. But there is the plan to solve inline-expect the same way then dbg. If this is implemented, then a zig-roc-binary would not need libc and could be statically linked.

I extracted the allocator code in its own module: https://github.com/ostcar/roc-aoc-platform/blob/main/host/RocAllocator.zig
It can be used by wrapping any zig-allocator:
// in main()
roc_allocator = RocAllocator{ .allocator = fba.allocator() };

// global
var roc_allocator = undefined;

export fn roc_alloc(size: usize, alignment: u32) [*]u8 {
    return roc_allocator.alloc(size, alignment) catch std.debug.panic("panic: OOM", .{});
}

export fn roc_realloc(ptr: [*]u8, new_size: usize, old_size: usize, alignment: u32) [*]u8 {
    return roc_allocator.realloc(ptr, new_size, old_size, alignment) catch std.debug.panic("panic: OOM", .{});
}

export fn roc_dealloc(ptr: [*]u8, alignment: u32) void {
    roc_allocator.dealloc(ptr, alignment);
}

The global stuff is necessary, until the allocator is provided to roc with each call.


Convert subclassing logic to instance logic for things like particle systems

refactor these particle systems to be instances of MacroParticleSystem etc. rather than instanceless subclasses. If each particle system is going to be a subclass but each subclass will have only one instance there's no point in them being subclasses imo since they don't implement any logic. This is currently an antipattern in fact and it's really confusing to a new reader that we're generating subclasses based on the contents of a json file somewhere & those subclasses contain no custom logic. We've essentially reimplemented OOP using subclass methods/fields instead of instancing which is weird
this PR could also include some code to move the UniquelyNamed functionality into a proper factory function instead (e.g. make creation of duplicates impossible rather than asserting this way)

This is issue is being addressed in this branch: rl-multiple-envs-systems


Credit rudimentary audit trail

Perhaps accomplish this by assigning IDs as part of configuration and send new environment variable?

In reality, this might be solved by the implementation of our blockchain this summer. We should keep logging to logging and let the chain do what it does.


Basic coverage testing with coverage.py+codecov, #61

Closes #61

To fix this we need @magicicada to sign the project to codecov, or alternatively we can scrap it out and merely release coverage artifacts.

I can do this, but was wondering - are we approaching the point where we should make this repo belong to an organisation so that multiple people have the sorts of permissions needed for this kind of thing?  Just thinking of the several months where I will be mostly unavailable coming up.

Agreed, we could move this repo into an organisation at some point.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@40307f1). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #62   +/-   ##
=========================================
  Coverage          ?   71.24%
=========================================
  Files             ?        7
  Lines             ?      539
  Branches          ?       63
=========================================
  Hits              ?      384
  Misses            ?      148
  Partials          ?        7

Flag
Coverage Δ

unittests
71.24% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40307f1...19c1797. Read the comment docs.


Extendable-presets support

Description
Properly configuring Detox is a nontrivial task with many layers. Even with automated inception scripts such as detox init, there are still many things to go over so as to tailor to a project/environment's needs. That leaves many potential points of failure, and makes the setup complicated.
As a solution, we should  introduce the ability to use pre-baked presets, which would hold most of the necessary configurations required for various environment flavours. These presets should be extendable in the sense that they could be used as configuration bases, allowing developers to simplify their configuration work a great deal -- by focusing strictly on the changes required in tailoring things for them. Potentially, presets can be part of standalone github repos.
Ideally, for a relatively simple project, the final outcome - which is feasible, should be such as the following (Assuming some imaginary preset node module called default-aws-preset):
"detox": {
  "preset": "default-aws-preset"
}

Extendability
There's probably more than one way to achieve extendability, but in a essence, the point is to be able to grab both the top-level Detox config from a preset, and sub-configs (e.g. of the test runner) and override their properties. A very efficient way to do that is - in the spirit of what was originally detailed here, is by using javascript overrides in a .js config file. For example, in order to add a custom file into the  Jest setupFilesAfterEnv list for a specific project, this can be applied:
// .detoxrc.js
const detoxConfig = require('some-detox-preset');
module.exports = {
  ...detoxConfig,
  runnerConfig: './jest.config.js',
};

And then:
// ./jest.config.js
const jestConfig = require('some-detox-preset/jest-config');
module.exports = {
  ...jestConfig,
  setupFilesAfterEnv: [
    ...jestConfig.setupFilesAfterEnv,
    './custom-jest-setup.js',
  ],
};

For simplicity, we should also consider allowing for complementary merging configurations in json-format: Namely, those that go alongside the preset property in the detox section.

Side effects
This also paves way to at least 2 benefits:

Allowing for a base detox configuration which would be the default for all users (even presented in the example apps) -- thus simplifying the setup process.
Allowing for an easy configuration alignment across projects that are related to each other (e.g. apps of the same organization).

Future Improvements
Merging multiple presets?

Can we close this issue now?


Long running API for dump and restore refs

Close https://github.com/treeverse/lakeFS/issues/6527

[ ] add test for dump/restore refs

@johnnyaug thanks! I've fixed all the comments and found more issues :(
I'll address the additional issues found and re-request.

Closing this one - reimplemented this PR and kept the old one as internal before removing compatability.