TypeError: tar.Extract is not a function

Hi,
I installed ArcticViewer and can start the usage menu.
When I try to download sample data I get the following:
_ArcticViewer --download-sample-data
| Available datasets for download (path: /Users/magdalenaschwarzl/Documents/uni/coursesUtahGIT/research/paraviewWeb)
|   (1)  40.0 MB  -  diskout-composite
|   (2)  94.2 MB  -  ensemble
|   (3)  13.7 MB  -  head_ct_3_features
|   (4)  13.1 MB  -  head_ct_4_features
|   (5)  50.8 MB  -  hydra-image-fluid-velocity
|   (6) 162.3 MB  -  mpas-composite-earth
|   (7)  37.5 MB  -  mpas-flat-earth-prober
|   (8) 552.5 MB  -  mpas-hd-500-7t
|
| Press Enter to quit or the dataset number to download: 1
| Press Enter to quit or the dataset number to download: 5
| Press Enter to quit or the dataset number to download: 8
| Press Enter to quit or the dataset number to download:
|
| => You will be able to try ArcticViewer with these commands:
|
/usr/local/lib/node_modules/arctic-viewer/node_modules/tarball-extract/tarball.js:11
.pipe(tar.Extract({ path: destination}))
^
TypeError: tar.Extract is not a function
at extractTarball (/usr/local/lib/node_modules/arctic-viewer/node_modules/tarball-extract/tarball.js:11:15)
at EventEmitter. (/usr/local/lib/node_modules/arctic-viewer/node_modules/tarball-extract/tarball.js:30:5)
at emitOne (events.js:115:13)
at EventEmitter.emit (events.js:210:7)
at IncomingMessage. (/usr/local/lib/node_modules/arctic-viewer/node_modules/wget/lib/wget.js:51:28)
at emitNone (events.js:110:20)
at IncomingMessage.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1047:12)
at _combinedTickCallback (internal/process/next_tick.js:102:11)
at process.tickCallback (internal/process/next_tick.js:161:9)

It seems that 5 and 8 are corrupted files. We'll investigate once back to the office.

Hi!
I believe its not about corrupted files.
I spent a day trying to fix this error and I could not solve it because I am begginer when it comes to node/npm/linux systems but I have come to a conclusion.
ArcticViewer has its own node_modules in /usr/local/lib/node-modules/arctic-viewer/node-modules
The thing is the 'tarball-extract' module uses 'tar' module
The tar module version in the ArctictViewer is 3.1.5 and the function "tar.Extract" has been modified to "tar.x" in the 3.1.5 version
The solution:
Rollback tar module version in the node-modules of ArcticViewer to 2.2.1(last one with "tar.Extract").
I don't know how to do that.
In my machine '/usr/local/lib/node_modules/arctic-viewer/node_modules/tar' has version 3.1.5
but '/home/user/node_modules/tar' has version 2.2.1
Copying the content fo version 2.2.1 to the '/usr/local/lib/node_modules/arctic-viewer/node_modules/tar' does not work.
Someone help me!
Thanks in advance!

It seems if we just update our dependency on tarball-extract to 0.0.5, we could be all set, as they have rolled back their own dependency on the tar package for the very reasons you discovered on your own:
https://github.com/matsieftw/tarball-extract/commit/970ab467e1726eaef21ba3c2f5fb42a36497cf91
I have not run into this problem yet, or verified the issue, but fairly soon we may have a chance to look into it.

Note that if you're in a hurry to get it working, you could try updating that dependency locally, removing your node_modules folder, and running npm install.  If you do that, let us know if it solves the problem for you.

I don't know how to do that.
I mean, I can delete my home/user/node_modules folder and do a npm install, but I don't believe it would change much, since my local  tar version is 2.2.1 and tarball-extract is 0.0.6
What I don't know what to do and believe it would help is how to change the tar(3.1.5) and tarball-extract(0.0.3) in the .../ArcticViewer/node_modules folder.
I am not used to Linux, and I am not used to node yet so things like the difference between '/usr/local/opt' -> '/opt' and '/home/node-modules' -> '/usr/local/lib/node_modules/arctic-viewer/node_modules/' when installing/using a software are still a obstacle for me.

If you got your hands on ArcticViewer by doing npm install -g arctic-viewer (perfectly reasonable, as that's what the README says to do), then it will be harder for you to test changing the dependency on tarball-extract.  If you cloned the arctic-viewer repository on github, then it would be easy to try out what I suggested.  We may have a chance to address this issue ourselves soon, but not immediately.

I did 'npm install -g arctic-viewer'.
I can wait.
Thank you Scott.

I had same problem here, using local clone of ArcticViewer.
Using tarball-extract  0.0.5, as suggested by @scottwittenburg works fine for me.


HDU to Projection

Add creating a Projection from a FITS HDU.

Coverage increased (+0.07%) to 73.978% when pulling a8cec1270fd896a2a37ed39ff938f29541675976 on e-koch:2d_from_hdu into dfa99edc6f3b6ee6e12efbbc35c9579d9d82f3c2 on radio-astro-tools:master.

I do this all the time.  Thanks!

Add a changelog entry?  Then we're good

👍


Add RWKV benchmark

Benchmark based on this:
https://github.com/BlinkDL/RWKV-LM
This is an interesting LLM architecture that is based on RNNs rather than transformers. This is an opportunity to have an RNN in the benchmarks.
Based on #88

Does it fit on 1 GPU? Based on the repo's doc RWKV-4 has 7.4B parameters. OPT-6.7B in comparison does not fit on 80GB RAM, when including gradients, activations, optimizer state and data.

@bouthilx It's configurable. I used one of the examples they gave that is configured to be smaller than that.

It fails on ROCm while trying to compile a torch extension, unfortunately, so I think this will be CUDA-only for the foreseeable future.

C'est quoi HIP?

https://github.com/ROCm-Developer-Tools/HIP

C'est le cuda de ROCm

Je suis pas certain lequel serait le plus important à ajouter entre celui-ci et Whisper. Whisper permet de couvrir un domaine différent, mais c'est encore un transformer, alors que RWKV permet d'ajouter un RNN sans transformer, mais ça reste du language modeling. Je dirais que le plus facile à ajouter devrait être priorisé.

(Incluant support AMD)

Looking for ptxas ? I think that is redistributed with CUDA.
but if you install pytorch triton ptxas might get redistributed through it as well https://github.com/pytorch/builder/issues/1318

Is there change we need to bring to the docker file ?

Yes, the change is already in the PR: https://github.com/mila-iqia/milabench/pull/90/files#diff-9d577c160bdb1afcf223f0f34f8194206efabfd0bda9a1629c52673b3997c5ebR44


Can't build because "generator/vulkan/build_integration.zig" tries to call b.graph.cache.obtain() which doesn't exist

I'm trying to set up a little game, but my build.zig fails:
$ zig build
/home/tq/.cache/zig/p/1220e6c4e31599c25cc7df2b300ee5ece3d89ddf0c566dbb5e4eb8d0d49e32ad5bb6/generator/vulkan/build_integration.zig:80:21: error: no field named 'graph' in struct 'Build'
        var man = b.graph.cache.obtain();
                    ^~~~~
/home/tq/.local/apps/zigup/zig/0.12.0-dev.2063+804cee3b9/files/lib/std/Build.zig:1:1: note: struct declared here
const std = @import("std.zig");
^~~~~
referenced by:
    create: /home/tq/.cache/zig/p/1220e6c4e31599c25cc7df2b300ee5ece3d89ddf0c566dbb5e4eb8d0d49e32ad5bb6/generator/vulkan/build_integration.zig:29:27
    build: /home/tq/.cache/zig/p/1220e6c4e31599c25cc7df2b300ee5ece3d89ddf0c566dbb5e4eb8d0d49e32ad5bb6/build.zig:50:31
    remaining reference traces hidden; use '-freference-trace' to see all reference traces

My build.zig:
const std = @import("std");

pub const Frontend = enum {
    client,
    server,
};

pub const LibType = enum {
    static,
    shared,
};

pub fn build(b: *std.Build) void {
    // Options
    const opt_target = b.standardTargetOptions(.{});
    const opt_optimize = b.standardOptimizeOption(.{});
    const opt_frontend = b.option(Frontend, "frontend", "Whether to produce a client or server executbale") orelse Frontend.client;
    const opt_serverLibType = b.option(LibType, "server_lib_type", "Whether to build the server as a static or a shared library") orelse LibType.static;

    // Server library
    const thServer_args = .{
        .name = "thorium_server",
        .root_source_file = .{ .path = "src/server/root.zig" },
        .target = opt_target,
        .optimize = opt_optimize,
    };
    const thServer = switch (opt_serverLibType) {
        .static => b.addStaticLibrary(thServer_args),
        .shared => b.addSharedLibrary(thServer_args),
    };

    // Client/server executable
    const thClient = switch (opt_frontend) {
        .client => b.addExecutable(.{
            .name = "thorium_client",
            .root_source_file = .{ .path = "src/client/main.zig" },
            .target = opt_target,
            .optimize = opt_optimize,
        }),
        .server => b.addExecutable(.{
            .name = "thorium_server_cli",
            .root_source_file = .{ .path = "src/server/main.zig" },
            .target = opt_target,
            .optimize = opt_optimize,
        }),
    };
    b.installArtifact(thClient);

    // Dependencies for server (todo)
    // Dependencies for client
    thClient.linkLibrary(thServer);
    if (opt_frontend == Frontend.client) {
        // GLFW dependency
        const glfw_dep = b.dependency("mach_glfw", .{
            .target = opt_target,
            .optimize = opt_optimize,
        });
        thClient.root_module.addImport("mach-glfw", glfw_dep.module("mach-glfw"));
        // Vulkan dependency
        const vulkan_dep = b.dependency("vulkan_zig", .{
            .registry = @as([] const u8, "deps/Vulkan-Headers/registry/vk.xml"),
        });
        thClient.root_module.addImport("vulkan", vulkan_dep.module("vulkan-zig"));
    }
}

Maybe it's because I'm using zig version 0.12.0-dev.2063+804cee3b9 (2024.1.0-mach to be exact) and the API doesn't exist there yet, but I can't upgrade to master because I'm using mach-glfw.

Yes, this was changed here: https://github.com/Snektron/vulkan-zig/pull/118. I suggest to just use vulkan-zig from a commit before here, for now. Alternatively, we can check the compiler version and pick the new or the old code.

This should be fixed with https://machengine.org/about/nominated-zig/#202430-mach

Alright, it works with 2024.3.0-mach. Thanks for the help :)