Files referenced in course content not appearing when Files page is absent

Files from my RC-coded mod (UTS2706 - A critique of markets in Singapore, AY24/25 S1) are not reflected in the folder saved on the personal directory through Lominus. This is because the files in Canvas for the particular course are listed at the Modules page, not at the Files page. Exporting the file contents returns a zip, of which the files can be retrieved through navigating within the extracted parent directory (viewer/files).
As such, I would like to recommend an inclusion of a function to extract files through the Export Course Content functionality if there are no files stored within a course's Files page.
Currently using Mac OS Sonoma, Macbook Pro M2 Pro (2023)
Screengrab from Canvas:

@randomwish sounds great ! this would be particularly useful for modules where lecturers do not upload certain files in the Files page, but only on the Modules page.
these documentations might be helpful: Canvas Modules API, specifically List Module Items and Show Module Item
Also a side note, just realised that NUS canvas is now using GraphQL queries instead of the traditional REST.
thanks !


Use getPhysicalFilename() instead of getFilename() if available (ESLint 7.28+)

Starting with 7.28.0, ESLint has a new feature distinguishing the physical filename from the virtual filename.

context.getPhysicalFilename()
Rules can now use the new method getPhysicalFilename() on the context object to get the full path of the file on disk without any code block information.
The difference between getPhysicalFilename and getFilename is observable when ESLint is used with processors:
context.getPhysicalFilename() // "/project/example.md"        - original file
context.getFilename()         // "/project/example.md/0_0.js" - virtual filename assigned to a code block

When there's a difference, this plugin needs the real, physical filename. (See
https://github.com/eslint/eslint/issues/11989)
I ran into the problem when I noticed no-unresolved was yielding false positives inside of code blocks in Storybook / Markdown files and helped push the necessary change through in ESLint.
To maintain backwards compatibility, this change uses context.getPhysicalFilename() if available and otherwise falls back to getFilename().

(Had to update the tests so the getPhysicalFilename stuff only applies in eslint v7.28+)


Get Ready to Extend to the Cloud

Group: https://developers.sap.com/group.hana-cloud-extend-1-get-ready.html
Tutorials 1&3 not working

They seem to be working now.


prelude.new_array type params differ between declaration and signature computation

The type_args are declared as [length, elem_type]
https://github.com/CQCL/hugr/blob/6bf6c82c9ec9d801ab43e311e5f815a3eea7d9c1/hugr-core/src/extension/prelude/array.rs#L249-L250
But the binary signature computation only uses the element type
https://github.com/CQCL/hugr/blob/6bf6c82c9ec9d801ab43e311e5f815a3eea7d9c1/hugr-core/src/extension/prelude/array.rs#L61-L67
Which one is correct?

Not an issue: the computation is two stage - first the length argument is used to compute a function type polymorphic over the element type, which is later initialised by the second type argument

It'd be nice to declare matching args/params on both places to avoid confusion.
Guppy used the ones in the signature definition, and that caused validation to fail.
It looks like using [length, ty] passes validation, so this is not an urgent issue.

agreed, it is confusing to say the least
Will defer to fallout from #1433 - we may yet get to some simplifications to the polymorphism that help


cant download the GEO files?

i cant download the GEO files with the HTTP CODE 401,it may means the shell has wrong account ID?

Elasticsearch version:7.10.1
Moloch version:2.7.1
OS name and version:centos7
How was Moloch built/installed: (rpm, deb, easybutton, ...)rpm
Provide logs, stack traces and steps to reproduce:
[root@server1 bin]# ./Configure
Found interfaces: eth0;lo
Semicolon ';' seperated list of interfaces to monitor [eth1] eth0
Install Elasticsearch server locally for demo, must have at least 3G of memory, NOT recommended for production use (yes or no) [no] no
Elasticsearch server URL [http://localhost:9200] http://<IP_ADDRESS>:9200
Password to encrypt S2S and other things [no-default] 111111
Moloch - Creating configuration files
Installing systemd start files, use systemctl
Download GEO files? (yes or no) [yes] yes
Moloch - Downloading GEO files
2021-01-10 15:12:48 URL:https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv [23322/23322] -> "/tmp/tmp.HFQkiov0dh" [1]
Received an unexpected HTTP status code of 401 from https://updates.maxmind.com/app/update_secure?db_md5=00000000000000000000000000000000&challenge_md5=58d974187385075fd96cbb244a742619&user_id=0&edition_id=GeoLite2-Country:
Invalid account ID

We use github for bugs and feature requests, please use slack for questions. The FAQ is always a great place to start too.

We use github for bugs and feature requests, please use slack for questions. The FAQ is always a great place to start too.


The just formatter always updates the timestamp

Describe the bug
The just formatter always updates the timestamp of the file regardless of whether the file is changed or not.
DEBU formatter | just: executing: /nix/store/0irlcqx2n3qm6b1pc9rsd2i8qpvcccaj-bash-5.2p37/bin/bash -euc for f in "$@"; do
  /nix/store/qj4vq5wqbjirx4x8l5z3w9wjgl8da4w6-just-1.36.0/bin/just --fmt --unstable --justfile "$f"
done
 -- .justfile
INFO formatter | just: 1 file(s) processed in 21.883899ms
DEBU file has changed path=.justfile prev_size=1266 prev_mod_time="2024-11-22 08:28:16 -0600 CST" current_size=1266 current_mod_time="2024-11-22 08:37:27 -0600 CST"
traversed 87 files
emitted 1 files for processing
formatted 1 files (1 changed) in 68ms

To Reproduce
Steps to reproduce the behavior:

Format once: treefmt-nix  -c --formatters=just --verbose=2
Format again: treefmt-nix  -c --formatters=just --verbose=2

Expected behavior
The file should not be reported as changed when it hasn't been.
System information
just is version 1.36.
Additional context

/cc @jbott
This should be brought up upstream ideally

This should be brought up upstream ideally

I've opened casey/just#2478

Fixed upstream in https://github.com/casey/just/pull/2479


Updated validations.py python script.

Closes: #1
Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Closing the pull request as a course exercise


Update Call Rest API Client - TFS

This commit has goal update version calle HttpClient.
For this page:
https://learn.microsoft.com/en-us/azure/devops/integrate/how-to/call-rest-api?toc=%2Fazure%2Fdevops%2Fmarketplace-extensibility%2Ftoc.json&view=azure-devops-2020&source=docs

@microsoft-github-policy-service agree [company="{DSY}"]

@microsoft-github-policy-service agree company="Developer Solution For You"

@dreyimm - Thanks for your pull request. A member of the content team will review it.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

@DreyIMM Thank you for this update. Our stale bot automation accidentally closed this PR before we had a chance to review it.
From the notes in the PR description, I believe that you want to make an update to the following article: https://learn.microsoft.com/en-us/azure/devops/integrate/how-to/call-rest-api?view=azure-devops
The source for that article in GitHub is here:
https://github.com/MicrosoftDocs/azure-devops-docs/blob/main/docs/integrate/how-to/call-rest-api.md
Can you go there and make your updates in a new PR that updates that article? Your current PR is adding a new file in the wrong location for articles, so I am closing it, but if you create an update to the article using the link above, we will review and and get it added. Thank you.


Warnings / Errors sporadically not updating

Often (but sporadically) certain warnings and errors aren't updated to reflect that they are no longer present.  For example, F841 - local variable '<VARNAME> is assigned to but never used`, will remain even after that variable is definitely being used.
Work-arounds:

Closing the open window/buffer and reopening it fixes the problem.
Cutting and pasting the line causing the error seems to force a re-check, and fixes the problem.

It seems like there is some underlying bug causing flake8 not to be updated, or for previous warnings not to be erased.

This would be an issue with linter or far less likely flake8, not linter-flake8. (I've seen this occasionally to from a variety of providers.
Marking as closed but please file an issue on linter about this if there isn't one already.


How can I change the asctime format?

The default date_format is HH:MM:SS.  I need what is the default logger date_format, i.e. %Y-%m-%d %H:%M:%S with milliseconds as .%(msecs)03d.

Figured it out.  Just set datefmt=None.  This will cause it to default.  In my case, since I'm using a yaml file, I can just not set the datefmt value to pass it None.
handlers:
  colored:
    class: rainbow_logging_handler.RainbowLoggingHandler
    datefmt:
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout


After enabling Android Camera permission I have to restart the app

ImagePicker.launchCamera is not invoking camera after enabling runtime camera permission. I
have to restart the app.

targetSDK: 26
Android 8.0

React Native version: 0.54
react-native-image-picker: 0.26.10

This is not working:
 static launchCamera(options, callback) {
        ImagePicker
            .checkAndRequest([PermissionsAndroid.PERMISSIONS.CAMERA], options)
            .then(() => {
                RNImagePicker.launchCamera(options, callback);
            })
            .catch(error => ImagePicker.invokeError(callback, error))
            .done();
    };

I solved the problem by replacing ImagePickerModule.java file with new version.


pentobi: 18.4 -> 18.5

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/pentobi/versions.
meta.description for pentobi is: "A computer opponent for the board game Blokus"
meta.homepage for pentobi is: "https://pentobi.sourceforge.io"
meta.changelog for pentobi is: ""
Updates performed

Version update

To inspect upstream changes

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 18.5 with grep in /nix/store/035y7y0aby9n4kiq2hw160lgvjb56bg0-pentobi-18.5

directory tree listing: https://gist.github.com/be361075b696484e94b962092d43bed2

du listing: https://gist.github.com/294806fd2a7578370fd13f1881f05049

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
0 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
pentobi

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/035y7y0aby9n4kiq2hw160lgvjb56bg0-pentobi-18.5 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A pentobi https://github.com/r-ryantm/nixpkgs/archive/4745c284177794807487dfa072e9ed698332d5bc.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/035y7y0aby9n4kiq2hw160lgvjb56bg0-pentobi-18.5
ls -la /nix/store/035y7y0aby9n4kiq2hw160lgvjb56bg0-pentobi-18.5/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

pentobi

Maintainer pings

Result of nixpkgs-review pr 116034 at 4745c284 run on aarch64-linux 1

1 package built successfully:

pentobi

5 suggestions:

warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?
Near pkgs/games/pentobi/default.nix:35:5:
   |
35 |     maintainers = [ ];
   |     ^

warning: patch-phase
patchPhase should not be overridden, use postPatch instead.
Near pkgs/games/pentobi/default.nix:19:3:
   |
19 |   patchPhase = ''
   |   ^

warning: unclear-gpl
gpl3 is a deprecated license, check if project uses gpl3Plus or gpl3Only and change meta.license accordingly.
Near pkgs/games/pentobi/default.nix:34:5:
   |
34 |     license = licenses.gpl3;
   |     ^

warning: unused-argument
Unused argument: gettext.
Near pkgs/games/pentobi/default.nix:1:57:
  |
1 | { lib, mkDerivation, appstream, fetchFromGitHub, cmake, gettext, libxslt, librsvg, itstool
  |                                                         ^

warning: unused-argument
Unused argument: libxslt.
Near pkgs/games/pentobi/default.nix:1:66:
  |
1 | { lib, mkDerivation, appstream, fetchFromGitHub, cmake, gettext, libxslt, librsvg, itstool
  |                                                                  ^


nydus-image: modify compact subcommand

Brief introduction
Modify command "nydus-image compact" to compress the small blob in the nydus image into a large blob
Basic Usage
nydus-image compact  --bootstrap /path/to/nydus_bootstrap --compact-config /path/to/compact.json --backend-config /path/to/backend-config.json

@newthifans Thanks for your contribution! Can you add some unit tests to cover your changes?


DHGR displayed incorrectly when monochrome mode is turned on

Fixed in https://github.com/V2RetroComputing/analog-firmware/commit/7c3fa52a3f9ccd4cae41440556b44ea920cd1d09


Error detected while compiling function

This error happens when opening a file for the first time. Tried with and without .vimrc and in both cases the error happened.
Error detected while compiling function <SNR>25_NetrwBrowseChgDir[194]..<SNR>25_NetrwEditFile[7]..BufRead Autocommands for "*"..function lsp#lsp#AddFile[1]..lsp#buffer#BufHasLspServer[1]..lsp#buffer#BufLspServerGet:
line   16:
E1095: Unreachable code after :throw
Press ENTER or type command to continue

vim --version
VIM - Vi IMproved 9.0 (2022 Jun 28, compiled May 17 2023 21:50:06)
Included patches: 1-1563

Facing the same issue.
Try commenting return after :throw
diff --git a/autoload/lsp/buffer.vim b/autoload/lsp/buffer.vim
index c717e0d..bb67b5f 100644
--- a/autoload/lsp/buffer.vim
+++ b/autoload/lsp/buffer.vim
@@ -66,7 +66,7 @@ export def BufLspServerGet(bnr: number, feature: string = null_string): dict<any
   if !SupportedCheckFns->has_key(feature)
     # If this happns it is a programming error, and should be fixed in the source code
     :throw $'Error: ''{feature}'' is not a valid feature'
-    return {}
+    # return {}
   endif

   var SupportedCheckFn = SupportedCheckFns[feature]

the new recent vim9 patch restricted that, could you open a pr for that?
// or i believe yegappan would update it some days later.

The PR #314 is merged to address this issue.


Extension methods named Run can cause computation expressions to fail to compile

Defining an extension method with the [<Extension>] attribute named Run that has one generic parameter or a tupled parameter where the first is generic with a constraint will cause computation expressions to fail compilation with various errors as though the computation expression builder's definition of Run ceased to exist.
There is no problem if the Run extension's parameter is not generic or if it is generic with no constraints.
It is unknown if other Builder method names are affected, although Combine and Delay seem to be unaffected.
Repro steps
Consider the following code:

type MyClass() =
    member val Prop = 0

[<Extension>]
type CompletelyUnrelatedExtensions =
    // these are fine
    [<Extension>] static member Run (x: int, y) = x + y
    [<Extension>] static member Run (x: MyClass, y) = x
    [<Extension>] static member Run (x: int) = x

    // produces error "None of the types 'AsyncBuilder, Async<'a>' support the operator '+'"
    //[<Extension>] static member Run (x, y) = x + y

    // produces error "The type 'AsyncBuilder' is not compatible with the type 'MyClass'."
    //[<Extension>] static member Run (x: #MyClass, y) = x

    // produces error "The member or object constructor 'Run' takes 0 argument(s) but is here given 1."
    //[<Extension>] static member Run (x) = x

let f () = async {
    do! Async.Sleep 100
    ()
}
f () |> Async.RunSynchronously

If any of the commented extensions are uncommented, the compiler will throw errors at the async { ... } computation expression. Note that the presence of the first three extensions does not affect the outcome in any way.
For convenience, this code sample has been shared on Repl.it where it can be run: https://repl.it/repls/LoyalRawVoxel.
Expected behavior
Extension methods named Run that are not type-compatible with a computation expression should not affect the compilation of the computation expression.
Actual behavior
Unrelated extension methods named Run cause code that worked flawlessly before to suddenly stop compiling.
Known workarounds

Don't include the interfering extension methods in the scope of the affected computation expression
Rename the extension method
Change the type of the first parameter on the extension method to be non-generic
Change the extension method to use curried parameters

Related information
None.

Extension methods named Run that are not type-compatible with a computation expression should not affect the compilation of the computation expression.

My understanding is that this is expected behavior - the presence of a "Run" method matters to the syntactic desugaring of computation expressions

From 6.3.10 of the F# 4.1 spec:
If the inferred type of b has one or more of the Run, Delay, or Quote methods when builder-expr is
checked, the translation involves those methods. For example, when all three methods exist, the
same expression translates to:
68
let b = builder-expr in b.Run (<@ b.Delay(fun () -> {| cexpr |}C) >@)

Does that make sense?

@dsyme The spec makes sense, the behavior I am observing does not. I worded the desired behavior poorly. Let me put it this way:
Extension methods named Run that are not type-compatible with a computation expression should not prevent the compiler from selecting the type-compatible Run method(s) on the computation builder itself.
Just the fact that importing a namespace with an unrelated extension method named Run can cause all computation expressions to no longer compile makes no sense at all.

Well, extension methods can interfere with type inference, especially ones as generic as above.  So although it is not intuitive it is per spec.


Combined Counter Electrode Potenial

When importing EC data from biologic, some techniques have Ewe/V for potential and some have /V, ixdat can combine these into "raw_potential". However, there is no equivalent for counter electrode potential, this would be very useful to have.

Thanks for this issue, @BeneVolere , which I'm finally solving 8 very distracted months later. See the code changes (and feel free to review!) here: https://github.com/ixdat/ixdat/pull/135


Tasks in the queue are not being processed

Hi, I'm not sure if I'm misunderstanding something about how django-q works, but any tasks which I push on to the queue using async() seem to just stay in the queue and never get processed, and I'm not sure why.
Here's an example of how I'm pushing tasks onto the queue:
async('django.core.mail.send_mail',
              'Test Subject',
              'Test Message',
<EMAIL_ADDRESS>              [user.email])

Below you can see that the tasks are getting registered to the queue, but they just stay in the "Queued tasks" and never get processed.

My configuration for django-q is:
Q_CLUSTER = {
    'name': 'DjangoORM',
    'timeout': 30,  # Timeout in secs for a task
    'save_limit': 10,  # Store latest 10 results only
    'orm': 'default'  # Django database connection
}

Disregard; as I suspected, I was misunderstanding how to use django-q. I wasn't running the python manage.py qcluster command so I never had a cluster running to actually process tasks in the queue.
It might be beneficial to update the docs so that the "Installation" or "Configuration" sections include some brief instructions or just a mention that the user needs to manually start the cluster so that queued tasks get processed. As someone who had no prior experience with this sort of async queueing functionality before, I didn't actually know this was necessary. It was only when I went through every section of the docs that I eventually realised I needed to manually start the cluster when reading the "Cluster" section.

Hi @jordanmkoncz in the tasks section there is already a clear note about this.
Note
For tasks to be processed you will need to have a worker cluster running in the background using
python manage.py qcluster or you need to configure Django Q to run in synchronous mode for
testing using the sync option.

the link: http://django-q.readthedocs.io/en/latest/tasks.html?highlight=qcluster#q-options
Cluster explain this in more details.

@Eagllus Ah yeah, I see that now. I did actually look through the Tasks section of the docs before getting stuck and submitting this issue, but I completely missed that note.
I feel like it would still be better to include some information about this in the "Installation" or "Configuration" section, or perhaps just move this note to one of those sections. Given that the requirement of having a worker cluster running is not specific to single async tasks but rather it applies to using django-q in general (both for async and schedule tasks), it seems like that would help to ensure users are aware of this requirement.

@jordanmkoncz thanks for the feedback will create a pr about this subject.
After looking in the 'Installation' documentation I believe this idd should be included.

@jordanmkoncz a short setup about starting a qcluster. #204

@Eagllus Looks good to me! I think new users (such as myself a few days ago) should see that part of the instructions when setting up django-q and be aware from the start that they need to manually start a cluster for django-q to actually run the tasks that are scheduled and in the queue.

I think this issue can be closed, #204 has been merged

Please reopen this issue if the problem has not been resolved

I have the same problem the task in the queue are not processed, but the task in scheduling are processed.
Q_CLUSTER = {
    "label": "Tasks",
    "workers": 1,
    "orm": "default",
    "ack_failures": True,
    "timeout": HOUR_IN_SECONDS * 4,
    "retry": HOUR_IN_SECONDS * 24 * 7
}

The cache is memcache 1.5.16


install failure

one error I resolved changing this in generate_set_map.py line 19
data_loc = r"C:\Program Files\Wizards of the Coast\MTGA\MTGA_Data\Downloads\Data"
when attempting to run commands listed in readme, I get this error
(venv) J:\python-mtga>python setup.py sdist bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'bdist_wheel'

command errors resolved, my VEN didn't have wheel or twine installed


[ZF3] [RFC] Removal of Zend\Loader

Composer as primary autoloader
Just a suggestion, but I think that composer pretty much took over the autoloading-related problems in all of PHP.
This is a good time to dismiss Zend\Loader.
EDIT: the git submodule update --init approach could also finally die :-)
Compatibility
Almost every module that is on github currently provides a composer.json file. The BC break would be only for most in-house modules that still rely on Module#getAutoloaderConfig().
We can get rid of getAutoloaderConfig() or provide a small compat layer that just merges the fetched configuration into the already existing Composer\Autoload\ClassLoader.
Advantages
One less component to maintain, nuff said.
Problems
The only issue with composer-first autoloading is security, but I think that can be dealt with until this issue becomes a PR.
Additionally, composer can be used without packagist and as a pure autoloader (composer.phar dump-autoload).

In case its relevant, I only ended up learning composer because I wanted to try out ZF2 and I'm forever greatful for it. First thing I ended up doing was abandoning the use of the zf2 autoloader. Since I'm using packages that require composer now I'd rather use 1 autoloader than 2.
One thing that struck my attention was the mention of enterprise systems that move a lot slower. I happen to be working for one of said enterprises where over the past few months I've been introducing the idea of composer and have started to use packages from it. Things do move very slowly indeed although the problems have been reluctance more than anything technical.
In short, those enterprise companies move so slow that by the time they end up wanting to use ZF3, will be when you guys are thinking about ZF4, so maybe isnt as much of a concern as you might think.
Just my two cents, hope it helps.

In case its relevant, I only ended up learning composer because I wanted to try out ZF2 and I'm forever greatful for it. First thing I ended up doing was abandoning the use of the zf2 autoloader. Since I'm using packages that require composer now I'd rather use 1 autoloader than 2.
One thing that struck my attention was the mention of enterprise systems that move a lot slower. I happen to be working for one of said enterprises where over the past few months I've been introducing the idea of composer and have started to use packages from it. Things do move very slowly indeed although the problems have been reluctance more than anything technical.
In short, those enterprise companies move so slow that by the time they end up wanting to use ZF3, will be when you guys are thinking about ZF4, so maybe isnt as much of a concern as you might think.
Just my two cents, hope it helps.

why reinvent the wheel, composer is good

This issue has been closed as part of the bug migration program as outlined here - http://framework.zend.com/blog/2016-04-11-issue-closures.html


Add request proto to add copy job

What changes are proposed in this pull request?
Initial commit to add proto to support copy job. Please ignore CopyJob.java since it's incomplete.
Why are the changes needed?
So we can work on user-facing command and backbone in parallel
Does this PR introduce any user facing changes?
na

@Xenorith mind merging this for me? Thanks

alluxio-bot, merge this please

alluxio-bot, cherry-pick this to dora please


Debug Info: Fix the emission of vector types.

Thanks to an improved IR verifier upstream we found that support for
vector types has been incomplete.
rdar://problem/38324861

@swift-ci test and merge


Importing TF checkpoint as BertForTokenClassificiation

Hello Everyone,
I've been stuck with trying to load TensorFlow checkpoints to be used by pytorch-pretrained-bert as BertForTokenClassification.
pytorch-pretrained-BERT Version: Installed from latest master branch.
What works:
config = BertConfig.from_json_file(CONFIG_FILE)
model = BertForPreTraining(config)
model = load_tf_weights_in_bert(model, "/home/bert/pt_baseuncased/model.ckpt-98000")

What I want to do:
config = BertConfig.from_json_file(CONFIG_FILE)
model = BertForTokenClassification(config, num_labels=num_labels)
# the difference is BertForTokenClassification instead of BertForPreTraining
model = load_tf_weights_in_bert(model, "/home/bert/pt_baseuncased/model.ckpt-98000")

When I try to do this it gives me:
AttributeError: 'BertForTokenClassification' object has no attribute 'bias'

Full Traceback:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-a8225a5966f7> in <module>
      1 config = BertConfig.from_json_file(CONFIG_FILE)
      2 model = BertForTokenClassification(config, num_labels=10)
----> 3 model = load_tf_weights_in_bert(model, "/home/gzhenfun/bert/pt_baseuncased/model.ckpt-98000")

~/pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     88                 pointer = getattr(pointer, 'weight')
     89             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 90                 pointer = getattr(pointer, 'bias')
     91             elif l[0] == 'output_weights':
     92                 pointer = getattr(pointer, 'weight')

~/anaconda3/envs/bert/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    533                 return modules[name]
    534         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 535             type(self).__name__, name))
    536
    537     def __setattr__(self, name, value):

AttributeError: 'BertForTokenClassification' object has no attribute 'bias'

To try to resolve this:
I followed https://github.com/huggingface/pytorch-pretrained-BERT/issues/580#issuecomment-489519231 from #580, and changed my modeling.py to this:
        pointer = model
        for m_name in name:
            if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
                l = re.split(r'_(\d+)', m_name)
            else:
                l = [m_name]
            if l[0] == 'kernel' or l[0] == 'gamma':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'cls')
                # added the line above
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'cls')
                # added the line above
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'squad':
                pointer = getattr(pointer, 'classifier')
            else:
                try:
                    pointer = getattr(pointer, l[0])
                except AttributeError:
                    print("Skipping {}".format("/".join(name)))
                    continue
            if len(l) >= 2:
                num = int(l[1])
                pointer = pointer[num]

However, that gives me:
AttributeError: 'FusedLayerNorm' object has no attribute 'cls'

Anybody here knows how I can fix this and properly import a TF checkpoint as BertForTokenClassification?
Will appreciate any help. Thank you!

Hello everyone,
I have temporarily come up with a workaround for this. Not sure if it's the best solution but it works. What I did was I essentially merged what load_tf_weights_in_bert() and what part of BertPreTrainedModel.from_pretrained() was doing. BertPreTrainedModel is the parent class of BertForTokenClassification, so if you are trying to do something similar for BertFor{TaskName}, it should work too.
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path):
    config = BertConfig.from_json_file(bert_config)
    model = BertForPreTraining(config)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = BertForTokenClassification(config)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model

model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "/home/bert/pt_baseuncased/model.ckpt-98000")

Hi chrisgzf, thank you for the quick fix. I saw it's been integrated in the latest version.
I am trying to do quite the same using BertForSequenceClassification but the "AttributeError: 'BertForTokenClassification' object has no attribute 'bias'" still shows up.
Any ideas how I could use your fix for BertForSequenceClassification too ?
Thank you

Hi @stormskidd,
in my code snippet here (https://github.com/huggingface/pytorch-pretrained-BERT/issues/676#issuecomment-501526327), just change
model = BertForTokenClassification(config, num_labels=num_labels)
to
model = BertForSequenceClassification(config, num_labels=num_labels)
and it should work.

Thank you for the quick response.
So I tried this as you said :
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
    config = BertConfig.from_json_file(bert_config)
    model = BertForSequenceClassification(config, num_labels=num_labels)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = **BertForSequenceClassification**(config, num_labels=num_labels)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model

Then:
CONFIG_FILE = "Bert/multi_cased_L-12_H-768_A-12/bert_config.json"
model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "model.ckpt-6032", num_labels = 2)

But still got the error:
`
 in load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels)
2     config = BertConfig.from_json_file(bert_config)
3     model = BertForSequenceClassification(config, num_labels=num_labels)
----> 4     load_tf_weights_in_bert(model, ckpt_path)
5     state_dict=model.state_dict()
6     model = BertForSequenceClassification(config, num_labels=num_labels)
C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
89
90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')
92
93             elif l[0] == 'output_weights':
C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in getattr(self, name)
537                 return modules[name]
538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).name, name))
540
541     def setattr(self, name, value):
AttributeError: 'BertForSequenceClassification' object has no attribute 'bias''`
Also, I wandered in the source code where the error occurs :

     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')

I tried to change getattr(pointer, 'bias') to getattr(pointer, 'beta') but then I got a slightly different error:
C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     89
     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'beta')
     92
     93             elif l[0] == 'output_weights':

C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540
    541     def __setattr__(self, name, value):

AttributeError: 'BertLayerNorm' object has no attribute 'beta'

Hope it helps. Please let me know you think of any workaround for this !
By the way, I'm on windows using Anaconda with Python 3.7.1
Greetings,
Maxime

Hi Maxime (@stormskidd),
Do read my comment carefully. Change
model = BertForTokenClassification(config, num_labels=num_labels)
to
model = BertForSequenceClassification(config, num_labels=num_labels)
Please leave model = BertForPreTraining(config) (line 3) as is, and do not change it.
Cheers.

Oh, you're right I posted the wrong version of the many things I tried :(
Anyway, I'm afraid the error still shows up with the following:
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
    config = BertConfig.from_json_file(bert_config)
    model = BertForPreTraining(config)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = BertForSequenceClassification(config, num_labels=num_labels)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model

Then:
CONFIG_FILE = "/Bert/multi_cased_L-12_H-768_A-12/bert_config.json"
model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "model.ckpt-6032", num_labels = 2)

And :
C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     89
     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')
     92
     93             elif l[0] == 'output_weights':

C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540
    541     def __setattr__(self, name, value):

AttributeError: 'BertForPreTraining' object has no attribute 'bias'

Greetings,
Max

@stormskidd,
this is odd.... I actually specifically tested my code above on BertForSequenceClassification as well and I was able to successfully import TF weights. The code snippet looks like it would work...
Just want to check:

are you running the latest pytorch-pretrained-BERT from upstream master?
did you make any other changes to the source?
have you tried https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py and successfully converted your TF ckpt to a pytorch dump? if my code snippet above doesn't work for you, this should at least work for you.

Do let me know if you face any errors. I'm curious about this issue. Btw, check out my edit in the previous comment.
Chris

Hi, not sure I fully understand the issue here. What is the kind of tensorflow checkpoint you are trying to convert? Is it a pretrained model like the original Bert checkpoints or is it a fine-tuned model with additional elements (like a classification layer on top)?

Hi @thomwolf,
I am trying to convert a pretrained model like the original Bert checkpoints, except that I did additional pretraining on top of the released models with run_pretraining.py from the BERT repo. I then wish to fine-tune these pretrained models in pytorch, which is why I had to do this conversion. I am not importing any fine-tuned TF models.

Ok so you can just convert your Tensorflow model using the command line script (see here in the readme) store the converted pytorch model in a folder with the configuration file and then load it in a BertForTokenClassification model as follow:
BertForTokenClassification.from_pretrained('PATH_TO_YOUR_CONVERTED_MODEL')

Hi guys,
I saw your discussion and it gave me the idea to try the following:
It seems like I can use both the function 'load_BFTC_from_TF_ckpt' and the script 'convert_tf_checkpoint_to_pytorch.py' to load a pretrained Google model : multi_cased_L-12_H-768_A-12\bert_model.ckpt.
However the error occurs when I tried to do the same with a fined-tuned model (from Bert script run_classifier.py ran on GluON/MRPC -like data).
Is there any ways I can load a TF checkpoint fine-tuned model directly in Pytorch ? Or do I have to re-finetune it with the pytorch_pretrained_bert library ?
Thank you !
Max

Oh, you're right I posted the wrong version of the many things I tried :(
Anyway, I'm afraid the error still shows up with the following:
def load_BFTC_from_TF_ckpt(bert_config, ckpt_path, num_labels):
    config = BertConfig.from_json_file(bert_config)
    model = BertForPreTraining(config)
    load_tf_weights_in_bert(model, ckpt_path)
    state_dict=model.state_dict()
    model = BertForSequenceClassification(config, num_labels=num_labels)

    # Load from a PyTorch state_dict
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if 'gamma' in key:
            new_key = key.replace('gamma', 'weight')
        if 'beta' in key:
            new_key = key.replace('beta', 'bias')
        if new_key:
            old_keys.append(key)
            new_keys.append(new_key)
    for old_key, new_key in zip(old_keys, new_keys):
        state_dict[new_key] = state_dict.pop(old_key)

    missing_keys = []
    unexpected_keys = []
    error_msgs = []
    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):
        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)
    if len(missing_keys) > 0:
        logger.info("Weights of {} not initialized from pretrained model: {}".format(
            model.__class__.__name__, missing_keys))
    if len(unexpected_keys) > 0:
        logger.info("Weights from pretrained model not used in {}: {}".format(
            model.__class__.__name__, unexpected_keys))
    if len(error_msgs) > 0:
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
                           model.__class__.__name__, "\n\t".join(error_msgs)))
    return model

Then:
CONFIG_FILE = "/Bert/multi_cased_L-12_H-768_A-12/bert_config.json"
model = load_BFTC_from_TF_ckpt(CONFIG_FILE, "model.ckpt-6032", num_labels = 2)

And :
C:\ProgramData\Anaconda3\Lib\site-packages\pytorch_pretrained_bert\modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
     89
     90             elif l[0] == 'output_bias' or l[0] == 'beta':
---> 91                 pointer = getattr(pointer, 'bias')
     92
     93             elif l[0] == 'output_weights':

C:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540
    541     def __setattr__(self, name, value):

AttributeError: 'BertForPreTraining' object has no attribute 'bias'

Greetings,
Max

you can try:
model = BertForPreTraining.from_pretrained(BERT_DIR, from_tf=True)

Hello guys,
here is something I found that seem to do the trick for now:
in modeling.py:
In class class BertForSequenceClassification(BertPreTrainedModel):
Add :
        self.weight = Variable(torch.ones(2, config.hidden_size), requires_grad=True)
        self.bias = Variable(torch.ones(2), requires_grad=True)

to the attributes.
Obviously juste change the right class regarding your needs. I'll let you know if it provokes another error, but for now I can load in memory the trained model I couldn't load before.
Greetings,
Max

if use BertFor* class, will not initiate classifier layer.
load tf model for  predict/evaluate need to modify load_tf_weights_in_bert
 if re.fullmatch(r'[A-Za-z]+_\d+', m_name):                 l = re.split(r'_(\d+)', m_name)             else:                 l = [m_name]             if l[0] == 'kernel' or l[0] == 'gamma':                 pointer = getattr(pointer, 'weight')             elif l[0] == 'output_bias' or l[0] == 'beta':                 if pointer == model:                     pointer = getattr(pointer, 'classifier')                 pointer = getattr(pointer, 'bias')             elif l[0] == 'output_weights':                 if pointer == model:                     pointer = getattr(pointer, 'classifier')                 pointer = getattr(pointer, 'weight')             elif l[0] == 'squad':                 pointer = getattr(pointer, 'classifier')

Ok so you can just convert your Tensorflow model using the command line script (see here in the readme) store the converted pytorch model in a folder with the configuration file and then load it in a BertForTokenClassification model as follow:
BertForTokenClassification.from_pretrained('PATH_TO_YOUR_CONVERTED_MODEL_FOLDER')
For the tokenizer, you can use the one associated to the original TensorFlow model from which you did the fine-tuning since you probably didn't change the vocabulary itself.

Hello @thomwolf,
Yes, I am aware of the conversion script and from_pretrained() being able to load full models (PyTorch dumps) from converted TF checkpoints. However, in my use case, I did pre-training with BERT using the script run_pretraining.py from the BERT repo, and I wanted to do fine-tuning on the many checkpoint steps that I have saved, so it would make more sense for me to load the checkpoints directly.
However, I am aware that my use case is a very niche one, and the others here are talking about a different use case (loading TF finetuned models). Since my issue has been resolved, I will close this issue.


Enable installation and fix typos

I made this PR to speedup compilation and development of projects using Qaterial. I noticed that including Qaterial into my own project increased both the configuration and building time significantly by parsing Qaterial's CMake scripts and running Auto MOC.
In addition to that, this enabled me to cache the Qaterial builds in my Github workflows, saving minutes from each pipeline.
I rearranged the headers to expose only the ones consumed by external projects, and also updated QOlm to have installation support (https://github.com/OlivierLDff/QOlm/pull/7).
There are also two changes made here:
Added the option to build spdlog or retrieve it using find_package for the same reasons above. And also fixed a typo in the name of one of the components.
Note: I had to disable the configuration of installation scripts when spdlog's source code is included instead of being imported using find_package() because spdlog does not have export sets (using export() command) which gives an error when trying to include it in the installation scripts.

Ok this is lot to review
Can I extract from this PR the typo things to merge that first?
You should we move headers?
What does PUBLIC_HEADER exactly is for? Couldn't understand.
I think best move for spdlog is to completely remove it and use qWarning instead.
I also think that QOlm should be move as a private dependency because all the type of Qaterial are only meant to be used from qml.

Ok this is lot to review
Can I extract from this PR the typo things to merge that first?
You should we move headers?
What does PUBLIC_HEADER exactly is for? Couldn't understand.
I think best move for spdlog is to completely remove it and use qWarning instead.
I also think that QOlm should be move as a private dependency because all the type of Qaterial are only meant to be used from qml.

1- Regarding the typos: please do, yes. I can do that for you if you don't have time.
2/3- Private headers are used only during the compilation of Qaterial itself, they are not referenced by other apps consuming Qaterial (such as Qaterial Gallery). While public ones are the only ones that need to be packaged together with the compiled binaries since other apps would reference them. I used separate folders to correctly do target_include_directories(${QATERIAL_TARGET} PUBLIC....) and target_include_directories(${QATERIAL_TARGET} PRIVATE....) where the former denotes the ones to be installed and the second one includes only the headers required during compilation of Qaterial.
4- I'm OK with refactoring the code to completely remove the dependency on spdlog. Are you sure that it's not required for some feature?
5- Regarding QOlm: that actually makes sense but remember that for static library compilation I will still have to install QOlm together with Qaterial so making it private here wouldn't make that much difference.

Ok so I started to merge and simplify stuff in v1.4.5 branch.

6a452ab4f6debb55e90dc2ffccc5bfb01488aadb IconLabelPositionner -> IconLabelPositioner
ec2fe43b77d04c4810334b78d635f96c1ab6baea  & 19c621229ad340a9e9967038299d7121e1438834 : I moved sources around, so now the only public header is Utils.hpp, that is itself included by Qaterial.hpp. C++ public API of Qaterial is only to register/load ressources. So public headers of Qaterial don't have any dependency even to Qt. Version singleton is know private header, so I added version getter in Utils.hpp
e8da53bc91e29f90b8be7a6e1129efc9d52f044e spdlog to QLoggingCategory
917e387d0a4c390a20d1f38b9bce4a88f669ed1b (QOlm is now private dependency)

Now I will merge the install code (same I did with QOlm, I will make it optional with QATERIAL_ENABLE_INSTALL).

So I merged everything in https://github.com/OlivierLDff/Qaterial/releases/tag/v1.4.5
Feel free to reopen if something is missing


import issue

This might be a noob python question, but:
git clone https://github.com/a-akram-98/face_lib.git
cd face_lib/
virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
pip install face-library
...
(venv) jim@Jim-i9-Win10:src (master)$ python
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

from face_lib import face_lib
Traceback (most recent call last):
File "", line 1, in
File "/home/jim/source/face_lib/src/face_lib/init.py", line 1, in
from .face_lib import face_lib
File "/home/jim/source/face_lib/src/face_lib/face_lib.py", line 5, in
from .BlazeDetector import BlazeFaceDetector
File "/home/jim/source/face_lib/src/face_lib/BlazeDetector.py", line 5, in
from blazeFaceUtils import gen_anchors, AnchorsOptions
ModuleNotFoundError: No module named 'blazeFaceUtils'

How do I find the modules?

I have modified the repo and releases patch 1.1.3, please update the package
pip install face-library -U


[BUG] Embeddings ko: localAI, Ollama, Mistral

Describe the bug
I am using Flowise 2.0.1
I am trying to setup a workflow and I prefer to work with embeddings which are not OpenAI.
Using a LocalAI Embeddings => Bad Request
Using MistralAI Embeddings => Bad Request
Using Ollama Embeddings => Request to Ollama server failed: 500 Internal Server Error
The local embeddings (LocalAI, Ollama) are correctly available, tested with Postman they work.  In addition, a test to exclude issues of ports/communication/infrastucture has been performed from another machine: all ok.
The flow is a RAG with a 20 pages PDF in Qdrant.
As soon as OpenAI embeddings is in place, it all work smoothly 🫤
To Reproduce
LocalAI : (text-embedding-ada-002)
(pls consider that when I use a LocalAI chat model I have no issues at all)
The setup and error..

The correct answer of the model when debugging it from the LocalAI side, where all seems fine ...

Mistral:
(pls consider that I have a key fully working and when using a Mistral Chat model I have no issues at all)

Ollama (nomic-embed-text)

Steps to reproduce the behavior:
I am including the screenshots and the details are visible. There is anything special.
Expected behavior
I'd expect to be able to use any of the embeddings alternatives.
Screenshots
IIncluded
Setup

Installation : Docker
Flowise Version : 2.0.1
OS: Win
Browser : Chrome

If you are running both Ollama and Flowise on Docker, try this - https://docs.flowiseai.com/integrations/langchain/chat-models/chatollama#additional

same with LocalIA - https://docs.flowiseai.com/integrations/langchain/chat-models/chatlocalai#flowise-setup

same with LocalIA - https://docs.flowiseai.com/integrations/langchain/chat-models/chatlocalai#flowise-setup

I hope the following details will help solving the issues.
My attempts :
Flowise (docker) => LocalAI (docker)
Flowise (docker) => Ollama (no-docker, just in Windows)
Case: Flowise (docker) => LocalAI (docker)
There is - apparently - the ability of the LocalAI Chat model to reach anyway the endpoint even when in the form
http://<IP_ADDRESS>:8080
See here the success, but in this case I used OpenAI embedding, which is the only one working in this environment.
.
I have my answer in the chat.
Now I bring in the LocalAI embedding (keep in the LocalAI Chat model) and still use the endpoint
http://<IP_ADDRESS>:8080 in both (embedding and chat model).
As visible here (LocalAI debug view) the call to the endpoint is successful and the embedding is transforming the text into vectors.

....but there is a "Bad Request" in the chat.
Now I update the Base path to: http://host.docker.internal:8080/v1 as per your suggestion.
Still the endpoint is reached and the embedding works

But again the problem is still the same, Bad Request

Bottomline. My understanding is that the embedding endpoint works apparently well but Flowise is not and is throwing exceptions no matter the embedding I am using (common problem or different I cannot tell).
The Bad Request message behind the scenes is still like this

Case Flowise (docker) => Ollama (no-docker, just in Windows)
The Ollama endpoint is at http://<IP_ADDRESS>:11434 in the test I am keeping the LocalAI chat model and just using Ollama Embedding model (installed in Win, no docker).
The issue here is shown in this picture

while FLowise error detail being displayed in Docker is

Now let's repeat the same in Postman. The endpoint , in Flowise I use http://<IP_ADDRESS>:11434, in Postman is visible here. The Postman test is successful.

Last case. Mistral Embeddings (model name: mistral-embed).
The outcome is this :

with Flowise debug being

My 2 cents:
One could think that using stuff locally brings in issues with infrastructures, ports, etc which is understandable. My tests are here showing, when possible (LocalAI debug) that the requests are reaching the endpoints and processed. In Mistral case I also am excluding these variables and the embedding should perform as nicely as with OpenAI.

Closing the ticket as it contains multiple issues, not connected into one.
The three topics have been separated into different threads :
2976 Ollama
2977 LocalAI
2978 Mistral


Fixing WordPress mu-plugin instruction

Fixing mu-plugin instruction now that vendor file is no longer in git.
Alternative command would be:
git clone git@github.com:spatie/wordpress-ray && cd wordpress-ray && composer install -o --no-dev

Addresses: https://github.com/spatie/wordpress-ray/discussions/66

Thank you!


The 'socket' plugin has to be enabled, the 'web' plugin should be deactivated.

The Documentation is very poor...
I cant find the socket plugin , i dont know where to enabled it...
also the web plugin where to disabled...

If you are using PokemonGo-Bot, then this UI will not work for you.


Sometimes "onBeforeSnapToItem" is working as expected.

Is this a bug report, a feature request, or a question?
bug report

Have you followed the required steps before opening a bug report?
(Check the step you've followed - put an x character between the square brackets ([]).)

[x] I have read the guidelines regarding bug report.
[x] I have reviewed the documentation in its entirety, including the dedicated documentations :books:.
[x] I have searched for existing issues and made sure that the problem hasn't already been reported.
[x] I am using (v3.9.0) the latest plugin version.
[x] I am following the issue template closely in order to produce a useful bug report.

Have you made sure that it wasn't a React Native bug?
yes

Is the bug specific to iOS or Android? Or can it be reproduced on both platforms?
Both
Is the bug reproductible in a production environment (not a debug one)?
Haven't tried it in PROD, testing in QUAL

Environment

Expected Behavior

(Write what you thought would happen.)
Actual Behavior

Reproducible Demo
(Paste the link to a Snack example in which the issue can be reproduced. Please follow the guidelines for providing a Minimal, Complete, and Verifiable example.)

⚠ You did not follow the contributing guidelines!
As stated in these:

You need to fill out the issue template completely. This step is mandatory! Not doing so will result in your issue getting closed. Don't take this personally if this happens, and feel free to open a new issue once you've gathered all the information required by the template.

Unless you provide a Snack example in which the issue can be reproduced, as well as a complete description of everything you've tried, we won't be able to help you and the thread will be closed.

Sorry, please allow me to advertise for my open source library! ~
I think this library react-native-reanimated-carousel will solve your problem. It is a high performance and very simple component, complete with React-Native reanimated 2


Improve liveness and readiness probes accuracy

What would you like to be added:
The liveness and readiness probes should succeed only when the operator is fully operational. This means they should report a health status only when all the components, i.e, webhooks, extension API servers, controllers, have started and are servicing requests.
Why is this needed:
At the moment, the liveness and readiness probes report a healthy status unconditionally, as soon as the controller-runtime manager has started, despite the operator and its components (webhooks, extension API servers, controllers) may not be operational yet.
That can happen over the period of time it takes for the certificates generated by cert-controller to be propagated into the secret volume mount for example.
In such cases, having the probes reporting an accurate status can help recovering issues, and other components correctly waiting for the operator readiness.
Completion requirements:
This enhancement requires the following artifacts:

[ ] Design doc
[ ] API change
[ ] Docs update

The artifacts should be linked in subsequent comments.

I believe this issue is already address with https://github.com/kubernetes-sigs/kueue/pull/1676 (and cherry-picked to 0.5.x)/
@astefanutti PTAL, if so then we can close.

@mimowo thanks for the update. I'd say #1676 fixes most of this issue.
There are a couple of things in the scope of this issue that we may still want to address like:

Including the status of the visibility API server in the readiness check
Improving the accuracy / usefulness of the liveness check

So I think we can close this one, and we can create some finer-grained ones for the two left-overs above. WDYT?

Including the status of the visibility API server in the readiness check

Seems like a valid improvement indeed, otherwise we may be getting errors from the server.

Agreed.

Improving the accuracy / usefulness of the liveness check
Any concrete example, other than the visibility API server, where we could improve accuracy?

I wonder if probing for the webhooks in the liveness probe, as done in the readiness probe now with #1676, would be useful to mitigate the situation where cert-controller might fail generating / injecting the certificates at start time, and possibly over cert rotation. Do you think that would be an improvement over the current liveness probe implementation?

So I think we can close this one, and we can create some finer-grained ones for the two left-overs above. WDYT?

I prefer more fine-grained issues, so that we can prioritize / close them independently.

Sounds good, I'll create them and close this one.

I wonder if probing for the webhooks in the liveness probe

Oh, interesting. I assumed the liveness probe = readiness probe, but it indeed the liveness probe = health probe in controller manager: https://github.com/kubernetes-sigs/controller-runtime/blob/7032a3cc91d2afc4c2d54e4a4891cf75da9f75f5/pkg/manager/internal.go#L281-L285.
And currently, the health probe is just ping in Kueue. So, yes I think using the liveness probe checking webhook server is better so that we can see if the server is down. cc @trasc @alculquicondor

/remove-lifecycle stale

/assign

@astefanutti  since this question was posted we have added the readiness probe to Kueue in https://github.com/kubernetes-sigs/kueue/pull/1676 (and later improved in follow ups).
Do we still have some known gaps to cover in the readiness probe, or scenarios that we want to cover in the liveness probe?
If not, I would suggest to park  this issue (close) until we have such scenarios.

@mimowo apologies for the late reply.
I think what's been done to improve the readiness mostly addresses this issue, and I agree with your suggestion to close it, and create some finer-grained ones if needed.

/close


Clubs page: have club leaders on top

It would be great in the Club's page to have the Club leaders sticky on top.
In the example below, Tim should be before Christos.

For context, it would be helpful so you can easily see the human points of contact.

Working on it


Refactor Plans Controller

Extract methods from plans controller. For exemple, there is many method related to pdf plan generation. Will be accepted multiple pull requests.

Hi, I'd like to take a look into this (part of it, at least). I'd like to extract the plan creation methods into a helper class, named something like PlansHashImporter, which would be called on line 317 of the controller.
If that's a suitable solution, where in the project would you like the importer? I was thinking maybe in a directory app/importers.

Hello @nathandonato, at some point I have searched to use rake tasks to import and export methods of a plan.
But inside the app/helpers there are files to be used as helpers. Refactor some methods to plans_helper.rb, it might be the solution for what we want.

Hi @eduqg, I opened up a PR for the import changes. I put the new code in an app/importers directory because I felt importers are more specific than helpers, but I'd be happy to move them wherever you'd like. Just let me know in the PR review.


Update government-feature-availability.md

PER
https://portal.microsofticm.com/imp/v3/incidents/details/236694659/home
The Microsoft Power Platform Build Tools for Azure DevOps is not available for the GCC regions. Our documentation needs to support this indication. Related PR:
https://github.com/MicrosoftDocs/power-platform/pull/752

@Funken1766
Thanks for the update. Taking to live via internal repo.


Dimming does not work in Hyper 2.0

I'm using Hyper 2.0 and I've tried the default setting of 0.6 for interactivePaneOpacity as well as a low value like 0.2 and am not seeing any dimming of inactive panes.  Is this a known issue or am I doing something wrong? Thanks!

It works as expected for me.
Can you give me your .hyper.js config file to help me to reproduce?

Sure: https://gist.github.com/jrust/9e2bdd283cc02300139f85235dc9b39c
I'm also on Windows 10.  Thanks

Sorry for the delay.
It could be a conflict with other plugin (hyper-opacity ?)

Hmm, I tried removing all my plugins, but no luck.  I'll keep poking around

@chabou
I'm having the same problem with indicatorPrefix as well as inactivePaneOpacity.  The only way I was able to get any configurations to change was by manually editing the index.js in the ~/.hyper_plugins/node_modules/hyper-pane directory.
Ubuntu      14.04
node        v6.7.0
npm         3.10.3
Hyper       2.1.2
hyper-pane  2.1.4

Bellow is my .hyper.js configurations, which are the same for:
~/.hyper.js
~/.config/Hyper/.hyper.js
~/.hyper_plugins/node_modules/index.js:3:defaultConfig
.hyper.js

// Future versions of Hyper may add additional config options,
// which will not automatically be merged into this file.
// See https://hyper.is#cfg for all currently supported options.

module.exports = {
  config: {
    // choose either `'stable'` for receiving highly polished,
    // or `'canary'` for less polished but more frequent updates
    updateChannel: 'stable',

    // default font size in pixels for all tabs
    fontSize: 12,

    // font family with optional fallbacks
    fontFamily: 'Menlo, "DejaVu Sans Mono", Consolas, "Lucida Console", monospace',

    // default font weight: 'normal' or 'bold'
    fontWeight: 'normal',

    // font weight for bold characters: 'normal' or 'bold'
    fontWeightBold: 'bold',

    // line height as a relative unit
    lineHeight: 1,

    // letter spacing as a relative unit
    letterSpacing: 0,

    // terminal cursor background color and opacity (hex, rgb, hsl, hsv, hwb or cmyk)
    cursorColor: 'rgba(248,28,229,0.8)',

    // terminal text color under BLOCK cursor
    cursorAccentColor: '#000',

    // `'BEAM'` for |, `'UNDERLINE'` for _, `'BLOCK'` for █
    cursorShape: 'BLOCK',

    // set to `true` (without backticks and without quotes) for blinking cursor
    cursorBlink: false,

    // color of the text
    foregroundColor: '#fff',

    // terminal background color
    // opacity is only supported on macOS
    backgroundColor: '#000',

    // terminal selection color
    selectionColor: 'rgba(248,28,229,0.3)',

    // border color (window, tabs)
    borderColor: '#333',

    // custom CSS to embed in the main window
    css: '',

    // custom CSS to embed in the terminal window
    termCSS: '',

    // if you're using a Linux setup which show native menus, set to false
    // default: `true` on Linux, `true` on Windows, ignored on macOS
    showHamburgerMenu: '',

    // set to `false` (without backticks and without quotes) if you want to hide the minimize, maximize and close buttons
    // additionally, set to `'left'` if you want them on the left, like in Ubuntu
    // default: `true` (without backticks and without quotes) on Windows and Linux, ignored on macOS
    showWindowControls: '',

    // custom padding (CSS format, i.e.: `top right bottom left`)
    padding: '12px 14px',

    // the full list. if you're going to provide the full color palette,
    // including the 6 x 6 color cubes and the grayscale map, just provide
    // an array here instead of a color map object
    colors: {
      black: '#000000',
      red: '#C51E14',
      green: '#1DC121',
      yellow: '#C7C329',
      blue: '#0A2FC4',
      magenta: '#C839C5',
      cyan: '#20C5C6',
      white: '#C7C7C7',
      lightBlack: '#686868',
      lightRed: '#FD6F6B',
      lightGreen: '#67F86F',
      lightYellow: '#FFFA72',
      lightBlue: '#6A76FB',
      lightMagenta: '#FD7CFC',
      lightCyan: '#68FDFE',
      lightWhite: '#FFFFFF',
    },

    // the shell to run when spawning a new session (i.e. /usr/local/bin/fish)
    // if left empty, your system's login shell will be used by default
    //
    // Windows
    // - Make sure to use a full path if the binary name doesn't work
    // - Remove `--login` in shellArgs
    //
    // Bash on Windows
    // - Example: `C:\\Windows\\System32\\bash.exe`
    //
    // PowerShell on Windows
    // - Example: `C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\powershell.exe`
    shell: '',

    // for setting shell arguments (i.e. for using interactive shellArgs: `['-i']`)
    // by default `['--login']` will be used
    shellArgs: ['-i'],

    // for environment variables
    env: {},

    // set to `false` for no bell
    bell: 'SOUND',

    // if `true` (without backticks and without quotes), selected text will automatically be copied to the clipboard
    copyOnSelect: false,

    // if `true` (without backticks and without quotes), hyper will be set as the default protocol client for SSH
    defaultSSHApp: true,

    // if `true` (without backticks and without quotes), on right click selected text will be copied or pasted if no
    // selection is present (`true` by default on Windows and disables the context menu feature)
    // quickEdit: true,

    // URL to custom bell
    // bellSoundURL: 'http://example.com/bell.mp3',

    // for advanced config flags please refer to https://hyper.is/#cfg
  },

  // a list of plugins to fetch and install from npm
  // format: [@org/]project[#version]
  // examples:
  //   `hyperpower`
  //   `@company/project`
  //   `project#1.0.1`
  plugins: ["hyper-monokai-extended", "hyper-pane"],

  paneNavigation: {
    debug: false,
    hotkeys: {
      navigation: {
        up: 'ctrl+alt+up',
        down: 'ctrl+alt+down',
        left: 'ctrl+alt+left',
        right: 'ctrl+alt+right'
      },
      jump_prefix: 'ctrl+alt', // completed with 1-9 digits
      permutation_modifier: 'shift', // Added to jump and navigation hotkeys for pane permutation
      maximize: 'shift+ctrl+x'
    },
    showIndicators: true, // Show pane number
    indicatorPrefix: '⫎⦗⌬⦘', // Will be completed with pane number
    indicatorStyle: { // Added to indicator <div>
      position: 'absolute',
      top: 0,
      left: 0,
      fontSize: '12px'
    },
    focusOnMouseHover: false,
    inactivePaneOpacity: 0.6 // Set to 1 to disable inactive panes dimming
  },

  // in development, you can create a directory under
  // `~/.hyper_plugins/local/` and include it here
  // to load it and avoid it being `npm install`ed
  localPlugins: [],

  keymaps: {
    // Example
    'window:devtools': 'ctrl+alt+o',
  },
};

@blakejwc your paneNavigation should be in config section (sorry for the delay).


MINOR: [C++] Make the utilities used to implement run_end_encode available to other compute kernels

Rationale for this change
Make it easier to implement correct kernels dealing with run-end encoded data.
What changes are included in this PR?
Extraction of code to a header so kernels like REE filter kernels (see #35001) can use them.
Are these changes tested?
Indirectly via the run_end_encode and run_end_decode tests.
Are there any user-facing changes?
No, these are internal compute headers.

@zeroshade @benibus


Feature Request: Avoid autostarting nvim-ghost

Hello again, please allow me to open 2 issues at once because this plugin seems extremely neat and seems to just work:tm: without too much effort :sweat_smile:
The issue
Upon installing the plugin, it seems to start the nvim-ghost server automatically. On the other hand, with the vim-ghost plugin, you had to explicitly start the server with a command :GhostStart.
While this is definitely very comfortable, I very rarely need GhostText, so having nvim-ghost launch itself every time I open neovim does not feel very ideal to me; I end up disabling the plugin all together.
Therefore, it would be nice to have the ability to either autostart nvim-ghost, or to explicitly start it using some sort of command (like :GhostStart).
At this point in time, I wasn't able to spot any mentions of this (i.e. not autostarting the plugin) being possible.
My installation using packer:
    use { 'subnut/nvim-ghost.nvim', disable=false, -- {{{
        run = ':call nvim_ghost#installer#install()',
    } -- }}}

Food for thought
It might even be interesting to allow autostarting based on autocommand events (i.e. when a certain filename is read, similar to your example in the readme). I suppose autostarting using autocommands would be managed by the plugin users, and will simply call whatever command allows to start the server. Glancing at how the plugin links the text box and neovim, I am not too sure what kind of events could come in question for this.

so having nvim-ghost launch itself every time I open neovim

nvim-ghost doesn't launch every time neovim is opened.
It's launched only once - when neovim is started for the first time.
When neovim is started for the first time after starting the OS, nvim-ghost launches a background process (the python binary), which keeps running even after neovim is closed. This background process orchestrates everything, which keeps the experience seamless.
I don't think it will be possible to have an explicit command to start the server and also keep everything working as seamless as it currently is. I had tried to do that when I created this plugin, but was unsuccessful.

nvim-ghost doesn't launch every time neovim is opened.
It's launched only once - when neovim is started for the first time.

Keeping the above in mind, do you still feel like autostarting nvim-ghost is necessary?

Thank you for the clarification! It wasn't clear to me that nvim-ghost launches a process in the background that other neovim instances essentially just attach to (if I understand you correctly now). Maybe I missed in in the README, or maybe it wasn't clearly mentioned.
The confusion must have been fueled from the fact that there is a message saying [nvim-ghost] Server running every time I launch vim (I am aware that I can silence this message). But together with your insights, I see now that it doesn't actually launch it every time anew.
With all that, I have 2 points of view now regarding autostarting nvim-ghost.

Since it only starts once and then sits in the background, I think it should be fine to leave it as it currently works. Silencing the message will basically render the whole experience pretty transparent.
On the other hand, since I roughly use nvim-ghost/GhostText five times a year (number is probably a bit exaggerated), it would still be nice to have the ability to "manually" launch this background process (and maybe also then provide a possibility to kill/close said process).

Regarding the second point: the rest can still work like it currently does; so when you close neovim, the process remains alive in the background. You would merely transfer starting/stopping of the background process to the user, instead of having neovim launch it when opening it the first time. Maybe a user defined setting could control the behaviour here: (i) automatically launch on startup (default), or (ii) launch/stop manually using certain functions/commands/etc

Initial implementaion - https://github.com/subnut/nvim-ghost.nvim#start-manually
Please test and let me know of any unexpected behaviour.

Thank you very much for the implementation! Let me tell you what I have tested, what I have found, and what I had expected.
I use the following command to observe the nvim-ghost processes that are running in the background.
watch -n 0.1 pgrep -a ghost

Default auto starting behaviour

When opening nvim and no processes are running, I will see 2 processes being spawned. They stick around after exiting nvim.
When I open nvim again, 2 new processes appear but quickly vanish again (I suspect they detected an already running instance and then exited).
Launching GhostText from the browser (assuming nvim is open) will directly sync a text field with nvim.

Everything works as expected here, keeping in mind the clarifications you provided earlier in this issue.
Enabling the start manually feature
Initial launch of nvim
Adding let g:nvim_ghost_disabled = 1 or vim.g.nvim_ghost_disabled = 1 (depending on VimL or Lua) has the following immediate effects:

No nvim-ghost processes will be spawned upon opening nvim (assuming none are running yet)
Launching GhostText from the browser will have it complain that it cannot connect to the editor (which makes sense, since nvim-ghost is not running)

Manually starting nvim-ghost
Now, running :GhostTextEnable will cause the following:

2 new nvim-ghost processes appear
Launching GhostText from the browser will sync the text field with vim

Everything works as expected thus far!
Closing nvim after manual launch
Since I manually launched the nvim-ghost process, I was under the impression that the process should exit once I leave nvim as well.
However, it might be annoying to have the process be taken down every time (?), so maybe an additional command such as :GhostTextDisable would make sense — Since I enabled it myself, I should also clean up after and disable it myself.
I am a little split on how to handle this, but I think automatically terminating the process in one way or another would make sense, mainly because it was enabled manually, meaning I don't want to keep it around all the time.
Launching nvim again
Here is where the behaviour starts to diverge from my expectations.
Since the nvim-ghost processes are still running after exiting nvim, I would assume that the plugin catches on to that and attaches to it upon opening nvim. After opening nvim now, I cannot yet use GhostText from the browser.
The unexpected behaviour here is that the nvim-ghost process is running, but the browser cannot connect to the editor.
Manually starting nvim-ghost again
Similar to the auto starting, running :GhostTextEnable will make 2 new nvim-ghost processes appear, which quickly vanish again.
Now GhostText in the browser can connect to the editor.
This works as expected
Conclusion from preliminary tests
Everything seems to behave as expected. The only problem lies in that the plugin does not seem to catch on to an existing nvim-ghost process.
However, seeing that the auto starting behaviour also launches new processes every time I start nvim, makes me suspect that the plugin does not catch onto running processes in this case either. Based on what I see in the pgrep output, the plugin just happens to see it because the new processes exited immediately. I may be completely wrong on this here of course!
With that said; here is my verdict.

Not starting nvim-ghost every time I start nvim seems to work great!
Because of this non-automatic behaviour, I would somehow expect the nvim-ghost process to also terminate after exiting vim. Here, I have two more thoughts.

Another :GhostTextDisable command would do the trick! It will be easy enough to add an autocommand myself into the config, which would run this upon exiting vim (or I can run this whenever I am done with GhostText).
One could add another option which tells the plugin to auto-terminate the nvim-ghost process upon exiting (in case start manually feature is enabled). That would avoid adding the autocommand on the user's end. A :GhostTextDisable command would still be handy, in case this auto-terminate is disabled.

Food for thought: In case my suspicion, about new processes being launched and exiting right away because another one is already there, is true. Wouldn't it be cool if the plugin connects to the still running nvim-ghost process instead of having to run :GhostTextEnable again? And if no process is running, well then I will have to manually start it. I imagine this could also be interesting for the autostarting behaviour since it avoids launching 2 processes every time nvim is launched. Maybe by storing the PIDs one could check for it more efficiently. I don't know how feasible it is to implement this though, and if it still fits in the scope of this issue. At any rate, I would really only consider this food for thought in particular as a "for fun" improvement since the 2 new processes clearly do not cause any problems (I didn't even know about them until making my tests here).

Attaching to the running nvim-ghost process does not seem to work properly anymore (I hope it is not a silly issue in my config). It observe the problem in both automatic and manual starting.
Assuming automatic starting; when launching nvim for the first time, the 2 nvim-ghost processes are being started. I can launch GhostText in the browser, and everything will be synced in nvim.
Upon closing nvim, and reopening it, the "new processes being launched" bug seems to be gone now! However, once inside of nvim (after the first time), the syncing with GhostText in the browser does not work.
Interestingly, the browser does not complain about not finding the editor, and when I un-focus the browser window, the selected text area has an orange outline (as opposed to blue if it is synced properly).
Not sure if this is a bug that was introduced by accident or not. If you cannot reproduce it, I can try to isolate the issue on my side.

Syncing should work now (3a4e443)

Default auto starting behaviour

When opening nvim and no processes are running, I will see 2 processes being spawned. They stick around after exiting nvim.
When I open nvim again, 2 new processes appear but quickly vanish again (I suspect they detected an already running instance and then exited).
Launching GhostText from the browser (assuming nvim is open) will directly sync a text field with nvim.

Everything works as expected here, keeping in mind the clarifications you provided earlier in this issue.

The processes shouldn't spawn anymore. They weren't expected to spawn. They were being spawned due to a bug.

Enabling the start manually feature
Initial launch of nvim
Adding let g:nvim_ghost_disabled = 1 or vim.g.nvim_ghost_disabled = 1 (depending on VimL or Lua) has the following immediate effects:

No nvim-ghost processes will be spawned upon opening nvim (assuming none are running yet)
Launching GhostText from the browser will have it complain that it cannot connect to the editor (which makes sense, since nvim-ghost is not running)

Manually starting nvim-ghost
Now, running :GhostTextEnable will cause the following:

2 new nvim-ghost processes appear
Launching GhostText from the browser will sync the text field with vim

Everything works as expected thus far!

Can confirm, everything is as expected.

Closing nvim after manual launch
Since I manually launched the nvim-ghost process, I was under the impression that the process should exit once I leave nvim as well.
However, it might be annoying to have the process be taken down every time (?), so maybe an additional command such as :GhostTextDisable would make sense — Since I enabled it myself, I should also clean up after and disable it myself.
I am a little split on how to handle this, but I think automatically terminating the process in one way or another would make sense, mainly because it was enabled manually, meaning I don't want to keep it around all the time.
Launching nvim again
Here is where the behaviour starts to diverge from my expectations.
Since the nvim-ghost processes are still running after exiting nvim, I would assume that the plugin catches on to that and attaches to it upon opening nvim. After opening nvim now, I cannot yet use GhostText from the browser.
The unexpected behaviour here is that the nvim-ghost process is running, but the browser cannot connect to the editor.
Manually starting nvim-ghost again
Similar to the auto starting, running :GhostTextEnable will make 2 new nvim-ghost processes appear, which quickly vanish again.
Now GhostText in the browser can connect to the editor.
This works as expected
Conclusion from preliminary tests
Everything seems to behave as expected. The only problem lies in that the plugin does not seem to catch on to an existing nvim-ghost process.

I'll come back to this later (in the next comment).

However, seeing that the auto starting behaviour also launches new processes every time I start nvim, makes me suspect that the plugin does not catch onto running processes in this case either. Based on what I see in the pgrep output, the plugin just happens to see it because the new processes exited immediately. I may be completely wrong on this here of course!
Food for thought: In case my suspicion, about new processes being launched and exiting right away because another one is already there, is true. Wouldn't it be cool if the plugin connects to the still running nvim-ghost process instead of having to run :GhostTextEnable again? And if no process is running, well then I will have to manually start it. I imagine this could also be interesting for the autostarting behaviour since it avoids launching 2 processes every time nvim is launched. Maybe by storing the PIDs one could check for it more efficiently. I don't know how feasible it is to implement this though, and if it still fits in the scope of this issue. At any rate, I would really only consider this food for thought in particular as a "for fun" improvement since the 2 new processes clearly do not cause any problems (I didn't even know about them until making my tests here).

Should be fixed by now. See my comment above (The processes shouldn't spawn anymore. They weren't expected to spawn. They were being spawned due to a bug.).

With that said; here is my verdict.

Not starting nvim-ghost every time I start nvim seems to work great!
Because of this non-automatic behaviour, I would somehow expect the nvim-ghost process to also terminate after exiting vim. Here, I have two more thoughts.

Another :GhostTextDisable command would do the trick! It will be easy enough to add an autocommand myself into the config, which would run this upon exiting vim (or I can run this whenever I am done with GhostText).
One could add another option which tells the plugin to auto-terminate the nvim-ghost process upon exiting (in case start manually feature is enabled). That would avoid adding the autocommand on the user's end. A :GhostTextDisable command would still be handy, in case this auto-terminate is disabled.

Will comment on this later (i'm sleepy right now)

Edit: Looking through the earlier messages in this issue, it appears that no new processes are being spawned after the first time (in case of the auto starting behaviour, and as a result also in the case of starting manually I suppose). So I am unsure what these two processes are that I see popping up. Apologies if I got something wrong here again!

Actually, I forgot to mention this as a technicality, but back then, atleast one extra process was being spawned. In order to increase efficiency, just as you mentioned, I moved some of the code to neovim, which was the root cause of the bug that I mentioned in my last comment (the bug that stopped syncing altogether). :sweat_smile:

BTW, it's nice to see such detailed reports from a user. Keep up the good work!
(You're the first person I've seen who's submitted such details. Thanks a lot, again.)

Syncing should work now (3a4e443)

Syncing from the browser now reports that the editor has "disconnected" immediately.
To make sure it was nothing on my side, I completely uninstalled and re-installed the plugin but the problem persists.
Also, now I observe 4 processes being spawned instead of 2. Not sure if the issue could be related :sweat_smile: Though, I would suspect that the call chanclose(l:connection) is misplaced and causes the channel to be closed too soon (I only looked at the commit you highlighted).

And thank you as well! I'm always unsure if I come off as rude or anything because of the long reports, but at the same time I can imagine that more information is always helpful. After all, such a detailed report is my best effort in helping as much as I can!

Phew... finally. Fixed.
Sorry, this has been a hectic week.
Now that the plugin is back to working normally, I can look into this (auto-starting) issue better.

Oops, wrong issue ⬆

Syncing should work now (3a4e443)

Good thing I forgot to press the comment button, it works great now! (it did not work a few days ago for some reason).
Only other "problem" I notice is that when closing the [Scratch] buffer that is opened in nvim, the focused textbox in the browser does not get un-selected. If I type anything in the text area, the browser reports that the editor has disconnected (which makes sense, since I closed the corresponding buffer). I am trying this with the start manually feature (I did not try with the auto-starting behaviour).
Also, now I observe 4 processes being spawned instead of 2. Not sure if the issue could be related :sweat_smile:
Though, I would take a random guess and suspect that the call chanclose(l:connection) is misplaced (I only looked at the commit you highlighted).

And thank you as well! I'm always unsure if I come off as rude or anything because of the long reports, but at the same time I can imagine that more information is always helpful. After all, such a detailed report is my best effort in helping as much as I can!

Awesome! The syncing works great now! (At least with the start manually method).
Only other issue I still notice is that when I close the [Scratch] buffer in neovim that gets opened after syncing with the browser, the focused text area does not get deselected (i.e. the blue outline is still there).
After I type something in the still focused text area in the browser, it will report that the editor has disconnected (which makes sense, since I closed the corresponding buffer).
I only tried this with the start manually feature for now as I suspect it is using the exact same code that the auto-starting behaviour under the hood! (Also now I see 5 nvim-ghost processes being spawned in case the issue might be related)

And if you got a notification for a comment that does not exist, I just deleted it because my browser was bamboozling me, sorry for the confusion

Only other issue I still notice is that when I close the [Scratch] buffer in neovim that gets opened after syncing with the browser, the focused text area does not get deselected (i.e. the blue outline is still there).

Bug, fixed in v0.3.2

I hope I am not doing anything wrong, but the problem does not seem to be resolved on my end. Closing the [Scratch] buffer in neovim still does not deselect the text area in the browser. The behaviour is still the same as I described before 🤔
(Also I just noticed, the text area does not get deselected even after closing neovim alltogether)

problem does not seem to be resolved on my end.

Yeah, my bad. I had fixed the bug in v0.3.2, but v0.4.0 reintroduced it due to a mistake in the git command I used to revert v0.2.5 (the commit where we shifted from threading to multiprocessing)

Everything works flawlessly now! At a glance I did not notice anything offending that remains! That said, the initial point in my issue has been addressed and is fully functional!
If there is nothing else left on this front which I am forgetting about, then feel free to finally close this issue!
Thanks a lot for the effort, and I hope I wasn't causing you too much trouble! You rock! o/

Closing nvim after manual launch
Since I manually launched the nvim-ghost process, I was under the impression that the process should exit once I leave nvim as well.

This is the ideal behaviour, yes. But it also creates the possibility of a race condition.
Say, this happens -

Open neovim
:GhostTextEnable
Server starts
Exit neovim
Server starts to stop
Open neovim
:GhostTextEnable
Neovim sees that there's already an existing server, and tries connecting to it instead of starting its own server
Server finishes stopping and exits.
Neovim fails to connect to the server, and GhostText doesn't work!

This problem wouldn't happen if the binary exits fast, but this is python we're talking about.
If I ever re-write this in Golang, this might be a feasible option.

However, it might be annoying to have the process be taken down every time (?), so maybe an additional command such as :GhostTextDisable would make sense — Since I enabled it myself, I should also clean up after and disable it myself.

But then, what if there are two (or more) running instances of neovim that are both connected to GhostText? In that case, what should the :GhostTextDisable command do?

If it causes the server to exit, then running :GhostTextDisable in any one neovim instance causes the rest of them to disconnect too.
If it just disconnects the current neovim instance, then, again, there are two options

Server never exits (no race conditions)
Server exits when all neovim sessions have run :GhostTextDisable (same race condition as above)

Also, please see this again -
https://github.com/subnut/nvim-ghost.nvim#start-manually
The variable name has been changed in https://github.com/subnut/nvim-ghost.nvim/commit/10242d4e6964c38ad052b0cbb45fa500daab2c0d

If it causes the server to exit, then running :GhostTextDisable in any one neovim instance causes the rest of them to disconnect too.

BTW this can be done by running :call nvim_ghost#helper#kill_server()

This problem wouldn't happen if the binary exits fast, but this is python we're talking about.

I take that back. Looks like it exits almost instantly! 😲

Try out the auto-exit branch

Also, please see this again - https://github.com/subnut/nvim-ghost.nvim#start-manually
The variable name has been changed in 10242d4

Awesome! The renamed variable and command are much clearer now! However, there appears to be a small issue that creeped in. I get the following error: E184: No such user-defined command: GhostTextEnable.
Looking at the code and the commit, it get the impression that there is one place where you forgot to rename :GhostTextEnable to :GhostTextStart. The following solves the issue for me and things behave as they did previously before the command was renamed!
diff --git a/plugin/nvim_ghost.vim b/plugin/nvim_ghost.vim
index 1c39f90..314c304 100644
--- a/plugin/nvim_ghost.vim
+++ b/plugin/nvim_ghost.vim
@@ -54,7 +54,7 @@ if !g:nvim_ghost_autostart
         \  let g:nvim_ghost_disabled = 0
         \| call nvim_ghost#init()
         \| doau <nomodeline> nvim_ghost UIEnter
-        \| delcommand GhostTextEnable
+        \| delcommand GhostTextStart
   finish
 endif

And I see the concerns around the auto-exit behaviour. I shall give the auto-exit branch a try later hopefully, and I will report back on whether things ultimately behaved as anticipated or not!

However, there appears to be a small issue that creeped in. I get the following error: E184: No such user-defined command: GhostTextEnable

Yeah, this has already been fixed by the auto-exit branch. That's why I suggested you to use that branch :sweat_smile:
Anyways, this has now been merged into the main branch. Do report further issues (if any).

Phew
Quite a journey, wasn't it?
Thanks for sticking with me! Your detailed reports helped me immensely!

Oh I see, my bad for not having tried the auto-exit branch then! But I confirm that everything works wonderfully! It appears that the processes are being killed after each nvim instance has exited (at least each one that ran :GhostTextStart).
It was quite the journey indeed, and I want to thank YOU as well for the time and effort!