README.md Update for Docker-compose file

I could not find instructions for editing docker-compose. But as I understand we should edit unavailabilityMessage, LINK, and API_Key values. Can you also add a short description of it? Thank you for the open-source code.

hi @emirsaidh i think the answer for your qustion is here Setup.


Issue with supported game : Palworld : Filestructure

When executing the extraction, files are generated but not in the expected file structure used by Steam.
I'm trying to export a single player, non-multiplayer flagged world containing 1 character (mine, the host).
A new "Level" folder is generated containing the file 01.sav
The "Level.sav" file is not generated in the base folder
When importing the generated files to the preprepared Steam save folder, the save appears to show correct information regarding the character. But when attempting to load the save, the game forces me to create a new character and fails to load the existing character or base structures.
Generated files and structure :

Im having the same issue, i have also tried renaming but nothing seems to work

Has anyone found any solutions? I'm having the same problem.

It's the game that does it, not the extractor - you can see the files for yourself. They changed the Game Pass and Xbox save format because they were having issues:
https://github.com/Z1ni/XGP-save-extractor/issues/136#issue-2177138621

Yeah, I tried using the extractor and a tool to fix a character and ended up breaking all characters. I'm waiting, hoping there will be a patch that fixes it:
https://www.nme.com/news/gaming-news/latest-palworld-update-is-causing-xbox-save-data-to-be-deleted-3615814


Rechunker Integration

I suspect(ed) that integrating rechunker will speed up our current workflow. However, I'm running some tests on GCP that give a more nuanced view. This issue is just meant to document that process.
1 Degree Replay Dataset
Everything listed below is relevant to the 1 degree replay dataset, found here. I'm running on GCP using a single c2-standard-60 node (60 vCPUs, 240GB memory, amd64) in zone us-central1-c. I'm using the default threaded scheduler with dask. I'm pulling data from s3 and storing to a persistent local storage there (/contrib in the parallel works cluster).
Using default dask .rechunk
Accessed by xarray's .chunk method results in download, rechunk, and store time of ~1 minute per day (i.e., 15 seconds per DA cycle).
Using rechunker
With max_mem=30GB based on my assumption of ~8 dask workers per node results in 1 min 40 seconds per day (i.e., 25 seconds per DA cycle).
Additionally, I cannot seem to properly append zarr stores in time. No matter what I get the following error unless I delete the target store - not the temp_store. Despite what the error says, I am very certain that this is being deleted.

The error
Traceback (most recent call last):
  File "/contrib/Tim.Smith/ufn/scripts/read_from_s3.py", line 38, in <module>
    replay.store_dataset(ds, **kw)
  File "/contrib/Tim.Smith/UFS2ARCO/src/UFS2ARCO/ufsdataset.py", line 211, in store_dataset
    self._store_data_vars(xds, **kwargs)
  File "/contrib/Tim.Smith/UFS2ARCO/src/UFS2ARCO/ufsdataset.py", line 248, in _store_data_vars
    chunk_plan = rechunk(
                 ^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/rechunker/api.py", line 303, in rechunk
    copy_spec, intermediate, target = _setup_rechunk(
                                      ^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/rechunker/api.py", line 464, in _setup_rechunk
    copy_spec = _setup_array_rechunk(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/rechunker/api.py", line 599, in _setup_array_rechunk
    target_array = _zarr_empty(
                   ^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/rechunker/api.py", line 147, in _zarr_empty
    return store_or_group.empty(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/hierarchy.py", line 1194, in empty
    return self._write_op(self._empty_nosync, name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/hierarchy.py", line 935, in _write_op
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/hierarchy.py", line 1200, in _empty_nosync
    return empty(store=self._store, path=path, chunk_store=self._chunk_store, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/creation.py", line 312, in empty
    return create(shape=shape, fill_value=None, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/creation.py", line 203, in create
    init_array(
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/storage.py", line 451, in init_array
    _init_array_metadata(
  File "/contrib/Tim.Smith/miniconda3/envs/ufs2arco/lib/python3.11/site-packages/zarr/storage.py", line 521, in _init_array_metadata
    raise ContainsArrayError(path)
zarr.errors.ContainsArrayError: path 'tmp' contains an array

On top of that it is not possible to do this without providing a temp_store to rechunker.rechunk, despite the fact that the documentation states that this is an optional argument.

Here is my script, note that this has an error when `rmtree(replay.path_out)` is not called each time (which is not what we want)
import os
import datetime
from shutil import rmtree

from UFS2ARCO import FV3Dataset
from UFS2ARCO.replay import cached_replay_path_1degree as replay_path

from timer import Timer

if __name__ == "__main__":

    walltime = Timer()
    localtime = Timer()

    walltime.start("Initializing job")

    replay = FV3Dataset(path_in=replay_path, config_filename="config-replay.yaml")
    kw = {"mode": "w"}
    year = 1994
    month = 1
    if os.path.isdir(replay.path_out):
        rmtree(replay.path_out)
    for day in range(1,3):
        for hour in [0, 6, 12, 18]:

            date = datetime.datetime(year=year, month=month, day=day, hour=hour)

            if os.path.isdir("../zstores/tmp"):
                rmtree("../zstores/tmp")
            # The code only runs if the following two lines are un-commented... but this is not at all what we want!
            #if os.path.isdir(replay.path_out):
            #    rmtree(replay.path_out)
            print(os.listdir("../zstores"))
            localtime.start(f"Pulling {str(date)}")
            ds = replay.open_dataset(date, fsspec_kwargs={"s3":{"anon": True}}, engine="h5netcdf")
            replay.store_dataset(ds, **kw)
            kw = {"mode": "a", "append_dim":"time"}
            localtime.stop()

So TL; DR I might just roll with the default chunking unless it becomes a problem because it seems to be not too bad, and avoids these issues.

And also passing mode="w" to temp_options to overwrite the temporary store does not work for some reason

Note also this requires this branch


coprocessor:  switch remaing code of tidb_query to using BufferReader

What have you changed?
This PR switch all the remaining encoding/decoding code of tidb_query to use BufferReader and BufferWriter.  Parts of issue https://github.com/tikv/tikv/issues/4040
What is the type of the changes?
Engineering
How is the PR tested?
unit test
Does this PR affect documentation (docs) or should it be mentioned in the release notes?
no
Does this PR affect tidb-ansible?
no.
Refer to a related PR or issue link (optional)
https://github.com/tikv/tikv/issues/4040

@iosmanthus @mapleFU @lonng PTAL

/run-all-tests

/auto-merge

/merge


Streamline GitHub labels

Current Behavior
We went a bit overboard with putting emojis in GitHub labels. The label names themselves also differ from what we use in https://github.com/DependencyTrack/dependency-track/labels.
Proposed Behavior
Streamline labels such that they align with what we use in https://github.com/DependencyTrack/dependency-track
Keep in mind that label names must be adjusted here as well:

https://github.com/DependencyTrack/hyades/blob/caab09c9abfc46575a0206f29efd184ee8e63d66/.github/release.yml#L2-L14
https://github.com/DependencyTrack/hyades/blob/caab09c9abfc46575a0206f29efd184ee8e63d66/.github/workflows/update-config-docs.yml#L63-L73
https://github.com/DependencyTrack/hyades-apiserver/blob/59f3d70c388d7809e102d438cd6fa4515530e199/.github/release.yml#L2-L14

Checklist

[X] I have read and understand the contributing guidelines
[X] I have checked the existing issues for whether this enhancement was already requested

Should also look at updating the default labels for new repositories here: https://github.com/organizations/DependencyTrack/settings/repository-defaults

Assigning to me since this needs admin privileges.

Added a few common labels at the organization level:

Labels are now managed centrally with Terraform for all repositories: https://github.com/DependencyTrack/admin/tree/main/terraform/github
Note, the admin repo is internal to the GitHub organization.


Use file-module to touch files, not shell

Use file-module to touch files, not shell. This is to avoid usage of warning during the provision.
[WARNING]: Consider using file module with state=touch rather than running touch

@rpsu - Please file issues against the upstream roles (e.g. geerlingguy.mysql in this case). Thanks!


Prohibit overwriting of WP global variables

To be merged once upstream has been merged back. Upstream PR https://github.com/WordPress-Coding-Standards/WordPress-Coding-Standards/pull/736 (already merged) contains a number of bug fixes which make this acceptable for use. Before that it was throwing too many false positives. See https://github.com/WordPress-Coding-Standards/WordPress-Coding-Standards/issues/300
Oh and before anyone asks: the $content_width variable is already excepted from the upstream sniff.

I think I found a valid case for overriding the global variable.
global $post;
$post = get_post( '10' );
setup_postdata( $post );
wp_reset_postdata();

@justintadlock Does this make sense? Are there any other cases where overwriting a WP global variable would make sense?

@grappler I'm not so sure about that one. As far as I know, you should be using a custom query in that case, i.e. new WP_Query(), rather than overwriting the global.

And even if overwriting that global $post is valid in that case, a theme should not be doing it... only a plugin.

I found the same code example in Twenty Seventeen. https://github.com/WordPress/WordPress/blob/master/wp-content/themes/twentyseventeen/inc/template-tags.php#L142


macdylibbundler: deleted duplicate dylibbundler

I deleted the dylibbundler but left an alias to the macdylibbundler package.
Motivation for this change
dylibbundler package is just an outdated version of macdylibbundler (see #119385)
fixes https://github.com/NixOS/nixpkgs/issues/119385
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@nomeata @alexfmpe The change we discussed. Feel free to make adjustments.

LGTM

Should we have added @alexfmpe as to macdylibbunder's maintainers?

Not necessary. See https://github.com/NixOS/nixpkgs/issues/119385#issuecomment-819893483. They said they haven't touched it in years.


Retrieve custom columns name in query

Dear all,
How can I get another value from columns that not exists in the table.
EX:
string sql = "SELECT round(avg(Value)) AS Value, max(Value) AS MaxValue, min(Value) AS MinValue FROM table"
database.QueryAsync(sql);
Please help!

Create a class to receive the results:
class QueryResults {
  public double Value { get; set; }
  public double MaxValue { get; set; }
  public double MinValue { get; set; }
}

and then query with it:
await database.QueryAsync<QueryResults> (sql);


"/" in title or artist will cause inaccessible folder and lrc file in explorer

Steps to reproduce

search lyric for a music with "/" in its title, such as "遺サレタ場所 / 斜光"
open lyrics will create a folder named "岡部啓一(MONACA) - 遺サレタ場所 " and " 斜光.lrc" as its content
windows explorer can't delete, rename, move or copy the created folder.
open lyrics can still load the .lrc file when foobar2000 plays the music.
foobar2000 console will log
WARN-OpenLyrics: Failed to open lyric file directory: 2
when trying to open file location

Expected behavior

Normally, open lyrics should create a single file named like "岡部啓一(MONACA) - 遺サレタ場所 / 斜光.lrc", and replace the "/" with other valid character, right?
Not only "/", all invalid characters for file system should be replaced.
Versions

foobar2000 version:v2.0 x64
OpenLyrics version:1.6

Debug logs

I guess this obvious bug don't need any more debug logs other than the one I said in steps to reproduce.
Additional information

The fact that it creates a folder is expected behaviour using the current default settings (see #244). However since you are not at all the first person to run into this, I've made a note to change the default filename format to include replacing / with some other valid character like dash or underscore. That'll probably be in the next release.
The fact that the created directory is difficult to delete is a known bug that is fixed for the next release. Take a look at #273 for discussion and a workaround.


fix: sessionId in first open and session start events

Description

fix the sessionId in first open and session start event
optimize session initialization

General Checklist

[x] Added new tests to cover change, if needed
[x] Build succeeds using Swift Package Manager
[x] All unit tests pass
[ ] Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
[ ] Documentation update for the change if required
[x] PR title conforms to conventional commit style
[ ] If breaking change, documentation/changelog update with migration instructions

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.71%. Comparing base (63e1df1) to head (5fec479).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #54      +/-   ##
==========================================
+ Coverage   95.66%   95.71%   +0.04%
==========================================
  Files          35       35
  Lines        1986     1985       -1
==========================================
  Hits         1900     1900
+ Misses         86       85       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Fixed the issue #46


Error while building pip package on Fedora

I am trying to compile the compute engine on Fedora using bazelisk following the tutorial at https://docs.larq.dev/compute-engine/build/converter/ , but I keep getting the same error:
ERROR: /home/crulis/.cache/bazel/_bazel_crulis/62e1e4d39f6188ec1a55a421a458a28d/external/pypi_lce_tensorflow/BUILD.bazel:22:11: no such package '@pypi_lce_tensorflow_io_gcs_filesystem//': The repository '@pypi_lce_tensorflow_io_gcs_filesystem' could not be resolved: Repository '@pypi_lce_tensorflow_io_gcs_filesystem' is not defined and referenced by '@pypi_lce_tensorflow//:pkg'
ERROR: Analysis of target '//:build_pip_pkg' failed; build aborted:

I tried the following commands:
bazelisk build :build_pip_pkg
bazelisk build :build_pip_pkg -c opt --config=release_cpu_linux --copt=-fvisibility=hidden --verbose_failures

I also tried to install different versions of tensorflow: 2.13.1, 2.16.1 and different python versions: 3.9, 3.10, 3.11, each time creating a new virtual environment and running a full clean using bazelisk clean --expunge. I am also regularly deleting the bazel cache directory just in case.
What am I doing wrong?
Bazel information
bazel 6.5.0
Git information
lce: HEAD detached at v0.16.0
third_party/tensorflow: HEAD detached at v2.16.1 (I also tried 2.13.1)
System information:
System:
Kernel: 6.9.9-200.fc40.x86_64 arch: x86_64 bits: 64 compiler: gcc v: 2.41-37.fc40
clocksource: tsc
Desktop: Cinnamon v: 6.2.4 tk: GTK v: 3.24.43 wm: Muffin vt: 2 dm: 1: GDM v: 46.2 2: LightDM
v: 1.32.0 note: stopped Distro: Fedora Linux 40 (Cinnamon)
Machine:
Type: Laptop System: Dell product: Latitude 5410 v: N/A serial:  Chassis:
type: 10 serial:
Mobo: Dell model: 06KF2W v: A00 serial:  part-nu: 09A0
uuid:  UEFI: Dell v: 1.28.0 date: 04/08/2024
CPU:
Info: quad core model: Intel Core i5-10310U bits: 64 type: MT MCP smt: enabled
arch: Comet/Whiskey Lake note: check rev: C cache: L1: 256 KiB L2: 1024 KiB L3: 6 MiB
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Info:
Compilers: clang: 18.1.6 gcc: 14.1.1 Client: Unknown Client: cinnamon (deleted) inxi: 3.3.34

Hi @BenCrulis, thank you for reporting this.
It seems that there's an error when bazel is resolving pip package dependencies.
Bazel says that the tensorflow package depends on tensorflow-io-gcs-filesystem but this latter package is not in the requirements.txt file: https://github.com/larq/compute-engine/blob/main/larq_compute_engine/requirements.txt
Could you try if re-generating requirements.txt solves it?
You can do that by running:
pip-compile --allow-unsafe --no-emit-index-url --strip-extras larq_compute_engine/requirements.in

Thanks @BenCrulis, glad to hear it worked. We'll update the documentation on the website.


PHP7 compatibility

It seems PHP7 doesn't work yet?

Thats correct. I no longer maintain the project. I have stopped using PHP many years ago. Should there be interest for anyone to make this package work with PHP 7, I will happily make you a contributor.


𝙊𝙥𝙩𝙞𝙤𝙣 𝙤𝙛 𝙘𝙝𝙤𝙤𝙨𝙞𝙣𝙜 𝙢𝙤𝙧𝙚 𝙩𝙝𝙖𝙣 𝙤𝙣𝙚 𝙥𝙝𝙤𝙩𝙤 𝙖𝙩 𝙖 𝙩𝙞𝙢𝙚

𝘐 𝘢𝘮 𝘳𝘦𝘢𝘭𝘭𝘺 𝘪𝘮𝘱𝘳𝘦𝘴𝘴𝘦𝘥 𝘸𝘪𝘵𝘩 𝘵𝘩𝘦 𝘢𝘱𝘱. 𝘖𝘯𝘦 𝘰𝘧 𝘮𝘺 𝘵𝘸𝘰 𝘪𝘴𝘴𝘶𝘦𝘴 𝘸𝘪𝘵𝘩 𝘪𝘵 𝘪𝘴 𝘐 𝘢𝘮 𝘶𝘯𝘢𝘣𝘭𝘦 𝘵𝘰 𝘱𝘪𝘤𝘬 𝘮𝘰𝘳𝘦 𝘵𝘩𝘢𝘯 𝘰𝘯𝘦 𝘱𝘩𝘰𝘵𝘰 𝘢𝘵 𝘢 𝘵𝘪𝘮𝘦. 𝘛𝘩𝘪𝘴 𝘱𝘳𝘰𝘭𝘰𝘯𝘨𝘴 𝘵𝘩𝘦 𝘢𝘮𝘰𝘶𝘯𝘵 𝘰𝘧 𝘵𝘪𝘮𝘦 𝘭𝘰𝘢𝘥𝘪𝘯𝘨 𝘢𝘭𝘭 𝘵𝘩𝘦 𝘱𝘩𝘰𝘵𝘰𝘴. 𝘐𝘴 𝘪𝘵 𝘱𝘰𝘴𝘴𝘪𝘣𝘭𝘦 𝘵𝘰 𝘭𝘰𝘢𝘥 𝘮𝘰𝘳𝘦 𝘵𝘩𝘢𝘯 𝘢 𝘴𝘪𝘯𝘨𝘭𝘦 𝘱𝘩𝘰𝘵𝘰 𝘢𝘵 𝘢 𝘵𝘪𝘮𝘦?

Was just about to open an issue for that.
This would be a really great feature but I'm not sure if this isn't an OS issue/limitation, as I've seen this in other apps to.
Device: Google Pixel 3a, Android 11
App: from play store, version 2.0.0

Was just about to open an issue for that.
This would be a really great feature but I'm not sure if this isn't an OS issue/limitation, as I've seen this in other apps to.
Device: Google Pixel 3a, Android 11
App: from play store, version 2.0.0

Updated in the latest release.


Remove mappings, type and provider from the ServiceBinding resource

The existing mappings on the ServiceBinding resource were introduced as
a way to make it easier for a user to enrich an existing Secret into a
form appropriate for binding to an application workload. This approch
had a few issues that can be better addressed by other resources that
interoperate with the ServiceBiding resource. The issues include:

the ServiceBinding controller needs to be able to read and write
Secrets
Go templates were used to compose new values, which worked for basic
templating, but were limited in their capabilities
the capabilities of the Go templates are an implemenation detail of
how the controller is built and could change over time independent of
the speced behavior.

The behavior applied by mappings can be reintroduced as dedicated
resources that can themselves expose a Secret as a ProvisionedService,
which can be consumed by a ServiceBinding. This change further separates
the concerns of provisioning a service from binding a service.
The .spec.type and .spec.provider fields are also removed, as
they are syntactic sugar on top of mappings.
Refs #145
Resolves #155
Signed-off-by: Scott Andrews<EMAIL_ADDRESS>

Added a clarification to resolve #155

Based on the call today, we want to reevaluate removing the .spec.type and .spec.provider fields. The thought is that we can create additional volumes/volumemounts for these specific files when needed. Before taking this approach, we need to ensure Kubernetes volume mounts can be safely nested, and what happens when there is a collision (e.g. the type exists, but is being overridden).

I tried the following configuration but seems to be not working.  This is the error message in the pod:
OCI runtime create failed: container_linux.go:367: starting container process caused:
process_linux.go:495: container init caused: rootfs_linux.go:60:
mounting "/var/lib/kubelet/pods/4f575521-2efb-4a06-9ea0-1ba93dfe1927/volume-subpaths/type/busybox/1"
to rootfs at
"/var/lib/docker/overlay2/30f1880fedd229c6bda44de1eda54038751dc49f2456ff6a15821732f6b9eab5/merged/bindings/type"
caused: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)?
Check if the specified host path exists and is the expected type

---
apiVersion: v1
kind: Secret
metadata:
  name: secret1
type: Opaque
stringData:
  provider: "backingservice"
  username: "guest"
  password: "passwd"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm1
data:
  type: custom
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    environment: test
spec:
  selector:
    matchLabels:
      environment: test
  template:
    metadata:
      labels:
        environment: test
    spec:
      containers:
        - name: "busybox"
          image: "busybox:latest"
          command: ["sleep"]
          args: ["1h"]
          volumeMounts:
            - name: bindings
              mountPath: /bindings
            - name: type
              mountPath: /bindings/type
              subPath: type
      volumes:
        - name: bindings
          secret:
            secretName: secret1
        - name: type
          configMap:
            name: cm1
            items:
              - key: type
                path: type

I just read the docs:

Volumes can not mount onto other volumes or have hard links to other volumes.

I tried to use life cycle hooks after setting readOnly: false in the volumeMounts. But due to security reasons setting readOnly to false has no impact, it will be always readOnly: true.
...
          volumeMounts:
            - name: bindings
              mountPath: /bindings
              readOnly: false
            - name: type
              mountPath: /extra-bindings/type
              subPath: type
          lifecycle:
            postStart:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - ln -sf /extra-bindings/type /bindings/type
...

This is the error message:
PostStartHookError: command '/bin/sh -c ln -sf /extra-bindings/type /bindings/type'
exited with 1: ln: /bindings/type: Read-only file system

@baijum thanks for the research

@scothis A projected volume maps several existing volume sources into the same directory.
Here is a modified example:
---
apiVersion: v1
kind: Secret
metadata:
  name: secret1
type: Opaque
stringData:
  type: "to-be-changed"
  provider: "backingservice"
  username: "guest"
  password: "passwd"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm1
data:
  type: custom
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    environment: test
spec:
  selector:
    matchLabels:
      environment: test
  template:
    metadata:
      labels:
        environment: test
    spec:
      containers:
        - name: "busybox"
          image: "busybox:latest"
          command: ["sleep"]
          args: ["1h"]
          volumeMounts:
            - name: all-in-one
              mountPath: /bindings
      volumes:
        - name: all-in-one
          projected:
            sources:
              - secret:
                  name: secret1
              - configMap:
                  name: cm1
                  items:
                    - key: type
                      path: type

As you can see the original value for type in the Secret resource is getting overridden by the value in the ConfigMap.
I confirmed it:
$ kubectl exec -it app-774f8fcd48-mp9g9 -- cat /bindings/type
custom

I tried to change some values in the Secret resource, and that is getting reflected in the file system immediately.
So, we are good go with merging this PR :-)

we are good to go with merging this PR

Based on the above information, we can retain .spec.type and .spec.provider as optional fields.  I think we can add an additional note about these special fields in the spec. Something like this:
Note: Users can employ the `.spec.type` and `.spec.provider` fields to override the original
values in the Provisioned Service Secret resource. Mainly, it would be helpful if the
Provisioned Service Secret resource doesn't provide those values.

(Feel free to rephrase)

The Application Projection section 2nd paragraph also needs some re-write. This is how it currently reads:

The Secret data MUST contain a type entry with a value that identifies the abstract classification of the binding.  The Secret type (.type verses .data.type) MUST reflect this value as service.binding/{type}, replacing {type} with the Secret data type.  It is RECOMMENDED that the Secret data also contain a provider entry with a value that identifies the provider of the binding.  The Secret data MAY contain any other entry.

@baijum updated to restore the type and provider fields on the ServiceBindingSpec. Removed references to "secrets" in the application projection as a Secret is more of an implementation detail at this point.

Remove mappings, type and provider from the ServiceBinding resource

@scothis Please update the title of this PR: type and provider is not removed.


check for errors

Codecov Report

Merging #321 (8deb44c) into main (b114fc2) will not change coverage.
The diff coverage is n/a.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@           Coverage Diff           @@
##             main     #321   +/-   ##
=======================================
  Coverage   77.83%   77.83%
=======================================
  Files         179      179
  Lines        9034     9034
=======================================
  Hits         7032     7032
  Misses       1719     1719
  Partials      283      283

Flag
Coverage Δ

unittests
77.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Replace internal/backend QuickPrint subsystem with displacing automation

Is your feature request related to a problem? Please describe.
Currently using AZ-CLI to, for example, delete a resource group with even just a simple application gateway in it can take up to 584.816 seconds.  This solution would speed it up to sub-second response time.
Describe the solution you'd like
Replacing the Quickprint System with an automation would solve this:  When provisioning or deprovisioning resources, instead of az-cli or the portal generating a PostScript formatted image and sending it to a dot matrix printer in one of the data centers and then requiring a person to walk across the data center, read the physical ticket, and then go wire up a server to those specifications after running cat5 cables (which, let's be honest, that's hard work and sometimes requires a toner, I've had to do it) and then sending the error code back to a relay attached to a multiplexer feeding the user's frontend cli utilities -- instead of the magnificently complex (it's best quality) Quickprint Deluxe 9000 system, I propose completely deleting your entire codebase and redesigning this entire provider ecosystem from the ground up using entirely new people with entirely new paradigms.  If this solution is considered it may even end up with Azure still existing in 5-10 years after the nation's CFO's start doing transition and operational overhead cost comparisons with other provider transitions in a way that doesn't rely exclusively on account management's relationship building and overengineered security theatre.
Describe alternatives you've considered
Swearing, begging, even filing polite tickets that get closed, additional training, and even resorting to debugging the portal with the browser's debugger features to observe the silent errors.
Additional context
There is humour intended here, but read between the lines:  This ecosystem is not sustainable and you should already know it.  If you don't, then Azure may have already failed.  (How could you not know?)

@jiasli  please help with this.

Thanks for your feedback.
If your issue is specifically regarding performance of resource group deletion in Azure CLI, we have an option called --no-wait to make the command asynchronous and unblock you. For example, when you run ...
az group delete -n xxx --no-wait

the command won't block and will return immediately.
That said, Azure CLI is just a client tool calling into Azure services. So performance of commands like resource group deletion varies depending on Azure services in the resource group.
So to help with that, if your issue is regarding performance of Azure services, we suggest that you post this to feedback.azure.com so other users can also vote on importance of the same. We do take this feedback seriously and have addressed performance feedback in that forum by making optimizations to the platform as shown here


Adding Vyper Support

Feature Request - Vyper Ethereum Programming Language
I've  been a long time user of spyder-ide and use it almost everyday. I don't know if it's possible to include Vyper (which works almost identically to python) to the ide so that it could  compile the code and highlight syntactic errors and use it for testing with the spyder debgging tools.
Vyper is used to code smart contracts in ethereum which is basically kind of a class in python however it can be used to work with programmable money.
I've asked myself  the devs there to include vy in conda, which they did recently, hoping that eventually it could help with my case to try and add it to the spyder ide.
I've include below some useful links on the topic below:

conda package
vy conda request
docs
vy on vs
vy resources

Hey @kaleb-keny, thanks for reporting. This is a very interesting idea, but unfortunately we're too tied to Python and its ecosystem to go in this direction.
Thanks for the suggestion though.

no worries, was worth a shot, maybe some day. thank you.


Variables not being found?

ATM i have a resource file with the following
  This sheet is globally accessible in all application sheets
  DO NOT add sheets with styles, this should ONLY be used for mixins and variables
 */
@import "~bootstrap/scss/variables";
@import "variables";
@import "~bootstrap/scss/utilities";
@import "~bootstrap/scss/mixins";

and in variables there is...
$link-color:            theme-color("primary") !default;
$link-decoration:       none !default;
$link-hover-color:      darken($link-color, 15%) !default;

yet i get this error...
./node_modules/css-loader!./node_modules/postcss-loader/lib?{"plugins":[null]}!./node_modules/sass-loader/lib/loader.js!./node_modules/sass-resources-loader/lib/loader.js?{"resources":"app/styles/_resources.scss"}!./app/scheduler/views/shared/assets/styles/EditShiftForm.scss
Module build failed:
$link-hover-color:      darken($link-color, 15%) !default;
                       ^
      Argument `$color` of `darken($color, $amount)` must be a color

Backtrace:
	node_modules/bootstrap/scss/_variables.scss:168, in function `darken`
	node_modules/bootstrap/scss/_variables.scss:168
      in /Users/clarktomlinson/Sites/When-I-Work/wheniwork-js/node_modules/bootstrap/scss/_variables.scss (line 168, column 25)
 @ ./app/scheduler/views/shared/assets/styles/EditShiftForm.scss 4:14-301 18:2-22:4 19:20-307
 @ ./app/scheduler/views/shared/containers/EditShiftForm.js
 @ ./app/scheduler/views/shared/containers/EditShiftModal.js
 @ ./app/scheduler/views/week/components/ScheduleCell.js
 @ ./app/scheduler/views/week/components/ScheduleRow.js
 @ ./app/scheduler/views/week/index.js
 @ ./app/scheduler/containers/ViewBuilder.js
 @ ./app/scheduler/containers/Builder.js
 @ ./app/scheduler/index.js
 @ ./app/routes.js
 @ ./app/root.js
 @ ./app/main.js
 @ multi ./app/shared/vendor ./app/main
msgErrors @ client?bf6e:157
onmessage @ socket.js?e5d0:38
EventTarget.dispatchEvent @ eventtarget.js?d62b:51
(anonymous) @ main.js?8e93:274
SockJS._transportMessage @ main.js?8e93:272
EventEmitter.emit @ emitter.js?a8c1:50
WebSocketTransport.ws.onmessage @ websocket.js?a354:35

with the following webpack config
        use: [
          'style-loader',
          'css-loader',
          {
            loader: 'postcss-loader',
            options: { plugins: [require('autoprefixer')({ browsers: ['last 2 versions'] })] },
          },
          'sass-loader',
          {
            loader: 'sass-resources-loader',
            options: {
              resources: 'app/styles/_resources.scss',
            },
          },
        ],```

@alexfedoseev Sorry to ping you but any ideas?

I'm running into the exact same problem.

i actually solved mine, without needing this lib by doing this
        test: /\.(scss|css)$/,
        use: [
          'style-loader',
          'css-loader',
          {
            loader: 'postcss-loader',
            options: { plugins: [require('autoprefixer')({ browsers: ['last 2 versions'] })] },
          },
          {
            loader: 'sass-loader',
            options: {
              data: '@import "styles/resources";',
              includePaths: [path.join(cwd, 'app')],
            },
          },
        ],
      },```

but with this lib the issue was in my resources file i forgot to add this import at the top which should resolve it

@import "~bootstrap/scss/functions";

Awesome, thank you.  After looking at your solution, I realized i was getting the error because in my .bootstraprc file I was loading my resources file in the "preBootstrapCustomizations" field, instead of the post "bootstrapCustomizations" field.

that'll do it too :)


Rename CC

Is it possible to rename the cc binary? If I install it in my PATH, it over shadows the soft link to my C compiler (which CMake looks for).

Possibly, but what is wrong with installing the 'bart' master-command, i.e. 'bart cc ...'

Could we make that the default for installation? The default make install installs all commands.

The default Makefile doesn't install all commands, or am I missing something? Did you use cmake instead? This was contributed by @hjmjohnson. Somebody would have to provide a patch with the necessary changes (also automatic build testing for cmake would be nice)

Sorry, yes this was a CMake build (I develop on a Mac). I'll try to sort out changing the default CMake install. Good excuse to learn how it works!


Wireless Transciever Class

Write a class to send and receive messages. Set up server and client status.

Can you adapt your code (the ino files) to a "WirelessTransciever" class? Also open a pull request when you're done.


registrySecret before buildCommand

If Dockerfile has a FROM to a private registry, it needs the credentials in .dockercfg
I need to complete testing, but creating PR to review.

Testing was successful


Add missing tests, fix bufconn client target

I've added some basic tests for input params and pb.* to local type converters. The scaffolding for client target is in place but for some reason the Dapr types are not reconciled at runtime. Once resolved, we will need to add additional tests (line 19 in  client/client_test.go)

tests added in the latest PR


⚠ Bofrost.de has degraded performance

In cdb948d, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2016 ms

Resolved: Bofrost.de performance has improved in 33b0e5e after 6 minutes.


[BUG] The attempt to rmdir the dist folder prevents cache the dist on an NFS share

Environment
Self-Hosted (Docker)
System
K8S
Version
3.1.1
Describe the problem
Dashy builds in /app/dist and unlike user-data that is customizable via ENV, /app/dist seems to be hardcoded (please tell me if this is not the case).
I was thrilled to read:

and it looks like it is not very close to be true for most scenarii... but not yet for all :)

Dashy itself does not need much resources, and since my K8S limits are low, I alos run into a:

which should be read as "it is ok, it just takes longer than expected".
Since the build takes a while, to avoid a new build, I considered mounting a volume for /app/dist.
This however fails when using a NFS share since, during the build, dashy attempts to rmdir /app/dist and the root of a NFS share cannot be removed.
I understand that wiping the dist folder is easier when an upgrade is required.
I hope it triggers only on upgrades btw :)
This NFS issue is not typical to dashy (people often run into it when using NFS shares, k8s and databases for instance). The Solution is known and simple: let the user define the /app/dist location with an ENV. This would allow telling dashy to work in say /app/dist/subdist (subdist is an example, it could be foobar, the key point is that it is a folder under the /app/dist mounting point).
The NFS share is then still mounted as /app/dist but using a sub folder, allows to let the build script to control the subdist folder (and remove it). It is not possible and won't be possible for the build script (or anything else for that matter) to rm/chmod/chown the root of the NFS share, in this example /app/dist.
Using Dashy V-3.1.1. Update Check Complete
✅ Dashy is Up-to-Date
-  Building for production...
 WARN  A new version of sass-loader is available. Please upgrade for best experience.
 ERROR  Error: EBUSY: resource busy or locked, rmdir '/app/dist'
Error: EBUSY: resource busy or locked, rmdir '/app/dist'
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "build" exited with 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.

Additional info
No response
Please tick the boxes

[X] You have explained the issue clearly, and included all relevant info
[X] You are using a supported version of Dashy
[X] You've checked that this issue hasn't already been raised
[X] You've checked the docs and troubleshooting guide
[X] You agree to the code of conduct

Hi
This has been discussed before: https://github.com/Lissy93/dashy/issues/1570
Dashy's config loading is fully dynamic, however smaller things like adding files (icons), changing default view, auth.... still require a rebuild, and as lissy was a bit busy, we tried without the rebuild on the container reboot, however this lead to quite some issues and we hotfix/reverted this back to the yarn build-and-start command.
I will close this issue as a duplicate.


translatesfx: translate metricsToInclude filters to expr

SA's metricsToInclude overrides any matching metricsToExclude. This change
translates SmartAgent metricsToInclude filters and appends them to filter
processor expr expressions.
For example, the following SA filters:
metricsToExclude:
  - metricNames:
      - node_filesystem_*

metricsToInclude:
  - metricNames:
      - node_filesystem_readonly

should translate to:
filter:
  metrics:
    exclude:
      match_type: expr
      expressions:
      - MetricName matches "^node_filesystem_.*$" and
        (not (MetricName matches "^node_filesystem_readonly$"))

Not necessarily affecting these changes, but do you have an idea for what the general expected functionality should be for monitorType? https://github.com/signalfx/signalfx-agent/blob/main/docs/config-schema.md#metricstoinclude

What do you think about emitting a warning that translating monitorType is unsupported?

I think in the short term that sounds good, but overall it may make sense to merge with referenced monitor's datapointsToExclude  or even create different processors and pipelines.  That may be overkill though.


[WIP] Support Encodable types natively

This is a follow-up on #1136 and is based on the current state of #1147. We should not merge this until #1147 was merged and Swift 4 nears completion.
What I have done here is to implement the intention behind #1136 with the new approach we have taken in #1147. Although the code is like I would expect it to work, the Swift compiler (currently) seems not to be able to work with this code. I'm still getting the same error as in #1136, for example:
case let .request(.jsonEncodable(encodable, encoder)):
    preparedRequest.httpBody = encoder.encode(encodable) // ERROR: Cannot invoke 'encode' with an argument list of type '(Encodable)'
    cancellableToken.innerCancellable = self.sendRequest(target, request: preparedRequest, queue: queue, progress: progress, completion: networkCompletion)

If you know how we can work around this issue, then please let me know. I couldn't find a good alternative solution or quick fix for this.
Here's what is done in this PR:

[x] Add jsonEncodable and propertyListEncodable cases to RequestDataType
[x] Add compositeJsonEncodable and compositePropertyListEncodable, too
[x] Integrate the new cases into the request creation logic

The following are TODOs:

[ ] Fix the above mentioned compiler error (find alternative)
[ ] Update Changelog
[ ] Update Tests
[ ] Update Documentation

          4 Warnings

:warning:
PR is classed as Work in Progress

:warning:
Any changes to library code should be reflected in the Changelog. Please consider adding a note there.

:warning:
Consider adding supporting documentation to this change. Documentation can be found in the docs directory.

:warning:
The library files were changed, but the tests remained unmodified. Consider updating or adding to the tests to match the library changes.

SwiftLint found issues
Warnings

File
Line
Reason

MoyaProvider+Internal.swift
103
TODOs should be avoided (Add exception handling here).

MoyaProvider+Internal.swift
111
TODOs should be avoided (Add exception handling here).

MoyaProvider+Internal.swift
120
TODOs should be avoided (Add exception handling here).

MoyaProvider+Internal.swift
129
TODOs should be avoided (Add exception handling here).

MoyaProvider+Internal.swift
139
TODOs should be avoided (Add exception handling here).

MoyaProvider+Internal.swift
159
TODOs should be avoided (Add exception handling here).

  Generated by :no_entry_sign: Danger

Please look at my comment here. Because of that and because we may have too many PRs opened at the same time and we want to make it easier for our contributors to find the ones that matter the most right, I'm gonna close it for now. We will continue the discussion later on.


Aside applies the indentation of content

When aside is currently in an indent. e.g.
*   Here is a bulleted list that has an aside

    {% aside "info" %}
    Here are the contents
    {% endaside %}

The shortcode will translate it into markdown as so:
*   Here is a bulleted list that has an aside

<litdev-aside type="info">
<!-- htmlmin:ignore -->

    Here are the contents

<!-- htmlmin:ignore -->
</litdev-aside>

Thus throwing off the MD compiler. In this PR we get the leading whitespace in front of the contents, and prepend them to all of our HTML nodes:
// content looks like this:

"    Here are the contents"

The regex matches ^\s* which is the first 4 spaces and applies it to our HTML nodes
*   Here is a bulleted list that has an aside

    <litdev-aside type="info">
    <!-- htmlmin:ignore -->

    Here are the contents

    <!-- htmlmin:ignore -->
    </litdev-aside

This is how it looks now at 3 different indentation levels +0, +1, +2

🥇


Add ability to define hci device

Now to the proper merge request. I have an external USB bluetooth dongle which I would like to use instead of built-in bluetooth module.
This patch allows the user to define which HCI device is being used by RuuviTag discovery (though noble).
I'm happy to hear your thoughts and improvement suggestions.

Thanks for this pull request: you're right, NOBLE_HCI_DEVICE_ID must be changeable.
But I don't like using a run.sh as the docker CMD, because it is the process which is monitored by docker and I would prefer that it be node.
The ENV variable is used here: https://github.com/abandonware/noble/blob/master/lib/hci-socket/hci.js#L92
So we can change NOBLE_HCI_DEVICE_ID in the node code.
Create an options.json file is a good thing: it will be only used for Home Assistant special config (as env variable can be set when running standalone node) and it must be defined before the start (so outside the UI). But it must be init to default value 0 and should be optional:
schema: { "hci_device_id": "int?" }
As I have already worked on reading the config with this special file for Home Assistant, I will integrate the feature ;)
I'll close your pull request when it will be done.

It's done in release 0.2.5.
Open an issue if it doesn't work as expected.

Thank you, I appreciate your effort.


[7] feat: add blinking cursor to code racer

title: Issue #7
Discord Username: @belm0069
What type of PR is this? (select all that apply)

[x] 🍕 Feature
[ ] 🐛 Bug Fix
[ ] 🚧 Breaking Change
[ ] 🧑💻 Code Refactor
[ ] 📝 Documentation Update

Description
i added a blinking cursor to code racer and fix some overflow problems
QA Instructions, Screenshots, Recordings
https://github.com/webdevcody/code-racer/assets/51862296/4c325f83-18af-44cb-8e33-8e625a887541
Added/updated tests?

[ ] 👍 yes
[x] 🙅 no, because they aren't needed
[ ] 🙋 no, because I need help

Try reverting your package-lock.json after npm install instead of deleting it.

@KaiBelmo  could you please try what @sudo-adduser-jordan suggested?


Nice error messages on parse failure

When a cron expression cannot be parsed it results in  'Err' value: ()', helpful messages pointing to the actual syntax error would be great.

Tracked over at https://github.com/zslayton/cron/issues/11


Docs readme revamp

Categorization, adding background reading, adding more examples

@nan-wang good to merge now?

Ping @nan-wang


BFS in cpp

@Ayush0431  hey can i work on this issue?

Plz assign it to me.

Can you assign it to me I'm good In Data Structures!

@Ayush0431 i have created a PR please check

Hey! @Ayush0431 I want to contribute in your repository so please assign me this task.


On - Make

Ubuntu 16.04
Trying to install make gives me this have tried numerous different variations of ./configure
I assume this is an ssl issue but I see no definition given for the ssl version to use?
CXX      libquark_wallet_a-db.o
CXX      libquark_wallet_a-crypter.o
CXX      libquark_wallet_a-rpcdump.o
CXX      libquark_wallet_a-rpcwallet.o
CXX      libquark_wallet_a-wallet.o
CXX      libquark_wallet_a-wallet_ismine.o
CXX      libquark_wallet_a-walletdb.o
AR       libquark_wallet.a
/usr/bin/ar: u' modifier ignored since D' is the default (see U')   CXXLD    quarkd quarkd-bitcoind.o: In function boost::asio::ssl::detail::openssl_init_base::do_init::~do_init()':
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:64: undefined reference to CRYPTO_set_id_callback' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:65: undefined reference to CRYPTO_set_locking_callback'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:66: undefined reference to ERR_free_strings' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:72: undefined reference to EVP_cleanup'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:73: undefined reference to CRYPTO_cleanup_all_ex_data' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:76: undefined reference to ENGINE_cleanup'
quarkd-bitcoind.o: In function boost::asio::ssl::detail::openssl_init_base::do_init::~do_init()': /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:64: undefined reference to CRYPTO_set_id_callback'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:65: undefined reference to CRYPTO_set_locking_callback' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:66: undefined reference to ERR_free_strings'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:72: undefined reference to EVP_cleanup' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:73: undefined reference to CRYPTO_cleanup_all_ex_data'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:76: undefined reference to ENGINE_cleanup' quarkd-bitcoind.o: In function boost::asio::ssl::detail::openssl_init_base::do_init::do_init()':
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:39: undefined reference to SSL_library_init' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:40: undefined reference to SSL_load_error_strings'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:41: undefined reference to OPENSSL_add_all_algorithms_noconf' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:43: undefined reference to CRYPTO_num_locks'
/usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:46: undefined reference to CRYPTO_set_locking_callback' /usr/include/boost/asio/ssl/detail/impl/openssl_init.ipp:47: undefined reference to CRYPTO_set_id_callback'
libquark_server.a(libquark_server_a-init.o): In function AppInit2(boost::thread_group&)': /opt/quark/src/init.cpp:777: undefined reference to SSLeay_version'
libquark_server.a(libquark_server_a-rpcserver.o): In function boost::asio::ssl::context::context(boost::asio::ssl::context_base::method)': /usr/include/boost/asio/ssl/impl/context.ipp:120: undefined reference to SSLv23_method'
libquark_util.a(libquark_util_a-util.o): In function CInit::~CInit()': /opt/quark/src/util.cpp:150: undefined reference to RAND_cleanup'
/opt/quark/src/util.cpp:152: undefined reference to CRYPTO_set_locking_callback' /opt/quark/src/util.cpp:153: undefined reference to CRYPTO_num_locks'
libquark_util.a(libquark_util_a-util.o): In function CInit::CInit()': /opt/quark/src/util.cpp:127: undefined reference to CRYPTO_num_locks'
/opt/quark/src/util.cpp:128: undefined reference to CRYPTO_num_locks' /opt/quark/src/util.cpp:130: undefined reference to CRYPTO_set_locking_callback'
/opt/quark/src/util.cpp:137: undefined reference to OPENSSL_no_config' libquark_wallet.a(libquark_wallet_a-crypter.o): In function CCrypter::Encrypt(std::vector<unsigned char, secure_allocator > const&, std::vector<unsigned char, std::allocator >&)':
/opt/quark/src/crypter.cpp:65: undefined reference to EVP_CIPHER_CTX_init' /opt/quark/src/crypter.cpp:69: undefined reference to EVP_CIPHER_CTX_cleanup'
/opt/quark/src/crypter.cpp:69: undefined reference to EVP_CIPHER_CTX_cleanup' libquark_wallet.a(libquark_wallet_a-crypter.o): In function CCrypter::Decrypt(std::vector<unsigned char, std::allocator > const&, std::vector<unsigned char, secure_allocator >&)':
/opt/quark/src/crypter.cpp:92: undefined reference to EVP_CIPHER_CTX_init' /opt/quark/src/crypter.cpp:96: undefined reference to EVP_CIPHER_CTX_cleanup'
/opt/quark/src/crypter.cpp:96: undefined reference to `EVP_CIPHER_CTX_cleanup'
collect2: error: ld returned 1 exit status
Makefile:2625: recipe for target 'quarkd' failed
make[2]: *** [quarkd] Error 1
make[2]: Leaving directory '/opt/quark/src'
Makefile:6396: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/opt/quark/src'
Makefile:574: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

I had the same probs on Ubuntu 16.04. The problem is that in 16.04 boost is a dependency to some packages and so you can't downgrade it. To get this to work, you need to compile it on 14.04. I personnally did this in a chroot but this could work with VMs too.

I tried inside a VM with 14.04 exactly the same error code above, a little more input as to what the requirements are would be helpful. Like versions of other programs required to compile would help.
This looks like an openssl issue but I can't figure out what version is required.
I'm going to cheat and use wine, unless anyone else has a solution.

i have a .deb of <IP_ADDRESS> compiled by myself if you're interested. You may not trust me however.

Hey buddy thanks but I just used wine and it works, it seems a little odd that a release dated may requires a version of Ubuntu which is soon be end of product life cycle to compile but TBH it's not just quarkcoin I've had similar issues with other coins. I just compiled an entire exchange and have managed to get it semi functional I was only using it as an engineering experiment to prove to myself I could get it up and working and I have. Thanks for your help tho. This could probably be closed but I'm going to leave it open as the developers should really find a way to make compiling easier on newer version of Ubuntu without having to install older dependencies, I know that SSL has changed a lot in the past few years but still.


Creating a new trad project with a required obs field puts it in weird limbo state

This has to be done via a non-staff account, I think. I used inatstafftestaccount

Start a new traditional project.
Add a place to it
Add an observation field and make it required
Save the project

The new project is really weird. See this example: https://www.inaturalist.org/projects/test-bfw-3
When logged in as the project creator:

I see no option for making a journal post:

when I go to the Members page, I'm not listed as a Manager:

I can edit the project but I can't delete it:

Someone else said they can't invite users to their invite-only project.

Thanks Jane. Doing some more testing, it seems like anytime I make a new traditional project and add a rule before creating the project, it causes these issues. If I make a new project without any rules, it's created fine and I can add rules later.


Dev optimizers notebook

New Notebook example. It is about the optimizers.
It has no number yet, before merging we should give it a number and change its name.

This is a very nice notebook! :tada:
It will also make sense to have something similar on the website :)

Very nice notebook, @raulconchello. Congratulations!
Just one comment, in the section for gradient-free optimizers, I can see newton-cg, and l-bfgs-b algorithms in the list as well. Be careful, since they require a jacobian for optimization, and therefore, are not gradient-free

@vishal-ph I have changed what you said.
However, before merging I think we should give it a number.

How about 13? :D
Also, is the notebook already featured in /tests/test_notebooks.py and in the docs?

@Q-lds
We can give it the 13, but I feel that the notebooks increase in difficulty and this one is not that technical.
What do you think, @vishal-ph ?

fair point, @raulconchello. Even though it was never a conscious plan to rank the notebooks in increasing order of complexity, it is possible that's how it looks at the moment.

@vishal-ph and @raulconchello, I think the number of the file is not that relevant. What is important is to organize the notebooks in the docs.
As we already have a lot of them, it probably makes more sense to divide them thematically in the docs and keep a simple increasing counter in the folder

sure, then 13 is fine for the moment


Post History View Liked Button Not Properly Colored

If you like a song, it should show as previously liked in Post History View

Does this only happen with other people's profiles. I can't reproduce otherwise.

No longer an issue.

What resolved this?

Liking and unliking not handled in post history view

Fixed in web commit https://github.com/cuappdev/icefishing-web/commit/1d3a835812a9a83fe4e09db1598c090c7264fbb0


Dubious !

Dubious! I'll start my explanation with this deliberately provocative adjective to make progress on the subject and find my mistake.
On the web you can see a craze for finetuning (with unsloth or other but I haven't tested it yet) and you can find many blog articles that often use the same codes (based on Dan's) with a few variations depending on the SFTTrainer() and dataset parameters...
I wanted to finetune llama 3.1 8B instruct 4bnb (and standard version, but the results are identical) on an ancient book I'm working on, "De Bello Gallico", in the hope that my finetuned model could provide a relatively accurate answer to the question "What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?".
I chose this question because GPT4o, Llama 3.0, 3.1, 8B 70B on groq, Gemini deal with approximation and big errors. Only Claude 3.5 sonnet answers correctly enough, at least that's the best answer.
My idea was therefore to specialize llama 3.1 (or other) on this corpus text (although RAG may be better direct suited to my situation). To do this, I created my own dataset which injects the text all corpus text in English, French and Latin, and a series of more 20.000 instruction-output questions and answers in English and French, split into a train dataset and an evaluation dataset (just QA and IO not corpus text for eval dataset). The results of train and eval by SFTTrainer() on 2 epochs :
-- Training completed stats :
TrainOutput(global_step=300, training_loss=0.37135863726337753, metrics={'train_runtime': 3545.2205, 'train_samples_per_second': 1.36, 'train_steps_per_second': 0.085, 'total_flos': 4.449468983153787e+17, 'train_loss': 0.37135863726337753, 'epoch': 1.9900497512437811})
-- Evaluation validation dataset results :
{'eval_loss': 0.22035124897956848, 'eval_runtime': 93.5197, 'eval_samples_per_second': 5.614, 'eval_steps_per_second': 0.706, 'epoch': 1.9900497512437811}
I initiated a fine-tuning run for 2 epochs on a Colab notebook using an A100 GPU and obtained a model that I saved in the Hugging Face merged 16-bit format.
I was able to perform inference using various methods: transformers.pipeline(), transformers model.generate() , and differents parameters like : do_sample=False (temperature=None top_p=None top_k=None), or do_sample=True (with temperature 0.7 top_p=0.9 top_k=50 in this case). I also converted the Hugging Face merged 16-bit model to the GGUF format and ran the same inference query on Ollama.
However, the responses to the prompt from these various inference (runs of my fine-tuned 3.1 8B Instruct 4 bnb model) were incorrect, very bad or unsatisfactory.
I am seeking to understand the root of the issue and have several, perhaps provocative, questions :

Are my fine-tuning parameters and dataset appropriate (see details below)? While they could be improved, are two epochs insufficient?

Does my dataset contain enough question-answer pairs or instruction-output to fine-tune this corpus text ? It's not easy to generate more QA IO...

I am disappointed with the responses to a specific question I used as an example, but I would like to benchmark all questions and instructions in my dataset against the entire text. Do you know of a convenient tool to do this relatively easily ? I have started asking Claude to evaluate the responses of my fine-tuned model based on the English and French text corpus provided in the prompt. While the evaluation seems fairly good, benchmarking manually without a script is impractical... but I could write a script.

All articles on the web provide examples of simple inference questions in the code, but I have yet to see a complete benchmark of the results. Are their results ultimately good ? I believe it would be beneficial to conduct a comprehensive benchmark of the results afterwards.

Is QLora/Lora 4-bit fine-tuning too simplistic for this kind of work ? RAG ? But in that case, what are the practical applications of these quickly quantized fine-tuned models ?

Thanks for advance, David.
max_seq_length = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    load_in_4bit=True,
    dtype=None,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "up_proj", "down_proj", "o_proj", "gate_proj"],
    use_rslora=True,
    use_gradient_checkpointing="unsloth"
)

---- train_dataset: Dataset({
    features: ['text'],
    num_rows: 20837
})

---- validation_dataset: Dataset({
    features: ['text'],
    num_rows: 4907
})

max_seq_length=2048
Min Number tokens in dataset train+eval : 192
Max Number tokens in dataset train+eval : 1373

Example corpus dataset :
{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are an expert historian specializing in the ancient world and antiquity, with a focus on the period from 800 BC to 400 AD. <|eot_id|><|start_header_id|>user<|end_header_id|>\n\nQuote me with the chapter section numbers between open and closed parenthesis characters (1) (2) to (n) : La Guerre des Gaules of Caius Julius Caesar's \"Commentarii de Bello Gallico\", French corpus text, Book 6, Chapter 21. A word of clarification : This text in Book 6 concerns events and military operations that took place in the year 53 BC.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nThe Germans differ much from these usages, for they have neither Druids to preside over sacred offices, nor do they pay great regard to sacrifices. They rank in the number of the gods those alone whom they behold, and by whose instrumentality they are obviously benefited, namely, the sun, fire, and the moon; they have not heard of the other deities even by report. Their whole life is occupied in hunting and in the pursuits of the military art; from childhood they devote themselves to fatigue and hardships. Those who have remained chaste for the longest time, receive the greatest commendation among their people; they think that by this the growth is promoted, by this the physical powers are increased and the sinews are strengthened. And to have had knowledge of a woman before the twentieth year they reckon among the most disgraceful acts; of which matter there is no concealment, because they bathe promiscuously in the rivers and [only] use skins or small cloaks of deer's hides, a large portion of the body being in consequence naked.<|eot_id|>"}

Example for QA and IO ::
{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are an expert historian specializing in the ancient world and antiquity, with a focus on the period from 800 BC to 400 AD. A word of clarification : This instruction about La Guerre des Gaules of Caius Julius Caesar's \"Commentarii de Bello Gallico\", French corpus text, Book 6, Chapter 21 concerns events and military operations that took place in the year 53 BC.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nDéterminez les croyances religieuses des Germains<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLes Germains adorent le Soleil, Vulcain et la Lune, et ne connaissent pas les autres dieux.<|eot_id|>"}

{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are an expert historian specializing in the ancient world and antiquity, with a focus on the period from 800 BC to 400 AD. A word of clarification : This instruction about La Guerre des Gaules of Caius Julius Caesar's \"Commentarii de Bello Gallico\", French corpus text, Book 6, Chapter 21 concerns events and military operations that took place in the year 53 BC.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nDécrivez les activités quotidiennes des Germains<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLa chasse et les exercices militaires<|eot_id|>"}

trainer=SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset = train_dataset,
    eval_dataset = validation_dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=True,
    args=TrainingArguments(
        learning_rate=3e-4,
        lr_scheduler_type="linear",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        num_train_epochs=2,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        warmup_steps=10,
        save_strategy="steps",
        save_steps=250,
        save_total_limit=2,
        output_dir=run_name_trainer,
        report_to="wandb",
        run_name=run_name_wandb,
        seed=0,
    ),
)

model.save_pretrained_merged("/content/drive/MyDrive/GdG_Model_merged_16b_fHF", tokenizer, save_method = "merged_16bit",)

on unsloth 2024.8.

Well, a couple of things come to mind. First, your rank is pretty low, it's set to 16. So it doesn't have much room to learn new information, and will instead pick up on superficial details of the training data. Lower rank LORAs are more for getting your model to follow a certain tone or style of writing, think something like RP. To learn new knowledge via a LORA, you'd need a larger rank.
The other issue, is your model of choice, that is Llama 3.1 8b. While it's a fantastic small model, it's only 8 billion parameters. So there is a pretty hard cap to it's knowledge and general intelligence, especially when loading it in 4bit. Maybe consider something larger like Mistral 12b nemo, or Gemma 27b

@dromeuf Actually wait did you use tokenizer.apply_chat_template? or our conversational notebooks - apologies but just after you posted, I updated all Llama 3.1 tokenizers - so probably why it's not giving you correct answers (maybe)
But @DaddyCodesAlot's analysis is also reasonable if my first point fails to improve results

Hi Dan, I am not using apply chat template. I am using my direct conversational dataset. Example in my post (example corpus qa io lines...). I try différent test and I return to you. Bests regards.

Wait when you mean direct conversational dataset - did you use our conversational style notebooks (ShareGPT / Llama -3?)

Wait when you mean direct conversational dataset - did you use our conversational style notebooks (ShareGPT / Llama -3?)

Hi Dan, @danielhanchen
I've already used your example of unsloth.chat_templates / def formatting_prompts_func(examples) code with your example for phi3 on my dataset in ShareGPT format. It's not a problem because my dataset preparation code can generate a dataset in Model Card Chat Llama3.1, ShareGPT, ALPACA...  But in this calculation run, as I have a Llama3.1 Model Card version of my dataset, I don't go through your unsloth.chat_templates / def formatting_prompts_func(examples) code, and I give it my dataset as a parameter, which has the following form.
Example of my JSONL line dataset ChatML LL3.1 Instruct, based on https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/ :
{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHow many parts is Gaul divided into according to Caesar?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nthree<|eot_id|>"}
The print result (another dataset line) :
---- train_dataset: Dataset({
features: ['text'],
num_rows: 20837
})
print("---- train_dataset:", train_dataset)
print("Sample entry:\n", train_dataset[0]["text"])

Sample entry:
 <|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>

Who did Cassivellaunus send as an intermediary to negotiate a surrender with Caesar?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Commius the Atrebatian<|eot_id|>

The same first example in my ShareGPT dataset JSON file version :
[
    {
        "conversations": [
            {
                "from": "system",
                "value": "You are an expert historian specializing in the ancient world and antiquity."
            },
            {
                "from": "human",
                "value": "How many parts is Gaul divided into according to Caesar?"
            },
            {
                "from": "gpt",
                "value": "three"
            }
        ]
    },
]

Do you think that my Llama 3.1 model card version is bad and that I'd be better off using my ShareGPT dataset version with your conversion function unsloth.chat_templates / def formatting_prompts_func(examples) ?
and obviously reload the latest version of unsloth since you explain in your reply that just after my question you updated unsloth's llama 3.1 tokenizer ? There are also @DaddyCodesAlot parameter improvements, but I'll do that later to evaluate the improvements...
Thanks for advance, David.

@dromeuf Actually wait did you use tokenizer.apply_chat_template? or our conversational notebooks - apologies but just after you posted, I updated all Llama 3.1 tokenizers - so probably why it's not giving you correct answers (maybe)
But @DaddyCodesAlot's analysis is also reasonable if my first point fails to improve results

@danielhanchen @DaddyCodesAlot I've used my ShareGPT version of my dataset with unsloth's llama-3 model card formatting function get_chat_template() and I've also increased the rank r 16 to 64, to a matrix of r=64, lora_alpha=64, lora_dropout=0.1, but the result is pretty much the same. The answers to my queries about inferring model 16 merged are not very good and poor.
IMPORTANT : I've checked my train and validation dataset for the inference test question I'm asking on book 6 chapter 21. In my train finetuning dataset, I have 41 questions-answers or instructions-outputs, including the complete text corpus in English, French and Latin (3), so 38 QA or IO, including QA on their gods Sun, Moon and Vulcan/Fire :
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>

Recognize the entities mentioned in the text as gods<|eot_id|><|start_header_id|>assistant<|end_header_id|>

**The sun, fire, and the moon**<|eot_id|>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>

Which gods do the Germans believe in?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

**The sun, fire, and the moon, as they are the only deities they can behold and by whose instrumentality they are obviously benefited.**<|eot_id|>

In my validation dataset part, I have 11 QA & IO.
I'm still surprised by the level of response to my inference questions. As DaddyCodesAlot suggested, Llama 3.1 instruct is probably not strong enough to learn this type of dataset ? or another probem ?
max_seq_length = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    load_in_4bit=True,
    dtype=None,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=64,
    lora_alpha=64,
    lora_dropout=0.1,
    target_modules=["q_proj", "k_proj", "v_proj", "up_proj", "down_proj", "o_proj", "gate_proj"],
    use_rslora=True,
    use_gradient_checkpointing="unsloth"
)

tokenizer = get_chat_template(
    tokenizer,
    chat_template = "llama-3",
    mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"},
)

def formatting_prompts_func(examples):
    convos = examples["conversations"]
    texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]
    return { "text" : texts, }
pass

logs :

Successfully installed datasets-2.21.0 dill-0.3.8 hf-transfer-0.1.8 multiprocess-0.70.16 pyarrow-17.0.0 sentencepiece-0.2.0 shtab-1.7.1 transformers-4.44.0 tyro-0.8.8 unsloth-2024.8 xxhash-3.5.0
Building wheel for unsloth (pyproject.toml) ... done
Created wheel for unsloth: filename=unsloth-2024.8-py3-none-any.whl size=143194 sha256=40f1749bf76faecfd472f3bd627845aee2e575ccc488df82bbdd93122675006f

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
==((====))==  Unsloth 2024.8: Fast Llama patching. Transformers = 4.44.0.
   \\   /|    GPU: NVIDIA A100-SXM4-40GB. Max memory: 39.564 GB. Platform = Linux.
O^O/ \_/ \    Pytorch: 2.3.1+cu121. CUDA = 8.0. CUDA Toolkit = 12.1.
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.26.post1. FA2 = False]

Unsloth: Dropout = 0 is supported for fast patching. You are using dropout = 0.1.
Unsloth will patch all other layers, except LoRA matrices, causing a performance hit.
Unsloth 2024.8 patched 32 layers with 0 QKV layers, 0 O layers and 0 MLP layers.
Paramètres du model préparé pour PEFT :
trainable params: 167,772,160 || all params: 8,198,033,408 || trainable%: 2.0465
None

print("Sample entry:\n", train_dataset[0]["text"])

Sample entry:
 <|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>

Who did Cassivellaunus send as an intermediary to negotiate a surrender with Caesar?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Commius the Atrebatian<|eot_id|>

GPU = NVIDIA A100-SXM4-40GB. Max memory = 39.564 GB.
6.457 GB of memory reserved.
==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1
   \\   /|    Num examples = 2,410 | Num Epochs = 2
O^O/ \_/ \    Batch size per device = 4 | Gradient Accumulation steps = 4
\        /    Total batch size = 16 | Total steps = 300
 "-____-"     Number of trainable parameters = 167,772,160

-- Training completed stats :
 TrainOutput(global_step=300, training_loss=0.32471315984924637, metrics={'train_runtime': 4099.8106, 'train_samples_per_second': 1.176, 'train_steps_per_second': 0.073, 'total_flos': 4.523655094264136e+17, 'train_loss': 0.32471315984924637, 'epoch': 1.9900497512437811})

-- Evaluation validation dataset results :
 {'eval_loss': 0.20628409087657928, 'eval_runtime': 107.9507, 'eval_samples_per_second': 4.863, 'eval_steps_per_second': 0.611, 'epoch': 1.9900497512437811}

----INFERENCE TEST----
print(model)

PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): LlamaForCausalLM(
      (model): LlamaModel(
        (embed_tokens): Embedding(128256, 4096)
        (layers): ModuleList(
          (0-31): 32 x LlamaDecoderLayer(
            (self_attn): LlamaAttention(
              (q_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (k_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=1024, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (v_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=1024, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (o_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (rotary_emb): LlamaExtendedRotaryEmbedding()
            )
            (mlp): LlamaMLP(
              (gate_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=14336, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=14336, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (up_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=14336, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=14336, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (down_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=14336, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=14336, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (act_fn): SiLU()
            )
            (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
            (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
          )
        )
        (norm): LlamaRMSNorm((4096,), eps=1e-05)
        (rotary_emb): LlamaRotaryEmbedding()
      )
      (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
    )
  )
)

Inference tests prompts poor :

[{'from': 'system', 'value': 'You are an expert historian specializing in the ancient world and antiquity.'}, {'from': 'human', 'value': 'What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?'}]

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>

What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

**They worship gods they have not seen, and have customs very different from the Gallic people**<|eot_id|>

[{'from': 'system', 'value': 'You are an expert historian specializing in the ancient world and antiquity.'}, {'from': 'human', 'value': "Que s'est-il passé à Gergovie en 52 BC ?"}]

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>user<|end_header_id|>

Que s'est-il passé à Gergovie en 52 BC?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

**Les habitants ont pris les armes, massacrés les citoyens romains et pillé leurs biens.**<|eot_id|>

@dromeuf Oh could you try chat_template = "llama-31", which is for Llama 3.1 - I just added it in last week.
On inference, did you enable FastLanguageModel.for_inference(model) before doing inference?

@dromeuf Oh could you try chat_template = "llama-31", which is for Llama 3.1 - I just added it in last week.

I did wonder about that, but I hadn't seen it in the doc notebook yesterday. I have a run in progress with the same dataset but using Mistral Nemo 12B Instruct. I'll relaunch with chat_template=“llama-31” tomorrow.

On inference, did you enable FastLanguageModel.for_inference(model) before doing inference?

Yes.
Kind regards.

@dromeuf Oh could you try chat_template = "llama-31", which is for Llama 3.1 - I just added it in last week.

Hi Dan @danielhanchen , I changed the chat_template to “llama-31”. I took the opportunity to change the rank to r=256 for this new run, but the result is worse. In fact a date now appears in the conversation system of my dataset ”Cutting Knowledge Date: December 2023 Today Date: 26 July 2024" (see logs below) !!!. One of the inference responses also contains the date !
tokenizer = get_chat_template(
    tokenizer,
    chat_template = "llama-31",
    mapping = {"role" : "from", "content" : "value", "user" : "human", "assistant" : "gpt"},
)

def formatting_prompts_func(examples):
    convos = examples["conversations"]
    texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]
    return { "text" : texts, }
pass

print()  : Sample entry :
 <|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 July 2024

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>human<|end_header_id|>

Who did Cassivellaunus send as an intermediary to negotiate a surrender with Caesar?<|eot_id|><|start_header_id|>gpt<|end_header_id|>

Commius the Atrebatian<|eot_id|>

logs :

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
==((====))==  Unsloth 2024.8: Fast Llama patching. Transformers = 4.44.0.
   \\   /|    GPU: NVIDIA A100-SXM4-40GB. Max memory: 39.564 GB. Platform = Linux.
O^O/ \_/ \    Pytorch: 2.3.1+cu121. CUDA = 8.0. CUDA Toolkit = 12.1.
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.26.post1. FA2 = False]
 "-____-"     Free Apache license: http://github.com/unslothai/unsloth

Unsloth: Dropout = 0 is supported for fast patching. You are using dropout = 0.1.
Unsloth will patch all other layers, except LoRA matrices, causing a performance hit.
Unsloth 2024.8 patched 32 layers with 0 QKV layers, 0 O layers and 0 MLP layers.

Paramètres du model préparé pour PEFT :
trainable params: 671,088,640 || all params: 8,701,349,888 || trainable%: 7.7125
None

GPU = NVIDIA A100-SXM4-40GB. Max memory = 39.564 GB.
8.332 GB of memory reserved.
==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1
   \\   /|    Num examples = 2,624 | Num Epochs = 2
O^O/ \_/ \    Batch size per device = 4 | Gradient Accumulation steps = 4
\        /    Total batch size = 16 | Total steps = 328
 "-____-"     Number of trainable parameters = 671,088,640

-- Training completed stats :
 TrainOutput(global_step=328, training_loss=5.754552464659621, metrics={'train_runtime': 4624.7583, 'train_samples_per_second': 1.135, 'train_steps_per_second': 0.071, 'total_flos': 5.272500370805883e+17, 'train_loss': 5.754552464659621, 'epoch': 2.0})

-- Evaluation validation dataset results :
 {'eval_loss': 2.709500551223755, 'eval_runtime': 121.7538, 'eval_samples_per_second': 4.731, 'eval_steps_per_second': 0.591, 'epoch': 2.0}

INFERENCE TEST (completely wrong answer with a date that appears in the system part of my question ??? !!!) :
[{'from': 'system', 'value': 'You are an expert historian specializing in the ancient world and antiquity.'}, {'from': 'human', 'value': 'What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?'}]

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 July 2024

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>human<|end_header_id|>

What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Commenthuman<|eot_id|>

[{'from': 'system', 'value': 'You are an expert historian specializing in the ancient world and antiquity.'}, {'from': 'human', 'value': "Que s'est-il passé à Gergovie en 52 BC ?"}]

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 July 2024

You are an expert historian specializing in the ancient world and antiquity.<|eot_id|><|start_header_id|>human<|end_header_id|>

Que s'est-il passé à Gergovie en 52 BC?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

<|start_header_id|>human<|end_header_id|>

<|eot_id|>

Another problem decidedly ;-) :
@danielhanchen Dan I ran a first computation run with Mistral Nemo 12B Instruct, but the inference results are curious too (there is a multiplication of characters). I didn't add EOS_TOKEN afterwards in the function to convert my ShareGPT dataset to chat_template=“mistral”.  Looking at your Mistral Nemo notebook, with the example for en ALPACA, I've just seen that you need to add EOS_TOKEN. In your blog you talk about a fix in progress. Do I still need to add EOS_TOKEN ?
Inference test Mistral Nemo 12B Instruct :

[{'from': 'system', 'value': 'You are an expert historian specializing in the ancient world and antiquity.'}, {'from': 'human', 'value': 'What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?'}]

<s>[INST] You are an expert historian specializing in the ancient world and antiquity. What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War? [/INST] the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, and, the, the, the, the, the, the, the, the, the, the, the, the, and, the,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

@dromeuf The date is correct - I asked the HF team to enquire about this, but Llama 3.1 Instruct's system prompt has the date.
Would it be possible for you to share the exact scripts of how you train these models (dataset part can be removed), so I can investigate further on my side - appreciate it a lot :)
Something seems wrong with the Mistral finetune as well

@dromeuf Oh saw your email - will definitely take a look

@dromeuf Oh saw your email - will definitely take a look

Hi Dan, were you able to work with my dataset and notebook to solve or identify the problem ? Kind regards,

@dromeuf Much apologies I did not have a chance - will try over the weekend sorry :(

@dromeuf Much apologies I did not have a chance - will try over the weekend sorry :(

@danielhanchen no problem Dan, the important thing is to understand what's going on, my error, dataset problem, 8B too low, other problems detected... so that we can move forward and other unsloth users can benefit from your expertise. I understand perfectly that you're very busy.

@danielhanchen Hello Dan, is your last blog Gradient Accumulation fix message related to my problem  ? Is it worth launching a new calculation run on my dataset to evaluate the new version and function or should I wait for your complete investigation ?

@dromeuf It might be related, but unsure - if you want to rerun it, you'll have to first use the nightly transformers version -
You can continue using unsloth_train or just use trainer.train() as in the blog:
!pip uninstall unsloth -y
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip uninstall transformers -y && pip install --upgrade --no-cache-dir "git+https://github.com/huggingface/transformers.git"

@danielhanchen Dan,  I ran the update of unsloth and transformers and then a new calculation run on my dataset with the code I sent you. The situation is identical, the response is delirious.
I wanted to test with your example and your new notebook for Llama 3.2 3B Instruct. I hardly changed any of your code and ran the training on my dataset. The response is a little less crazy, but not at all convincing. In your example, if I use the first inference without TextStreamer, nothing comes out at all (blank). If I use your inference code with TextStreamer then something comes out (but not convincingly).
I don't know what's going on with my dataset. Maybe it's too heavy for 3.1 8B and 3.2 3B because it contains texts in English, French and Latin ? I'm going to try injecting only English texts into a new learning run to see what happens.

@danielhanchen Dan, I've simplified my dataset by leaving only English texts, so I've removed the French and Latin texts. I re-run a learning calculation run on my Llama 3.1 8B Instruct code and your Llama 3.2 3B Instruct notebook code. Unfortunately, it's the same thing: the results of the inference questions are not good, if not delirious.
ex TextStreamer :
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 July 2024
You are an expert historian specializing in the ancient world and antiquitys.<|eot_id|><|start_header_id|>human<|end_header_id|>
What are the customs and gods of the Germans in chapter 21 of book 6 of the Gallic War?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
provide, You<|eot_id|>

Maybe consider open sourcing your dataset so that people can take a crack at it. My gut feeling is that there are 3 issues that are all contributing to this error.

You're using quantization of low parameter models, and thus increasing perplexity.
There's a rank issue. LORAs for small models at low ranks do not learn new information, but rather adapt the format and overfit.
There's an issue with the training process, maybe with certain parameters being off.

@DaddyCodesAlot My dataset isn't completely finished yet, and I wanted to complete and optimize it, particularly with regard to the proper names of the Romans, to make the characters perfectly identifiable and avoid homonyms. That's why I didn't want to publish a version that I think can be improved and completed. But if someone wants to work on it and, above all, benchmark the results of learning (which I don't see much of in the many publications), I'm willing to give them the rights to my google drive directory and my notebooks... you can send me an e-mail on my gmail (simple first.last@gmail.com). I'd love to be able to use unsloth for a larger project but for the moment there's a problem somewhere. I agree with your 3 points.
Kind regards, David Romeuf.


Why not use KKT principle?

Hello, your code has left a deep impression on me. SVR use Lagrange duality  and KKT to get the answer,but your code use gradient decent to train the model. I wondered if there are some thesis to stand for it? Had Cost function and gradient descent function in svr_rain_predictor.cpp  any Mathematical formulas?
Waiting for your reply,thank you!

Oh, sorry for this late response, the cost function is not actually right, there is no explicit hinging there, that is just one line. but it is there for the sake of memoir of how I started out. The mathematical part, it is based on the fact that SVMs have a solution due to its convex nature, i.e, one minima, so gradient decent can be used. Please let me know if I am wrong.


Searching for videos in Django admin doesn't work

Before I fix this: are you aware that these GlossVideo models you want to search are rarely used by Signbank, @ocrasborn ? For example, the video you see on the gloss detail page is just there because the page checks if there is a video file that matches the gloss name and id, and ignores the GlossVideo model completely. I think they are only created when you upload a new video on the gloss detail page.

Hm, hadn't realised that, no. OK, let's not spend time on this now, then.