Claims of being fast and scalable with no benchmarks

Can you run some benchmarks please comparing this to other golang routers / muxes in addition to the stdlib mux?
A few very popular ones:

chi
httprouter
govuk router

Hello SEJeff,
i will do it.
Best regards,
Marcel

Thanks!


Remove explicit use of vs2017_win-64

Fix https://github.com/conda-forge/ace-feedstock/issues/54 .
Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@conda-forge-admin, please rerender


ctrl-click in terminal doesn't handle trailing ':' in file:line:col: links

Issue Type: Bug
gcc generates errors in the terminal window that look like this:
File.cpp:52:32: error: blah blah blah

if you ctrl-click on the File.cpp bit in the terminal window, it populates the vscode file search popup with the text File.cpp:52:32:, but this yields no search results.
however, if you delete the trailing : from that text (resulting in File.cpp:52:32 then the file search popup shows the correct file and hitting <enter> takes you to the correct file/line/column.
Either:

the terminal window location parsing code should omit the trailing :, or
the file search popup parser should handle the trailing :

left-hand meet right-hand.
 Though I may not be super responsive at the moment.

Hi @Spongman, asking for my clarification.
In my case, nothing is happening on ctrl + mouseclick. But the system is asking for alt+click which is navigating me to the source of the error. I am adding the screen recording of what I did. I am new to this environment, can you help me with any configurations needed using which I can reproduce this?
Thanks

@tabaddor 👍
@kosuruvarunkumar it will be alt or ctrl depending on editor.multiCursorModifier

I think this only reproduces on Linux, OP was using a remote Linux host

I figured it out, this is because the file is at the root and so it has no slashes at all.
The example OP gave was with "File.cpp:52:32:".
The Ctrl+click and popup he's talking about isn't the suggestion of "open file in editor", it's the "search workspace" one.

part the issue is that when the string in the terminal is
/file:line:col:

the underline does NOT include the trailing ':'
and when the string in the terminal is
file:line:col:

the underline DOES include the trailing ':'
regardless of this difference, the search popup should handle the trailing ':' whether or not it is there.


patch for log messages from MirroringAudioTrack.java

Changed some "System.out" to "log.debug"

ty!


perf(linter): no_shadow_restricted_names only look up name in hashmap once

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#4472  👈
#4470
#4469
#4468
#4467
#4466
#4465
#4464
#4463
#4461
#4460
#4459
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite


[Suggestion] - Make examples in the Bitbucket DC Chart more clear

Suggestion
The section of the Bitbucket DC Chart about the jdbc URLs is a bit misleading.
See
https://github.com/atlassian/data-center-helm-charts/blob/6ac600c2e996ba61929cf5a23290199e702eb522/src/main/charts/bitbucket/values.yaml#L136
The URL needs to be enterd without the hyphen. Otherwise one would get an array in the statefulset, which prevents Bitbucket from starting.
Suggestion:
Remove the hyphens.
Product
Bitbucket
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Fair point, but i still find it misleading since a hypen in yaml is something different than a bullet point.
I'm also not the only one falling for this. See the last comment at https://community.atlassian.com/t5/Bitbucket-questions/helm-install-failed-with-cannot-understand-errors/qaq-p/1808153


Logout Function Error "Invalid Payload"

Version
nuxt-directus-next: 0.0.14
nuxt: 3.11.1
Steps to reproduce
Login with simple function:
async function attemptLogin() {
   const result = await login(email, password);
}

Successfully logs in and creates two cookies: 'directus_access_token' and 'directus_refresh_token'.
Attempt logout with simple function:
const handleLogout = async () => {
    await logoutUser();
})

What is Expected?
Expected to logout user and remove tokens.
Error response: "Invalid payload. The refresh token is required in either the payload or cookie."

Could you also share any directus configs within your nuxt.config.ts?
where is logoutUser() defined?
What happens when you do the following?
<template>

    <button @click="logout()">Logout</button>

</template>

@Sandros94 Thanks for the response.  Yeah I forgot to include that statement in my ticket.  I have that exact code in the logout component...just with some styling.
In terms of my nuxt.config.ts settings for the directus module, I have this:
directus: {
		url: 'https://admin.huestudios.company',
		moduleConfig: {
			devtools: true,
			readMeQuery: {
				fields: ['*'],
				updateState: true,
			},
			autoRefresh: {
				redirectTo: '/auth/signin',
			},
		},
	},

I am self-hosting directus with a Docker Compose instance.  Here is a screenshot of the response I get:

I reverted nuxt-directus-next to 0.0.12 and all works as intended.  That reverts the @directus/sdk dependency to 14.0.0.
From what I have read - as of 15.0.0 - the SDK auth composable has a new session auth mode.  I think this might be where the Invalid Payload error is coming from.
Looking through the docs at the moment.

I reverted nuxt-directus-next to 0.0.12 and all works as intended. That reverts the @directus/sdk dependency to 14.0.0.
From what I have read - as of 15.0.0 - the SDK auth composable has a new session auth mode. I think this might be where the Invalid Payload error is coming from.
Looking through the docs at the moment.

yeah I'm also looking for a way to reproduce it, but haven't been able to atm. Indeed the session auth mode has been introduced in<EMAIL_ADDRESS>but it is optional and not yet implemented in the module. So if I have to guess this shouldn't be it

I just did a basic test with the following and same result - login works, logout gives 'Invalid Payload' response:
package.json
{
  "name": "nuxt-app",
  "private": true,
  "type": "module",
  "scripts": {
    "build": "nuxt build",
    "dev": "nuxt dev",
    "generate": "nuxt generate",
    "preview": "nuxt preview",
    "postinstall": "nuxt prepare"
  },
  "dependencies": {
    "nuxt": "^3.11.1",
    "vue": "^3.4.21",
    "vue-router": "^4.3.0"
  },
  "devDependencies": {
    "nuxt-directus-next": "^0.0.14"
  }
}

nuxt.config.ts
export default defineNuxtConfig({
  devtools: { enabled: true },
  modules: ["nuxt-directus-next"],
  directus: {
    url: "************",
  },
});

app.vue

<template>

    Login
    Logout

Logged in as {{ user.email }}

</template>

Sorry for delay, but I'm not able to reproduce this.
Are you, by any chance, also logging in with the Directus App interface?
By default both this module and Directus use the same cookie name, but since by default Directus also set it to http only, Nuxt will lose the ability to read it and pass it for logging out.
To solve this you could either use useNuxtCookie: false (and let Directus handle everything) or change the Nuxt's cookie name via:
https://github.com/Intevel/nuxt-directus/blob/8030e515fde818ae9065404b6d4311eb7c842433/playground/nuxt.config.ts#L8-L12

Hello,
I get the same error using the suggested config.

https://github.com/Intevel/nuxt-directus/blob/8030e515fde818ae9065404b6d4311eb7c842433/playground/nuxt.config.ts#L8-L12

Directus: 10.10.5
@directus/sdk: ^15.1.0
nuxt-directus-next: ^0.0.14
The /auth/logout POST requests includes this body:
{
    "refresh_token": "***",
    "mode" :"cookie"
}

but errors with:
{
    "errors": [
       {
          "message": "Invalid payload. The refresh token is required in either the payload or cookie.",
          "extensions": {
             "code": "INVALID_PAYLOAD",
             "reason": "The refresh token is required in either the payload or cookie"
          }
       }
    ]
}

"directus": {
    "authConfig": {
        "useNuxtCookies": true,
        "refreshTokenCookieName": "nuxt-directus_refresh_token"
    },
    "moduleConfig": {
        "nuxtImage": {
            "useAuthToken": false,
            "useStaticToken": true
        }
    }
},

I noticed that the /auth/refresh POST request uses JSON
{
    "refresh_token": "***",
    "mode" :"json"
}

I copied the /auth/logout POST request to Postman and changed the mode to "json".
This returned 204 No content, which is expected and seems to make it work.
So we might need to use the json mode instead of the cookie mode for the logout request as well?

@ymilhahn you might have found it! But I need to do some refactoring and tests. I'll update

Yup, confirmed. The actual issue is upstream but since I need to also implement the session based auth mode I'll take this opportunity to prevent it from happening.

@Sandros94
It worked with client and now it works with nuxt-directus-next 0.0.15 too!
Thanks a bunch for the fast fix!

Thanks for the update @ymilhahn, @pvenableh feel free to reopen if you still have issues

Thank you @Sandros94 and @ymilhahn!
Updated and all works as intended.
"nuxt": "^3.11.2",
"nuxt-directus-next": "^0.0.15"

Which is using '@directus/sdk': 15.1.0 as a dependency.
Really appreciate the help and quick updates.

Maybe I'm missing something crucial, but all these changes described above are not reflected in any way in the current documentation. useDirectusUser() seems now undefined. Am I looking at the wrong docs here https://nuxt-directus.de/composables/usedirectususer?

@bennyzen if you are referring to the messages back in april then yes you have to look at this other documentation (for nuxt-directus-next)
https://next.nuxt-directus.de
But please do keep an eye on #273 to be notified when it gets merged, since it will break a lot of functionalities of how nuxt-directus-next currently works.
I haven't yet rewritten neither user nor auth logics, so I cannot comment on how they will work. If you want to leave an opinion of how you use them you can do so at #271


pdksync - Add weekly scheduled workflows

Add weekly scheduled workflows
pdk version: 1.16.0

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@321a7bc). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #411   +/-   ##
=========================================
  Coverage          ?   16.66%
=========================================
  Files             ?        1
  Lines             ?        6
  Branches          ?        0
=========================================
  Hits              ?        1
  Misses            ?        5
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 321a7bc...c3d187b. Read the comment docs.


Sticky Footer

Hey!
I'm using the NavigationControllerIOS. I was wondering if there is a way to persist a sticky footer across all view that get pushed in? Any help would be much appreciated!
thanks!

Hey, we didn't quite understand your request.
Can you please open an issue in react-native-navigation and add a screenshot/example of your request?
Thanks!

@drorbiran maybe this can be achieved using bottom in-app notification.
Not really a documented feature, see a reference here: https://github.com/wix/react-native-controllers/blob/c86f28c69a60385ce43c9a2cfb3d5cecd10ca3ef/index.js#L261
Examples are available in the example project


Create GD.innovation.txt

This is my college name

@iyyann Please let me know what is your actual email domain? What your email address end with? @innovation.gd.edu?

I'm sorry, I'm a Chinese college student, but I don't have edu mailbox. At present, I have only one school number. The mailbox now being used is<EMAIL_ADDRESS>and a1422476675@163. com, I've been using phpStorm for more than a year), and I want to get help

@iyyann I am sorry, if you do not have an email address which is in your university email domain (such as, for example, username@gdut.edu.cn), and you do not have an ISIC card, the only way to apply for free JetBrains license for you is to provide us with an official document which proves that you are a student. Usually, a university gives to the students such a document with your name, photo and name of the university. You can apply at https://www.jetbrains.com/shop/eform/students, click on the tab "OFFICIAL DOCUMENT" and attach the photo of your document there.
I have to close your request without merging. Please reopen and comment if you have an update regarding the abovementioned issue. Thank you for the understanding!

Thank you for doing something for me. I'll try it.


Xiaomi MiIO Remote: Lazy discover disabled

The Chuang Mi IR Remote Controller wants to be re-discovered every 5 minutes. As long as polling is disabled the device should be re-discovered in front of every command.
https://github.com/rytilahti/python-miio/issues/210#issuecomment-368411932
Improves sporadic timeouts/retries (if the device needs to be re-discovered):
2018-02-26 07:07:48 DEBUG (Thread-19) [miio.device] <IP_ADDRESS>:54321 >>: {'method': 'miIO.ir_play', 'id': 4, 'params': {'code': 'Z6XHAL0BAACkAgAA0QIAAOYFAAARBgAABxEAAKgRAAD2EQAASBIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABWMQExMQEBMQEBMQEBMTEBMQExMTExATExMQEBAQExAQExMTEBAQEBAQEBATExMTExgVcxATExAQExAQExAQExMQExAUExMTEBMTExAQEBATEBATExMQEBAQEBAgEBMTExMTEB', 'freq': 38400}}
2018-02-26 07:07:53 ERROR (Thread-19) [miio.device] Got error when receiving: timed out
2018-02-26 07:07:53 WARNING (Thread-19) [miio.device] Retrying with incremented id, retries left: 3

Sorry, somehow forgot about this, looks good even if a bit verbose :-)


VATLM: ModuleNotFoundError: No module named 'fairseq.data.audio.multi_corpus_dataset_audio'

Hi, there. I'm trying to extract features with the released VATLM pre-trained models. Here's what I did.
Firstly, I tried loading the pre-trained model:
# cwd: ..../av_hubert
import vathubert
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task(["./finetune_large_vox2_v_433h.pt"])

but an error occurs: AssertionError: Could not infer task type from {'_name': 'vat_hubert_pretraining'....
It's because the task vat_hubert_pretraining wasn't been successfully registered by Fairseq, and the reason for this error is that the sub-package vathubert.tasks is not successfully initialized since there's not __init__.py in it (from what I know, if there're not __init__.py, it's just a "NameSpace" package instead of a normal package). To fix it, I add __init__.py to ./vathubert/tasks.py, which content is
from .vathubert_pretraining import VATHubertPretrainingTask

This is for registering the task in Fairseq. However, another error occurs in
https://github.com/microsoft/SpeechT5/blob/996bb4219ce1890a9984eff9dda65855af4cb446/VATLM/vat_hubert/vathubert/tasks/vathubert_pretraining.py#L40

ModuleNotFoundError: No module named 'fairseq.data.audio.multi_corpus_dataset_audio'

I read the source code of Fairseq and didn't find the file multi_corpus_dataset_audio.py in the directory fairseq.data.audio
Did I miss anything? Or is there some bug in the code? Any help is appreciated, Thanks!

@xiabingquan Thanks for the question, I forgot to put multi_corpus_dataset_audio.py in the folder. You put multi_corpus_dataset_audio.py in the fairseq/data/audio/ folder and this problem will be solved.
# Copyright (c) Facebook, Inc. and its affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

import logging
from os import replace
import time
from collections import OrderedDict
from typing import Any, Dict, List, Optional

import numpy as np
from fairseq.data import data_utils

from fairseq.data import FairseqDataset

logger = logging.getLogger(__name__)

class MultiCorpusDataset(FairseqDataset):
    """
    Stores multiple instances of FairseqDataset together.
    Unless batch_sample=True, requires each instance
    to be the same dataset, as the collate method needs to work on batches with
    samples from each dataset.

    Allows specifying a distribution over the datasets to use. Note that unlike
    MultiCorpusSampledDataset, this distribution allows sampling for each item,
    rather than on a batch level. Note that datasets with sampling probabilty
    of 0 will be skipped.

    Each time ordered_indices() is called, a new sample is generated with
    the specified distribution.

    Args:
        datasets: a OrderedDict of FairseqDataset instances.
        distribution: a List containing the probability of getting an utterance from
                        corresponding dataset
        seed: random seed for sampling the datsets
        sort_indices: if true, will sort the ordered indices by size
        batch_sample: if true, will ensure each batch is from a single dataset
    """

    def __init__(
        self,
        datasets: Dict[str, FairseqDataset],
        max_positions: Dict,
        distribution: List[float],
        max_tokens_ratio: List[float],
        seed: int,
        sort_indices: bool = False,
        distributed_rank: Optional[int] = None,
    ):
        super().__init__()
        assert isinstance(datasets, OrderedDict)
        assert len(datasets) == len(distribution)
        # assert sum(distribution) == 1
        self.datasets = datasets
        self.distribution = distribution
        self.max_tokens_ratio = max_tokens_ratio
        self.seed = seed
        self.sort_indices = sort_indices
        self.distributed_rank = distributed_rank
        self.max_positions = max_positions

        # Avoid repeated conversions to list later
        self.dataset_list = list(datasets.values())
        self.total_num_instances = 0

        # first_dataset = self.dataset_list[0]

        self.num_instances_per_dataset = []
        self.dataset_offsets = []
        for i, dataset in enumerate(self.dataset_list):
            assert isinstance(dataset, FairseqDataset)
            # assert type(dataset) is type(first_dataset)
            self.num_instances_per_dataset.append(
                0 if self.distribution[i] == 0 else len(dataset)
            )
            self.dataset_offsets.append(self.total_num_instances)
            self.total_num_instances += self.num_instances_per_dataset[i]

    def ordered_indices(self):
        start = time.time()
        with data_utils.numpy_seed(self.seed, self.epoch):
            logger.info(f"sampling new dataset with seed {self.seed} epoch {self.epoch}")
            sampled_indices = {}

            # For each dataset i, sample self.distribution[i] * self.total_num_instances
            for i, key in enumerate(self.datasets):
                tp = time.time()
                if self.distribution[i] == 0:
                    # skip dataset if sampling probability is 0
                    continue

                if i < len(self.datasets) - 1:
                    num_instances = int(self.distribution[i] * self.total_num_instances)
                    high = self.dataset_offsets[i + 1]
                else:
                    num_instances = int(self.distribution[i] * self.total_num_instances)
                    high = self.total_num_instances

                logger.info(f"sampling {num_instances} from {key} dataset")

                # First, add k copies of the dataset where k = num_instances // len(dataset).
                # This ensures an equal distribution of the data points as much as possible.
                # For the remaining entries randomly sample them
                dataset_size = len(self.datasets[key])
                num_copies = num_instances // dataset_size
                dataset_indices = np.random.permutation(high - self.dataset_offsets[i])[: num_instances - num_copies * dataset_size]
                if num_copies > 0:
                    dataset_indices = np.concatenate(
                            (
                                np.repeat(
                                    np.arange(high - self.dataset_offsets[i]), num_copies
                                ),
                                dataset_indices,
                            )
                        )
                # ### filter by size
                # if self.max_positions[key] is not None:
                #     dataset_indices, ignored = self.datasets[key].filter_indices_by_size(
                #         dataset_indices,
                #         self.max_positions[key],
                #     )
                #     if len(ignored) > 0:
                #         logger.warning(
                #             (
                #                 "{:,} samples have invalid sizes and will be skipped, "
                #                 "max_positions={}, first few sample ids={}"
                #             ).format(len(ignored), self.max_positions[key], ignored[:10])
                #         )

                if self.sort_indices:
                    logger.info(" - sampled indices took {}s".format(time.time() - tp))
                    tp = time.time()
                    dataset_indices = np.sort(dataset_indices)
                    dataset_indices = self.datasets[key].ordered_indices()[dataset_indices] + self.dataset_offsets[i]
                    logger.info(" - ordered_indices took {}s".format(time.time() - tp))
                else:
                    np.random.shuffle(dataset_indices)

                sampled_indices[key] = dataset_indices

            logger.info(
                "multi_corpus_dataset ordered_indices took {}s".format(
                    time.time() - start
                )
            )
            return sampled_indices

    def _map_index(self, index: int):
        """
        If dataset A has length N and dataset B has length M
        then index 1 maps to index 1 of dataset A, and index N + 1
        maps to index 1 of B.
        """
        counter = 0
        for num_instances, key in zip(self.num_instances_per_dataset, self.datasets):
            if index < counter + num_instances:
                return index - counter, key
            counter += num_instances
        raise ValueError(
            "Invalid index: {}, max: {}".format(index, self.total_num_instances)
        )

    def __len__(self):
        """
        Length of this dataset is the sum of individual datasets
        """
        return self.total_num_instances

    def __getitem__(self, index):
        new_index, key = self._map_index(index)
        try:
            item = self.datasets[key][new_index]
            item["full_id"] = index
            return item
        except Exception as e:
            e.args = (f"Error from {key} dataset", *e.args)
            raise

    def collater(self, samples):
        """
        If we are doing batch sampling, then pick the right collater to use.

        Otherwise we assume all collaters are the same.
        """
        if len(samples) == 0:
            return None

        samples_dict = {key: [] for key in self.datasets}
        for s in samples:
            _, key = self._map_index(s["full_id"])
            samples_dict[key].append(s)

        batch = {}
        for key in samples_dict:
            if len(samples_dict[key]) == 0:
                continue
            batch[key] = self.datasets[key].collater(samples_dict[key])

        return batch

    def num_tokens(self, index: int):
        index, key = self._map_index(index)
        return self.datasets[key].num_tokens(index)

    def size(self, index: int):
        index, key = self._map_index(index)
        return self.datasets[key].size(index)

    @property
    def can_reuse_epoch_itr_across_epochs(self):
        return False

    def set_epoch(self, epoch, **unused):
        super().set_epoch(epoch)
        logger.info(f"setting epoch of multi_corpus_dataset to {epoch}")
        for ds in self.dataset_list:
            if hasattr(ds, "set_epoch"):
                ds.set_epoch(epoch)
        self.epoch = epoch

    @property
    def supports_prefetch(self):
        return False

    @property
    def supports_fetch_outside_dataloader(self):
        return all(
            self.datasets[key].supports_fetch_outside_dataloader
            for key in self.datasets
        )

    def batch_by_size(
        self,
        indices,
        max_tokens=None,
        max_sentences=None,
        required_batch_size_multiple=1,
    ):
        dataset_indices = indices
        batches_dict = {}
        for n, key in enumerate(dataset_indices):
            max_tokens_ratio = self.max_tokens_ratio[n]
            cur_batches = super().batch_by_size(
                np.array(dataset_indices[key], dtype=np.int64),
                round(max_tokens * max_tokens_ratio),
                max_sentences,
                required_batch_size_multiple,
            )
            logger.info(f"Created {len(cur_batches)} batches for dataset {key}")
            batches_dict[key] = cur_batches

        return batches_dict

    def get_batch_sampler(
        self,
        indices,
        num_shards,
        seed,
        max_tokens=None,
        max_sentences=None,
        required_batch_size_multiple=1,
        split_modality_batch=False,
    ):

        def batch_sampler(dataset, epoch):
            start = time.time()
            batches_dict = dataset.batch_by_size(
                indices,
                max_tokens=max_tokens,
                max_sentences=max_sentences,
                required_batch_size_multiple=required_batch_size_multiple,
            )
            logger.info(f"multi_corpus_dataset, batch_by_size took {time.time() - start}s")
            start = time.time()
            new_batches = []

            ### shuffle inner group size, split into speech/text batches
            shuffled_batches_list = []
            speech_batches = []
            ### we should specify the speech_batches because: we need concatenate different speech datasets (e.g. ltr or km) instead of loading them parellelly.
            for name, batches in batches_dict.items():
                batches = inner_bucket_shuffle(batches, seed+epoch, num_shards*10)
                batches = batches[: (len(batches) // num_shards) * num_shards]
                if len(batches) == 0:
                    logger.warning(f"Sample 0 batch for {name}, you should ensure that no {name} data provided.")
                elif name.startswith("speech"):
                    speech_batches += batches
                else:
                    batches = shuffle_buckets(batches, seed=seed+epoch, inner_shuf=False)
                    shuffled_batches_list.append(batches)
            # speech_batches = shuffle_buckets(speech_batches, seed=seed+epoch, inner_shuf=False)
            # shuffled_batches_list.append(speech_batches)

            ### create the final new_batches
            num_batch = min(len(batches) for batches in shuffled_batches_list)
            if split_modality_batch:
                for i in range(0, num_batch, num_shards):
                    for batches in shuffled_batches_list:
                        new_batches += batches[i: i + num_shards]
            else:
                for i in range(num_batch):
                    new_batches.append(np.concatenate([batches[i] for batches in shuffled_batches_list]))

            logger.info(f"multi_corpus_dataset sample {len(new_batches)} batches, took {time.time() - start}s")
            return new_batches

        def inner_bucket_shuffle(batches, seed, bucket_size=10, thr=0):
            """we assert batches is sorted form long to short.
                shuffle samples in a buctet(e.g. 10 batches).
                batches: a list of numpy array"""
            num_batch = len(batches)
            new_batches = []
            num_buckets = len(batches) // bucket_size
            i = 0
            while i < num_batch:
                if (i < bucket_size * thr or
                    i >= bucket_size * (num_buckets - thr)
                ):
                    new_batches.append(batches[i])
                    i += 1
                else:
                    group = np.concatenate(batches[i: i+bucket_size])
                    with data_utils.numpy_seed(seed):
                        np.random.shuffle(group)
                    new_batches += np.array_split(group, bucket_size)
                    i += bucket_size
            assert all([len(batch) > 0 for batch in new_batches])
            return new_batches

        def shuffle_buckets(batches, seed, inner_shuf=True):
            if inner_shuf:
                batches = inner_bucket_shuffle(batches, seed, num_shards*10)
            batches = [batches[i: i + num_shards] for i in range(0, len(batches)-num_shards+1, num_shards)]
            assert len(batches[-1]) == num_shards
            new_batches = []
            with data_utils.numpy_seed(seed):
                np.random.shuffle(batches)
                for group in batches:
                    new_batches += group
            return new_batches

        return batch_sampler

@zqs01 Thanks for your quick reply. It did solve the problem. However, I come across another problem. When I'm trying to load the pre-trained model VatLM-Large	VoxCeleb2 + LRS3 + paired audio+text+audio-no finetune using the following codes, an error occurred
ckpt_ph = "./VoxCeleb2_LRS3_paired-audio-text-audio_pretrain.pt"
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task(
            [ckpt_ph]
        )

The full tracebrack is as follows:

Traceback (most recent call last):
File "", line 2, in
File "/home/xiabingquan/repos/fairseq/fairseq/checkpoint_utils.py", line 473, in load_model_ensemble_and_task
model = task.build_model(cfg.model, from_checkpoint=True)
File "/home/xiabingquan/repos/fairseq/fairseq/tasks/fairseq_task.py", line 340, in build_model
model = models.build_model(cfg, self, from_checkpoint)
File "/home/xiabingquan/repos/fairseq/fairseq/models/init.py", line 106, in build_model
return model.build_model(cfg, task)
File "/home/xiabingquan/repos/SpeechT5/VATLM/vathubert/models/vathubert.py", line 456, in build_model
model = VATHubertModel(cfg, task.cfg, task.dictionaries, **kwargs)
File "/home/xiabingquan/repos/SpeechT5/VATLM/vathubert/models/vathubert.py", line 407, in init
self.encoder = TransformerEncoder(cfg)
File "/home/xiabingquan/repos/fairseq/fairseq/models/wav2vec/wav2vec2.py", line 953, in init
self.required_seq_len_multiple = args.required_seq_len_multiple
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 305, in getattr
self._format_and_raise(key=key, value=None, cause=e)
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
format_and_raise(
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
_raise(ex, cause)
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
raise ex  # set end OC_CAUSE=1 for full backtrace
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 303, in getattr
return self._get_impl(key=key, default_value=DEFAULT_VALUE_MARKER)
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 361, in _get_impl
node = self._get_node(key=key)
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 383, in _get_node
self._validate_get(key)
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 135, in _validate_get
self._format_and_raise(
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
format_and_raise(
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
_raise(ex, cause)
File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: Key 'required_seq_len_multiple' not in 'VATHubertConfig'
full_key: required_seq_len_multiple
reference_type=Optional[VATHubertConfig]
object_type=VATHubertConfig

After reading the source code, I found that this error is caused by the incorrect configuration files used in vathubert/models/vathubert.py when initializing the TransformerEncoder.
https://github.com/microsoft/SpeechT5/blob/a7b72c6932e82930eb01331c2c922a0687157367/VATLM/vat_hubert/vathubert/models/vathubert.py#L402
The TransformerEncoder is imported from Wav2Vec2, so the input cfg should be a Wav2Vec2Config instead of a VATHubertConfig
https://github.com/microsoft/SpeechT5/blob/a7b72c6932e82930eb01331c2c922a0687157367/VATLM/vat_hubert/vathubert/models/vathubert.py#L24-L27
Since the above two dataclasses have a lot of different keys, Fairseq will throw errors when accessing a key that exists in Wav2Vec2Config but not int VATHubertConfig, such as the above required_seq_len_multiple. This error could be fixed by manually adding the keys of Wav2Vec2Config to VATHubertConfig, but this strategy isn't elegant and may lead to some potential errors (For example, Wav2Vec2 may quantize and mask the inputs, which is definitely what we want when extracting features). How can we fix this problem? Or am I doing something wrong?
One thing is note is that is I encountered the same problem when I was trying to load the pre-trained models of AVHuBERT(since both of you use the same template), but it seems that other people is working fine with the original codes. Bothered a lot.

@xiabingquan Since the AV-HuBERT and VATLM codes are not placed in the fairseq default folder, you need to specify the loading path. The following code can load AV-HuBERT and VATLM. I hope these can help you.
from argparse import Namespace
import fairseq
from fairseq import checkpoint_utils, utils

utils.import_user_module(Namespace(user_dir='/path/to/vat_hubert/vathubert'))
models, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task(["/path/to/model"])

print(saved_cfg)

@xiabingquan Since the AV-HuBERT and VATLM codes are not placed in the fairseq default folder, you need to specify the loading path. The following code can load AV-HuBERT and VATLM. I hope these can help you.
from argparse import Namespace
import fairseq
from fairseq import checkpoint_utils, utils

utils.import_user_module(Namespace(user_dir='/path/to/vat_hubert/vathubert'))
models, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task(["/path/to/model"])

print(saved_cfg)

Thanks for the hint, but it's not my problem here. My problem is that, inside the __init__ method of VATHubertModel,
https://github.com/microsoft/SpeechT5/blob/a7b72c6932e82930eb01331c2c922a0687157367/VATLM/vat_hubert/vathubert/models/vathubert.py#L24-L27
https://github.com/microsoft/SpeechT5/blob/a7b72c6932e82930eb01331c2c922a0687157367/VATLM/vat_hubert/vathubert/models/vathubert.py#L402
when initializing the TransformerEncoder, it passes a VATHubertConfig instead of a Wav2Vec2Config, which leads to further errors

File "/home/xiabingquan/miniconda3/envs/lrs/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
raise ex # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: Key 'required_seq_len_multiple' not in 'VATHubertConfig'
full_key: required_seq_len_multiple
reference_type=Optional[VATHubertConfig]
object_type=VATHubertConfig

For example, as the above trackback shows, the program tries to access the attribute required_seq_len_multiple  of the input args  by calling self.required_seq_len_multiple = args.required_seq_len_multiple, but required_seq_len_multiple is an valid key only for Wav2Vec2Config while not for VATHubertConfig. AVHubert has exactly the same issue.
I solved my problem by adding the keys needed to initialize TransformerEncoder to VATHubertConfig with the following two steps.
Step1.
Add  LAYER_TYPE_CHOICES = ChoiceEnum(["transformer", "conformer"]) after https://github.com/microsoft/SpeechT5/blob/a7b72c6932e82930eb01331c2c922a0687157367/VATLM/vat_hubert/vathubert/models/vathubert.py#L61-L62
Step2: Add the following lines
    layer_type: LAYER_TYPE_CHOICES = field(
        default="transformer", metadata={"help": "layer type in encoder"}
    )   # No masking
    pos_conv_depth: int = field(
        default=1,
        metadata={"help": "depth of positional encoder network"},
    )
    max_positions: int = field(default=100000, metadata={"help": "Max positions"})
    checkpoint_activations: bool = field(
        default=False,
        metadata={"help": "recompute activations and save memory for extra compute"},
    )
    # FP16 optimization
    required_seq_len_multiple: int = field(
        default=1,
        metadata={
            "help": "pad the input to encoder such that the sequence length is divisible by multiple"
        },
    )
    # Conformer
    depthwise_conv_kernel_size: int = field(
        default=31,
        metadata={
            "help": "depthwise-conv-kernel-size for convolution in conformer layer"
        },
    )
    attn_type: str = field(
        default="",
        metadata={"help": "if espnet use ESPNET MHA"},
    )
    pos_enc_type: str = field(
        default="abs",
        metadata={"help": "Positional encoding type to use in conformer"},
    )
    fp16: bool = field(default=False, metadata={"help": "If fp16 is being used"})

after https://github.com/microsoft/SpeechT5/blob/a7b72c6932e82930eb01331c2c922a0687157367/VATLM/vat_hubert/vathubert/models/vathubert.py#L318
This way, the missing keys are added to VATHubertConfig and TransformerEncoder could be initialized happily.
I'm not sure whether I've done it the right way, but it does work. The question is, some keys missing shouldn't happen since it's a very big bug and could be easily spotted, so I guess it's mostly my problem instead of the repo's. Any reply on how to load models without any modification to the original codes is welcomed. Thanks! : )

btw, thanks for the great work!!!

@xiabingquan I haven't tried this method for loading models before, thank you for your solution. If you have new questions, you can reopen issues or contact me by email.


Implement Db cluster CRUD

This PR implements basic db cluster crud operations.

seeing an error in the tests
liskl@DESKTOP-D9PIEKG:~/k8s-rds$ git checkout db-cluster
branch 'db-cluster' set up to track 'origin/db-cluster'.
Switched to a new branch 'db-cluster'
liskl@DESKTOP-D9PIEKG:~/k8s-rds$ git ffwd-update
Fetching origin
 branch restore-from-snapshot was 1 commit(s) behind of origin/restore-from-snapshot. reseting local branch to remote
liskl@DESKTOP-D9PIEKG:~/k8s-rds$ make docker-push
docker build --load -t liskl/k8s-rds:db-cluster-1663267453 .
[+] Building 58.0s (16/18)
 => [internal] booting buildkit                                                                                1.4s
 => => starting container buildx_buildkit_competent_haibt0                                                                                1.4s
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 384B                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                2.0s
 => [internal] load metadata for docker.io/library/golang:1.17.0-alpine                                                                                2.1s
 => [auth] library/golang:pull token for registry-1.docker.io                                                                                0.0s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                                                0.0s
 => [builder 1/7] FROM docker.io/library/golang:1.17.0-alpine@sha256:c03a922dfb0d6d3a94d46cc5be69002011763595ac2786068fde63ee174d797b                  0.0s
 => => resolve docker.io/library/golang:1.17.0-alpine@sha256:c03a922dfb0d6d3a94d46cc5be69002011763595ac2786068fde63ee174d797b                          0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 408.38kB                                                                                0.0s
 => [stage-1 1/3] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                0.0s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                        0.0s
 => CACHED [builder 2/7] WORKDIR /app                                                                                0.0s
 => CACHED [builder 3/7] RUN apk --no-cache add git make                                                                                0.0s
 => [builder 4/7] COPY Makefile go.mod go.sum ./                                                                                0.1s
 => [builder 5/7] RUN make mod tools                                                                                38.7s
 => [builder 6/7] COPY . .                                                                                0.6s
 => ERROR [builder 7/7] RUN make test lint build                                                                                15.0s
------
 > [builder 7/7] RUN make test lint build:
#0 0.111 CGO_ENABLED=0 go test ./... -v
#0 13.32 # github.com/sorenmat/k8s-rds [github.com/sorenmat/k8s-rds.test]
#0 13.32 ./main_test.go:53:25: not enough arguments in call to excluded
#0 13.32        have (*crd.Database, []string, []string)
#0 13.32        want (string, string, []string, []string)
#0 13.32 note: module requires Go 1.18
#0 14.51 FAIL   github.com/sorenmat/k8s-rds [build failed]
#0 14.51 ?      github.com/sorenmat/k8s-rds/client      [no test files]
#0 14.51 === RUN   TestMarshal
#0 14.51 apiVersion: k8s-rds.io/v1
#0 14.51 kind: Database
#0 14.51 metadata:
#0 14.51   creationTimestamp: null
#0 14.51   name: my_db
#0 14.51   namespace: default
#0 14.51 spec:
#0 14.51   MaxAllocatedSize: 20
#0 14.51   backupretentionperiod: 10
#0 14.51   class: db.t2.micro
#0 14.51   dbname: database_name
#0 14.51   encrypted: true
#0 14.51   engine: postgres
#0 14.51   multiaz: true
#0 14.51   password:
#0 14.51     key: key
#0 14.51     name: DB-Secret
#0 14.51   provider: local
#0 14.51   size: 20
#0 14.51   storagetype: gp2
#0 14.51   username: dbuser
#0 14.51   version: ""
#0 14.51 status: {}
#0 14.51
#0 14.51 --- PASS: TestMarshal (0.00s)
#0 14.51 === RUN   TestCRDValidationWithValidInput
#0 14.51 --- PASS: TestCRDValidationWithValidInput (0.00s)
#0 14.51 === RUN   TestCaseInsensitiveInput
#0 14.51 --- PASS: TestCaseInsensitiveInput (0.00s)
#0 14.51 === RUN   TestDatabaseSizeIsTooSmall
#0 14.51 --- PASS: TestDatabaseSizeIsTooSmall (0.00s)
#0 14.51 === RUN   TestDatabaseSizeIsTooBig
#0 14.51 --- PASS: TestDatabaseSizeIsTooBig (0.00s)
#0 14.51 === RUN   TestBackupRetentionPeriodIsTooLong
#0 14.51 --- PASS: TestBackupRetentionPeriodIsTooLong (0.01s)
#0 14.51 === RUN   TestInvalidDatabaseNameWithDashSeparator
#0 14.51 --- PASS: TestInvalidDatabaseNameWithDashSeparator (0.00s)
#0 14.51 === RUN   TestInvalidDatabaseNameStartingWithANumber
#0 14.51 --- PASS: TestInvalidDatabaseNameStartingWithANumber (0.00s)
#0 14.51 === RUN   TestInvalidUsername
#0 14.51 --- PASS: TestInvalidUsername (0.00s)
#0 14.51 === RUN   TestInvalidStorageType
#0 14.51 --- PASS: TestInvalidStorageType (0.00s)
#0 14.51 === RUN   TestIopsTooSmall
#0 14.51 --- PASS: TestIopsTooSmall (0.00s)
#0 14.51 === RUN   TestIopsTooBig
#0 14.51 --- PASS: TestIopsTooBig (0.00s)
#0 14.51 PASS
#0 14.51 ok     github.com/sorenmat/k8s-rds/crd 0.048s
#0 14.51 ?      github.com/sorenmat/k8s-rds/kube        [no test files]
#0 14.51 === RUN   TestConvertSpecToDeployment
#0 14.51 --- PASS: TestConvertSpecToDeployment (0.00s)
#0 14.51 === RUN   TestCreateDatabase
#0 14.51 2022/09/15 18:45:06 creating pvc mydb
#0 14.51 2022/09/15 18:45:06 creating database mydb
#0 14.51 --- PASS: TestCreateDatabase (0.00s)
#0 14.51 === RUN   TestUpdateDatabase
#0 14.51 2022/09/15 18:45:06 creating pvc mydb
#0 14.51 2022/09/15 18:45:06 creating database mydb
#0 14.51 2022/09/15 18:45:06 updating pvc mydb
#0 14.51 2022/09/15 18:45:06 Specs mydb has same size: not updating pvc
#0 14.51 2022/09/15 18:45:06 updating database mydb
#0 14.51 --- PASS: TestUpdateDatabase (0.00s)
#0 14.51 PASS
#0 14.51 ok     github.com/sorenmat/k8s-rds/local       0.009s
#0 14.51 ?      github.com/sorenmat/k8s-rds/provider    [no test files]
#0 14.52 === RUN   TestConvertSpecToInput
#0 14.52 --- PASS: TestConvertSpecToInput (0.00s)
#0 14.52 === RUN   TestGetIDFromProvider
#0 14.52 --- PASS: TestGetIDFromProvider (0.00s)
#0 14.52 === RUN   TestToTags
#0 14.52 2022/09/15 18:45:11 WARNING: Not Adding annotation KV to tags: kubectl-key2-test test-value
#0 14.52 2022/09/15 18:45:11 WARNING: Not Adding annotation KV to tags: annotation-key1-test banjmdvgeezuadqvehvqaxxmzwykirejkwvktkxmvjdevcfhqootqyfdfvqatjiebglktdswnvzxcpnstvrurpfjfuxhsjvgogrnhazjizakttdncmjnbofvwcsccigfcyxzlunfndcjteuqmjpslqvefvobfnejjxtwbyrkcvsvqokkrskrryzbhhayegyuwhugyorkltmsipvznxkonqzzwihjdejqgzfjivjdqmieidkowryfjnnyrxszsyhnpfeepxyoliskexxpjtxn
#0 14.52 2022/09/15 18:45:11 WARNING: Not Adding annotation KV to tags: kubectl-key2-test test-value
#0 14.52 2022/09/15 18:45:11 WARNING: Not Adding CRD labels KV to tags: label-key2-test banjmdvgeezuadqvehvqaxxmzwykirejkwvktkxmvjdevcfhqootqyfdfvqatjiebglktdswnvzxcpnstvrurpfjfuxhsjvgogrnhazjizakttdncmjnbofvwcsccigfcyxzlunfndcjteuqmjpslqvefvobfnejjxtwbyrkcvsvqokkrskrryzbhhayegyuwhugyorkltmsipvznxkonqzzwihjdejqgzfjivjdqmieidkowryfjnnyrxszsyhnpfeepxyoliskexxpjtxn
#0 14.52 --- PASS: TestToTags (0.00s)
#0 14.52 === RUN   TestTags
#0 14.52 --- PASS: TestTags (0.00s)
#0 14.52 === RUN   TestTagsWithSpaces
#0 14.52 --- PASS: TestTagsWithSpaces (0.00s)
#0 14.52 === RUN   TestConvertSpecToDeleteInput_enabled
#0 14.52 --- PASS: TestConvertSpecToDeleteInput_enabled (0.00s)
#0 14.52 === RUN   TestConvertSpecToDeleteInput_disabled
#0 14.52 --- PASS: TestConvertSpecToDeleteInput_disabled (0.00s)
#0 14.52 === RUN   TestConvertSpecToModifyInput_withApplyImmediately
#0 14.52 --- PASS: TestConvertSpecToModifyInput_withApplyImmediately (0.00s)
#0 14.52 === RUN   TestConvertSpecToModifyInput_withoutApplyImmediately
#0 14.52 --- PASS: TestConvertSpecToModifyInput_withoutApplyImmediately (0.00s)
#0 14.52 === RUN   TestConvertSpecToRestoreFromSnapshotInput
#0 14.52 --- PASS: TestConvertSpecToRestoreFromSnapshotInput (0.00s)
#0 14.52 PASS
#0 14.52 ok     github.com/sorenmat/k8s-rds/rds 0.005s
#0 14.52 FAIL
#0 14.59 make: *** [Makefile:17: test] Error 2
------
Dockerfile:10
--------------------
   8 |
   9 |     COPY . .
  10 | >>> RUN make test lint build
  11 |
  12 |     FROM alpine
--------------------
error: failed to solve: process "/bin/sh -c make test lint build" did not complete successfully: exit code: 2
make: *** [Makefile:24: docker-build] Error 1

weird, it's working fine locally

just made one message regarding requiring go 1.18 disappear, but still that test is failing and it appears to be related to wrong number of arguments.
liskl@DESKTOP-D9PIEKG:~/k8s-rds$ CGO_ENABLED=0 go test ./... -v
# github.com/sorenmat/k8s-rds [github.com/sorenmat/k8s-rds.test]
./main_test.go:53:48: not enough arguments in call to excluded
        have (*crd.Database, []string, []string)
        want (string, string, []string, []string)
FAIL    github.com/sorenmat/k8s-rds [build failed]

can you give the output of make docker-tag from your end @amazzalel-habib ?

just made one message regarding requiring go 1.18 disappear, but still that test is failing and it appears to be related to wrong number of arguments.
liskl@DESKTOP-D9PIEKG:~/k8s-rds$ CGO_ENABLED=0 go test ./... -v
# github.com/sorenmat/k8s-rds [github.com/sorenmat/k8s-rds.test]
./main_test.go:53:48: not enough arguments in call to excluded
        have (*crd.Database, []string, []string)
        want (string, string, []string, []string)
FAIL    github.com/sorenmat/k8s-rds [build failed]

could it be the cache? because if you check main_test.go:53:48, it has the signature (string, string, []string, []string) not (*crd.Database, []string, []string)

Line 53 shows
                        if actual := excluded(test.db, test.exclNS, test.inclNS); actual != expected {

That is only three arguments. On your branch.

image builds now will test functionality shortly.

so created and deleting the database cluster object works, with no instances inside as expected with the CRUD, but there is a ModifyCluster happening right after as decribed here when the cluster is still in the creating state
2022/09/16 16:55:04 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:55:04 services "pgsql-cluster-1" not found
2022/09/16 16:55:04 delete of service pgsql-cluster-1 failed in namespace rds-operator: services "pgsql-cluster-1" not found
2022/09/16 16:55:04 Deletion of db cluster pgsql-cluster-1 done
2022/09/16 16:55:23 New resource version of the DB cluster pgsql-cluster-1 is 736737428
2022/09/16 16:55:23 trying to get kubectl
2022/09/16 16:55:23 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:55:23 Found node with ID: i-04b6432d9ebeb2e3b in region us-west-2
2022/09/16 16:55:24 Taking subnets from node i-04b6432d9ebeb2e3b
2022/09/16 16:55:24 trying to describe instance
2022/09/16 16:55:27 trying to get subnets
2022/09/16 16:55:27 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:55:27 Found VPC vpc-0fdfd77cda33f681c will search for subnet in that VPC
2022/09/16 16:55:27 Found the follwing subnets:
2022/09/16 16:55:27 subnet-0312bd982d89f088e
2022/09/16 16:55:27 subnet-0791eebb4d3e05f81
2022/09/16 16:55:27 subnet-0d97c4fef202c5aae
2022/09/16 16:55:27 subnet-07982cc3f1c41d778
2022/09/16 16:55:27 subnet-067c1fbeb25342eb7
2022/09/16 16:55:27 subnet-091d17ae01b066f88
2022/09/16 16:55:27 trying to get security groups
2022/09/16 16:55:27 Taking security groups from node i-04b6432d9ebeb2e3b
2022/09/16 16:55:27 trying to describe instance
2022/09/16 16:55:27 got instance response
2022/09/16 16:55:27 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:55:27 Found the follwing security groups:
2022/09/16 16:55:27 sg-0f3ce7d47d2175a68
2022/09/16 16:55:27 Trying to find the correct subnets
2022/09/16 16:55:28 Subnets: [subnet-0312bd982d89f088e subnet-0791eebb4d3e05f81 subnet-0d97c4fef202c5aae subnet-07982cc3f1c41d778 subnet-067c1fbeb25342eb7 subnet-091d17ae01b066f88]
2022/09/16 16:55:28 Moving on seems like db-subnetgroup-vpc-0fdfd77cda33f681c exsits
2022/09/16 16:55:28 getting secret: Name: rds-credentials Key: password
2022/09/16 16:55:28 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:55:28 WARNING: Not Adding annotation KV to tags: kubectl.kubernetes.io/last-applied-configuration {"apiVersion":"k8s-rds.io/v1","kind":"DBCluster","metadata":{"annotations":{},"name":"pgsql-cluster-1","namespace":"rds-operator"},"spec":{"BackupRetentionPeriod":1,"DBClusterIdentifier":"pgsql-cluster-junk","DeletionProtection":false,"Engine":"aurora-postgresql","EngineVersion":"14.3","MasterUserPassword":{"key":"password","name":"rds-credentials"},"MasterUsername":"postgres","MultiAZ":false,"Port":5432,"Provider":"aws","ServerlessV2ScalingConfiguration":{"MaxCapacity":11,"MinCapacity":1},"SkipFinalSnapshot":true,"Tags":"key=value,key1=value1"}}
2022/09/16 16:55:28 Trying to find db cluster pgsql-cluster-junk
2022/09/16 16:55:28 DB Cluster instance 0xc00081a660 not found trying to create it
2022/09/16 16:55:30 Waiting for db cluster instance with ID pgsql-cluster-junk  to become available
2022/09/16 16:56:15 DB cluster pgsql-cluster-junk is now available
2022/09/16 16:56:15 Creating service 'pgsql-cluster-1' for pgsql-cluster-junk.cluster-c2xf8ljleb1r.us-west-2.rds.amazonaws.com
2022/09/16 16:56:15 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:56:15 New resource version of the DB cluster pgsql-cluster-1 is 736737428
2022/09/16 16:56:15 Creation of DB cluster pgsql-cluster-1 done
2022/09/16 16:56:15 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:56:16 Found node with ID: i-04b6432d9ebeb2e3b in region us-west-2
2022/09/16 16:56:16 Taking subnets from node i-04b6432d9ebeb2e3b
2022/09/16 16:56:16 trying to describe instance
2022/09/16 16:56:19 trying to get subnets
2022/09/16 16:56:19 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:56:19 Found VPC vpc-0fdfd77cda33f681c will search for subnet in that VPC
2022/09/16 16:56:19 Found the follwing subnets:
2022/09/16 16:56:19 subnet-0312bd982d89f088e
2022/09/16 16:56:19 subnet-0791eebb4d3e05f81
2022/09/16 16:56:19 subnet-0d97c4fef202c5aae
2022/09/16 16:56:19 subnet-07982cc3f1c41d778
2022/09/16 16:56:19 subnet-067c1fbeb25342eb7
2022/09/16 16:56:19 subnet-091d17ae01b066f88
2022/09/16 16:56:19 trying to get security groups
2022/09/16 16:56:19 Taking security groups from node i-04b6432d9ebeb2e3b
2022/09/16 16:56:19 trying to describe instance
2022/09/16 16:56:19 got instance response
2022/09/16 16:56:19 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:56:19 Found the follwing security groups:
2022/09/16 16:56:19 sg-0f3ce7d47d2175a68
2022/09/16 16:56:19 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:56:19 Trying to find db cluster instance pgsql-cluster-junk to update
2022/09/16 16:56:20 DB cluster instance pgsql-cluster-junk found trying to update it
2022/09/16 16:56:20 Updating DB cluster failed: ModifyDBCluster:operation error RDS: ModifyDBCluster, https response error StatusCode: 400, RequestID: 98507bbf-ae02-4ab7-87c7-e4617f9edf5a, InvalidDBClusterStateFault: Cannot modify engine version without a healthy primary instance in DB cluster: pgsql-cluster-junk
2022/09/16 16:56:20 Updating db cluster: UpdateDBCluster:ModifyDBCluster: operation error RDS: ModifyDBCluster, https response error StatusCode: 400, RequestID: 98507bbf-ae02-4ab7-87c7-e4617f9edf5a, InvalidDBClusterStateFault: Cannot modify engine version without a healthy primary instance in DB cluster: pgsql-cluster-junk
2022/09/16 16:56:20 db cluster update failed: ModifyDBCluster: operation error RDS: ModifyDBCluster, https response error StatusCode: 400, RequestID: 98507bbf-ae02-4ab7-87c7-e4617f9edf5a, InvalidDBClusterStateFault: Cannot modify engine version without a healthy primary instance in DB cluster: pgsql-cluster-junk
2022/09/16 16:56:20 New resource version of the DB cluster pgsql-cluster-1 is 736737428

so created and deleting the database cluster object works, with no instances inside as expected with the CRUD, but there is a ModifyCluster happening right after as decribed here when the cluster is still in the creating state.
maybe we should hold off on ModifyDBCluster calls until there is an healthy instance and the cluster State is Available? what are your thoughts @amazzalel-habib
2022/09/16 16:55:04 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:55:04 services "pgsql-cluster-1" not found
2022/09/16 16:55:04 delete of service pgsql-cluster-1 failed in namespace rds-operator: services "pgsql-cluster-1" not found
2022/09/16 16:55:04 Deletion of db cluster pgsql-cluster-1 done
2022/09/16 16:55:23 New resource version of the DB cluster pgsql-cluster-1 is 736737428
2022/09/16 16:55:23 trying to get kubectl
2022/09/16 16:55:23 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:55:23 Found node with ID: i-04b6432d9ebeb2e3b in region us-west-2
2022/09/16 16:55:24 Taking subnets from node i-04b6432d9ebeb2e3b
2022/09/16 16:55:24 trying to describe instance
2022/09/16 16:55:27 trying to get subnets
2022/09/16 16:55:27 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:55:27 Found VPC vpc-0fdfd77cda33f681c will search for subnet in that VPC
2022/09/16 16:55:27 Found the follwing subnets:
2022/09/16 16:55:27 subnet-0312bd982d89f088e
2022/09/16 16:55:27 subnet-0791eebb4d3e05f81
2022/09/16 16:55:27 subnet-0d97c4fef202c5aae
2022/09/16 16:55:27 subnet-07982cc3f1c41d778
2022/09/16 16:55:27 subnet-067c1fbeb25342eb7
2022/09/16 16:55:27 subnet-091d17ae01b066f88
2022/09/16 16:55:27 trying to get security groups
2022/09/16 16:55:27 Taking security groups from node i-04b6432d9ebeb2e3b
2022/09/16 16:55:27 trying to describe instance
2022/09/16 16:55:27 got instance response
2022/09/16 16:55:27 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:55:27 Found the follwing security groups:
2022/09/16 16:55:27 sg-0f3ce7d47d2175a68
2022/09/16 16:55:27 Trying to find the correct subnets
2022/09/16 16:55:28 Subnets: [subnet-0312bd982d89f088e subnet-0791eebb4d3e05f81 subnet-0d97c4fef202c5aae subnet-07982cc3f1c41d778 subnet-067c1fbeb25342eb7 subnet-091d17ae01b066f88]
2022/09/16 16:55:28 Moving on seems like db-subnetgroup-vpc-0fdfd77cda33f681c exsits
2022/09/16 16:55:28 getting secret: Name: rds-credentials Key: password
2022/09/16 16:55:28 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:55:28 WARNING: Not Adding annotation KV to tags: kubectl.kubernetes.io/last-applied-configuration {"apiVersion":"k8s-rds.io/v1","kind":"DBCluster","metadata":{"annotations":{},"name":"pgsql-cluster-1","namespace":"rds-operator"},"spec":{"BackupRetentionPeriod":1,"DBClusterIdentifier":"pgsql-cluster-junk","DeletionProtection":false,"Engine":"aurora-postgresql","EngineVersion":"14.3","MasterUserPassword":{"key":"password","name":"rds-credentials"},"MasterUsername":"postgres","MultiAZ":false,"Port":5432,"Provider":"aws","ServerlessV2ScalingConfiguration":{"MaxCapacity":11,"MinCapacity":1},"SkipFinalSnapshot":true,"Tags":"key=value,key1=value1"}}
2022/09/16 16:55:28 Trying to find db cluster pgsql-cluster-junk
2022/09/16 16:55:28 DB Cluster instance 0xc00081a660 not found trying to create it
2022/09/16 16:55:30 Waiting for db cluster instance with ID pgsql-cluster-junk  to become available
2022/09/16 16:56:15 DB cluster pgsql-cluster-junk is now available
2022/09/16 16:56:15 Creating service 'pgsql-cluster-1' for pgsql-cluster-junk.cluster-c2xf8ljleb1r.us-west-2.rds.amazonaws.com
2022/09/16 16:56:15 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:56:15 New resource version of the DB cluster pgsql-cluster-1 is 736737428
2022/09/16 16:56:15 Creation of DB cluster pgsql-cluster-1 done
2022/09/16 16:56:15 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:56:16 Found node with ID: i-04b6432d9ebeb2e3b in region us-west-2
2022/09/16 16:56:16 Taking subnets from node i-04b6432d9ebeb2e3b
2022/09/16 16:56:16 trying to describe instance
2022/09/16 16:56:19 trying to get subnets
2022/09/16 16:56:19 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:56:19 Found VPC vpc-0fdfd77cda33f681c will search for subnet in that VPC
2022/09/16 16:56:19 Found the follwing subnets:
2022/09/16 16:56:19 subnet-0312bd982d89f088e
2022/09/16 16:56:19 subnet-0791eebb4d3e05f81
2022/09/16 16:56:19 subnet-0d97c4fef202c5aae
2022/09/16 16:56:19 subnet-07982cc3f1c41d778
2022/09/16 16:56:19 subnet-067c1fbeb25342eb7
2022/09/16 16:56:19 subnet-091d17ae01b066f88
2022/09/16 16:56:19 trying to get security groups
2022/09/16 16:56:19 Taking security groups from node i-04b6432d9ebeb2e3b
2022/09/16 16:56:19 trying to describe instance
2022/09/16 16:56:19 got instance response
2022/09/16 16:56:19 Security groupid:  sg-0f3ce7d47d2175a68
2022/09/16 16:56:19 Found the follwing security groups:
2022/09/16 16:56:19 sg-0f3ce7d47d2175a68
2022/09/16 16:56:19 Seems like we are running in a Kubernetes cluster!!
2022/09/16 16:56:19 Trying to find db cluster instance pgsql-cluster-junk to update
2022/09/16 16:56:20 DB cluster instance pgsql-cluster-junk found trying to update it
2022/09/16 16:56:20 Updating DB cluster failed: ModifyDBCluster:operation error RDS: ModifyDBCluster, https response error StatusCode: 400, RequestID: 98507bbf-ae02-4ab7-87c7-e4617f9edf5a, InvalidDBClusterStateFault: Cannot modify engine version without a healthy primary instance in DB cluster: pgsql-cluster-junk
2022/09/16 16:56:20 Updating db cluster: UpdateDBCluster:ModifyDBCluster: operation error RDS: ModifyDBCluster, https response error StatusCode: 400, RequestID: 98507bbf-ae02-4ab7-87c7-e4617f9edf5a, InvalidDBClusterStateFault: Cannot modify engine version without a healthy primary instance in DB cluster: pgsql-cluster-junk
2022/09/16 16:56:20 db cluster update failed: ModifyDBCluster: operation error RDS: ModifyDBCluster, https response error StatusCode: 400, RequestID: 98507bbf-ae02-4ab7-87c7-e4617f9edf5a, InvalidDBClusterStateFault: Cannot modify engine version without a healthy primary instance in DB cluster: pgsql-cluster-junk
2022/09/16 16:56:20 New resource version of the DB cluster pgsql-cluster-1 is 736737428

that's weird, I'll check it and get back to you

seeing an error on build
#0 23.21 rds/rds_provider.go:449:27: Error return value is not checked (errcheck)
#0 23.21                                        waitForDBAvailability(ctx, rdsdb.DBClusterMembers[0].DBInstanceIdentifier, rdsCli)
#0 23.21                                                             ^
#0 23.28 make: *** [Makefile:19: lint] Error 1

seeing an error on build
#0 23.21 rds/rds_provider.go:449:27: Error return value is not checked (errcheck)
#0 23.21                                        waitForDBAvailability(ctx, rdsdb.DBClusterMembers[0].DBInstanceIdentifier, rdsCli)
#0 23.21                                                             ^
#0 23.28 make: *** [Makefile:19: lint] Error 1

Fixed

the deletion of a cluster using only the DBCluster object kubectl delete dbcluster.k8s-rds.io pgsql-cluster-1 resulted in the deletion of the instance but failed to finally delete the cluster itself


Documentation structure draft

This is a draft for the RTWG documentation structure:

Guides

Real-time Operating System setup

Real Time Linux

Build a kernel using Preempt-RT
Configure kernel
Additional settings (ulimits, interrupt threads, network, cgroups, ....)

Configuring VXworks => https://github.com/Wind-River/vxworks7-layer-for-ros2
Configuring QNX => https://github.com/ros-realtime/community/issues/5

How to configure rmw implementations

FastDDS
CycloneDDS
ConnextMicro/Pro
Iceoryx
...

How to run a ROS 2 real-time application
Best practices

ROS 2 real-time buildfarm

Test environment
How to run the tests locally

Concepts

Memory management
Exceptions
Logging
...

External resources (articles, guides, ....)
Roadmap
Contributing
Feature ideas
Subprojects
Related projects

I added an additional item to configure DDS and other rmw implementations. This would include vendor specific guidelines, such as configuring XML QoS profiles, using static memory support, memory pools, thread priority, etc.


build: add postcss loader

Added postcss-loader to project - issue #95

Codecov Report

Merging #106 into development will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           development     #106   +/-   ##
============================================
  Coverage        91.78%   91.78%
============================================
  Files               22       22
  Lines              487      487
============================================
  Hits               447      447
  Misses              40       40

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ce1592...b41defa. Read the comment docs.

@prasanna1211 Another advantage ( by what I understood ) about not using css-loader ( since now postcss-loader is used ) is that we won't need to use @import statements in scss files.

@rishichawda will look at it and let you know. Trying POST CSS for first time.

@rishichawda I guess we need to drop this idea. The main reason for choosing POSTCSS was that it gives cross browser compatibility. But now I realize even SASS can do this. #86 will solve this.

Something like https://stackoverflow.com/questions/19351530/cross-browser-mixin-using-sass

@rishichawda I guess we need to drop this idea. The main reason for choosing POSTCSS was that it gives cross browser compatibility. But now I realize even SASS can do this. #86 will solve this.

Yes. Only advantage was that we could automate a lot of things using postcss and reduce how much code we need to write for each component.

Something like https://stackoverflow.com/questions/19351530/cross-browser-mixin-using-sass

This looks similar to what I did for flex.


SAMZA-2397: Updating gauge-val function on newGauge on same metric name

Updating gauge-val function on newGauge on same metric name.
Symptom: RocksDB-specific metric value like estimate-table-readers-mem, cur-size-active-mem-table, etc, are not being emitted after Samza 1.1; only a 0 value is emitted instead.
Cause: Samza 1.1 uses an optimization where a RocksDBStore is created first in BulkLoad mode for optimizing bootstrap write, after bootstrapping is complete the store is closed (to trigger compaction), and a new store is created in ReadWrite mode. The rocksdb metrics are calculated using a function which checks if the store is open (see https://github.com/apache/samza/pull/588 by @xinyuiscool , SAMZA-1778), otherwise it emits 0.
See RocksdbKeyValueStore#95.
However, because the metric-gauge-object is not updated after the store is recreated, the gauge keeps checking the first (closed) store and emits 0.
Fix: Added logic that ensures that the second store's metric overrides the previous gauge for the same name, and logs a message.
Tests: Added unit test, test with a samza job that emits non-zero RocksDB metrics.

@bkonold can you review?


⚠ Bofrost.lu has degraded performance

In 5a27bf1, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2125 ms

Resolved: Bofrost.lu performance has improved in 49e9969 after 6 minutes.


add RTL support complete

This repository is merely an automated translation of the upstream repository.
All substantive, non-Sass-specific changes should be submitted upstream instead:
https://github.com/twbs/bootstrap


Styling for #729 indicator plan table

I've changed the strategy for full-width (usually tabular) pages like this and the IPTT. See https://github.com/mercycorps/TolaActivity/issues/729#issuecomment-459483876

that's a good point. Jenny and I ran through these scrolling issues with the IPTT last summer (but even trickier bc of the collapsing filter bar). This is our go-to pattern for "responsive" tables and FWIW this is also the generic bootstrap solution


python312Packages.oci: 2.128.2 -> 2.129.4

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.oci is: Oracle Cloud Infrastructure Python SDK
meta.homepage for python312Packages.oci is: https://github.com/oracle/oci-python-sdk
meta.changelog for python312Packages.oci is: https://github.com/oracle/oci-python-sdk/blob/v2.129.3/CHANGELOG.rst
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.129.3 with grep in /nix/store/960vmj109i7mprlyss3yccwcfs9sgsgk-python3.12-oci-2.129.3
found 2.129.3 in filename of file in /nix/store/960vmj109i7mprlyss3yccwcfs9sgsgk-python3.12-oci-2.129.3

Rebuild report (if merged into master) (click to expand)

8 total rebuild path(s)

8 package rebuild(s)

First fifty rebuilds by attrpath

oci-cli
python311Packages.certbot-dns-ovh
python311Packages.dns-lexicon
python311Packages.oci
python311Packages.ocifs
python312Packages.oci
python312Packages.ocifs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/960vmj109i7mprlyss3yccwcfs9sgsgk-python3.12-oci-2.129.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.oci https://github.com/r-ryantm/nixpkgs/archive/fa56d8e4b2428cce9b056b7affcdd62995decf5c.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/fa56d8e4b2428cce9b056b7affcdd62995decf5c#python312Packages.oci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/960vmj109i7mprlyss3yccwcfs9sgsgk-python3.12-oci-2.129.3
ls -la /nix/store/960vmj109i7mprlyss3yccwcfs9sgsgk-python3.12-oci-2.129.3/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

14 packages built:

oci-cli
oci-cli.dist
python311Packages.certbot-dns-ovh
python311Packages.certbot-dns-ovh.dist
python311Packages.dns-lexicon
python311Packages.dns-lexicon.dist
python311Packages.oci
python311Packages.oci.dist
python311Packages.ocifs
python311Packages.ocifs.dist
python312Packages.oci
python312Packages.oci.dist
python312Packages.ocifs
python312Packages.ocifs.dist

Maintainer pings
cc @ilian for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.oci is: Oracle Cloud Infrastructure Python SDK
meta.homepage for python312Packages.oci is: https://github.com/oracle/oci-python-sdk
meta.changelog for python312Packages.oci is: https://github.com/oracle/oci-python-sdk/blob/v2.129.4/CHANGELOG.rst
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.129.4 with grep in /nix/store/q5q1bnnqkw1mdh42syg4x3v8hl65piyl-python3.12-oci-2.129.4
found 2.129.4 in filename of file in /nix/store/q5q1bnnqkw1mdh42syg4x3v8hl65piyl-python3.12-oci-2.129.4

Rebuild report (if merged into master) (click to expand)

8 total rebuild path(s)

8 package rebuild(s)

First fifty rebuilds by attrpath

oci-cli
python311Packages.certbot-dns-ovh
python311Packages.dns-lexicon
python311Packages.oci
python311Packages.ocifs
python312Packages.oci
python312Packages.ocifs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/q5q1bnnqkw1mdh42syg4x3v8hl65piyl-python3.12-oci-2.129.4 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.oci https://github.com/r-ryantm/nixpkgs/archive/ff6c501ca539bcbdb8ce16752e86dccc424e748f.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/ff6c501ca539bcbdb8ce16752e86dccc424e748f#python312Packages.oci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/q5q1bnnqkw1mdh42syg4x3v8hl65piyl-python3.12-oci-2.129.4
ls -la /nix/store/q5q1bnnqkw1mdh42syg4x3v8hl65piyl-python3.12-oci-2.129.4/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

14 packages built:

oci-cli
oci-cli.dist
python311Packages.certbot-dns-ovh
python311Packages.certbot-dns-ovh.dist
python311Packages.dns-lexicon
python311Packages.dns-lexicon.dist
python311Packages.oci
python311Packages.oci.dist
python311Packages.ocifs
python311Packages.ocifs.dist
python312Packages.oci
python312Packages.oci.dist
python312Packages.ocifs
python312Packages.ocifs.dist

Maintainer pings
cc @ilian for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.oci is: Oracle Cloud Infrastructure Python SDK
meta.homepage for python312Packages.oci is: https://github.com/oracle/oci-python-sdk
meta.changelog for python312Packages.oci is: https://github.com/oracle/oci-python-sdk/blob/v2.131.0/CHANGELOG.rst
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.131.0 with grep in /nix/store/frnj61chjvrppvwr6qszcak6xqn7d48j-python3.12-oci-2.131.0
found 2.131.0 in filename of file in /nix/store/frnj61chjvrppvwr6qszcak6xqn7d48j-python3.12-oci-2.131.0

Rebuild report (if merged into master) (click to expand)

12 total rebuild path(s)

12 package rebuild(s)

First fifty rebuilds by attrpath

certbot-full
lexicon
oci-cli
python311Packages.certbot-dns-ovh
python311Packages.dns-lexicon
python311Packages.oci
python311Packages.ocifs
python312Packages.certbot-dns-ovh
python312Packages.dns-lexicon
python312Packages.oci
python312Packages.ocifs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/frnj61chjvrppvwr6qszcak6xqn7d48j-python3.12-oci-2.131.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.oci https://github.com/r-ryantm/nixpkgs/archive/b89df26f9b63b1d1a38bf157d61dd0e9caaa80af.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/b89df26f9b63b1d1a38bf157d61dd0e9caaa80af#python312Packages.oci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/frnj61chjvrppvwr6qszcak6xqn7d48j-python3.12-oci-2.131.0
ls -la /nix/store/frnj61chjvrppvwr6qszcak6xqn7d48j-python3.12-oci-2.131.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

5 packages failed to build:

certbot-full
python311Packages.certbot-dns-ovh
python311Packages.certbot-dns-ovh.dist
python312Packages.certbot-dns-ovh
python312Packages.certbot-dns-ovh.dist

14 packages built:

lexicon (python312Packages.dns-lexicon)
lexicon.dist (python312Packages.dns-lexicon.dist)
oci-cli
oci-cli.dist
python311Packages.dns-lexicon
python311Packages.dns-lexicon.dist
python311Packages.oci
python311Packages.oci.dist
python311Packages.ocifs
python311Packages.ocifs.dist
python312Packages.oci
python312Packages.oci.dist
python312Packages.ocifs
python312Packages.ocifs.dist

Maintainer pings
cc @ilian for testing.

Add a :+1: reaction to pull requests you find important.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.oci is: Oracle Cloud Infrastructure Python SDK
meta.homepage for python312Packages.oci is: https://github.com/oracle/oci-python-sdk
meta.changelog for python312Packages.oci is: https://github.com/oracle/oci-python-sdk/blob/v2.131.1/CHANGELOG.rst
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.131.1 with grep in /nix/store/22k0z7y4ak7knbdzbjqjxf96l2x7absc-python3.12-oci-2.131.1
found 2.131.1 in filename of file in /nix/store/22k0z7y4ak7knbdzbjqjxf96l2x7absc-python3.12-oci-2.131.1

Rebuild report (if merged into master) (click to expand)

12 total rebuild path(s)

12 package rebuild(s)

First fifty rebuilds by attrpath

certbot-full
lexicon
oci-cli
python311Packages.certbot-dns-ovh
python311Packages.dns-lexicon
python311Packages.oci
python311Packages.ocifs
python312Packages.certbot-dns-ovh
python312Packages.dns-lexicon
python312Packages.oci
python312Packages.ocifs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/22k0z7y4ak7knbdzbjqjxf96l2x7absc-python3.12-oci-2.131.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.oci https://github.com/r-ryantm/nixpkgs/archive/863e3a0c00815b5f6c54cf8ed3e178ec721f5860.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/863e3a0c00815b5f6c54cf8ed3e178ec721f5860#python312Packages.oci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/22k0z7y4ak7knbdzbjqjxf96l2x7absc-python3.12-oci-2.131.1
ls -la /nix/store/22k0z7y4ak7knbdzbjqjxf96l2x7absc-python3.12-oci-2.131.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

19 packages built:

certbot-full
lexicon (python312Packages.dns-lexicon)
lexicon.dist (python312Packages.dns-lexicon.dist)
oci-cli
oci-cli.dist
python311Packages.certbot-dns-ovh
python311Packages.certbot-dns-ovh.dist
python311Packages.dns-lexicon
python311Packages.dns-lexicon.dist
python311Packages.oci
python311Packages.oci.dist
python311Packages.ocifs
python311Packages.ocifs.dist
python312Packages.certbot-dns-ovh
python312Packages.certbot-dns-ovh.dist
python312Packages.oci
python312Packages.oci.dist
python312Packages.ocifs
python312Packages.ocifs.dist

Maintainer pings
cc @ilian for testing.

Add a :+1: reaction to pull requests you find important.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.oci is: Oracle Cloud Infrastructure Python SDK
meta.homepage for python312Packages.oci is: https://github.com/oracle/oci-python-sdk
meta.changelog for python312Packages.oci is: https://github.com/oracle/oci-python-sdk/blob/v2.132.0/CHANGELOG.rst
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.132.0 with grep in /nix/store/bzs5m3xzvwsyjsg9i8mlswwmwj55k642-python3.12-oci-2.132.0
found 2.132.0 in filename of file in /nix/store/bzs5m3xzvwsyjsg9i8mlswwmwj55k642-python3.12-oci-2.132.0

Rebuild report (if merged into master) (click to expand)

12 total rebuild path(s)

12 package rebuild(s)

First fifty rebuilds by attrpath

certbot-full
lexicon
oci-cli
python311Packages.certbot-dns-ovh
python311Packages.dns-lexicon
python311Packages.oci
python311Packages.ocifs
python312Packages.certbot-dns-ovh
python312Packages.dns-lexicon
python312Packages.oci
python312Packages.ocifs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/bzs5m3xzvwsyjsg9i8mlswwmwj55k642-python3.12-oci-2.132.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.oci https://github.com/r-ryantm/nixpkgs/archive/28ebe38768d709b084fa5c3233a32510f11aa933.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/28ebe38768d709b084fa5c3233a32510f11aa933#python312Packages.oci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/bzs5m3xzvwsyjsg9i8mlswwmwj55k642-python3.12-oci-2.132.0
ls -la /nix/store/bzs5m3xzvwsyjsg9i8mlswwmwj55k642-python3.12-oci-2.132.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

19 packages built:

certbot-full
lexicon (python312Packages.dns-lexicon)
lexicon.dist (python312Packages.dns-lexicon.dist)
oci-cli
oci-cli.dist
python311Packages.certbot-dns-ovh
python311Packages.certbot-dns-ovh.dist
python311Packages.dns-lexicon
python311Packages.dns-lexicon.dist
python311Packages.oci
python311Packages.oci.dist
python311Packages.ocifs
python311Packages.ocifs.dist
python312Packages.certbot-dns-ovh
python312Packages.certbot-dns-ovh.dist
python312Packages.oci
python312Packages.oci.dist
python312Packages.ocifs
python312Packages.ocifs.dist

Maintainer pings
cc @ilian for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.oci is: Oracle Cloud Infrastructure Python SDK
meta.homepage for python312Packages.oci is: https://github.com/oracle/oci-python-sdk
meta.changelog for python312Packages.oci is: https://github.com/oracle/oci-python-sdk/blob/v2.133.0/CHANGELOG.rst
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.133.0 with grep in /nix/store/lfbinvagxpcdb1923dh7fzhikk4g6ka7-python3.12-oci-2.133.0
found 2.133.0 in filename of file in /nix/store/lfbinvagxpcdb1923dh7fzhikk4g6ka7-python3.12-oci-2.133.0

Rebuild report (if merged into master) (click to expand)

12 total rebuild path(s)

12 package rebuild(s)

First fifty rebuilds by attrpath

certbot-full
lexicon
oci-cli
python311Packages.certbot-dns-ovh
python311Packages.dns-lexicon
python311Packages.oci
python311Packages.ocifs
python312Packages.certbot-dns-ovh
python312Packages.dns-lexicon
python312Packages.oci
python312Packages.ocifs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/lfbinvagxpcdb1923dh7fzhikk4g6ka7-python3.12-oci-2.133.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.oci https://github.com/r-ryantm/nixpkgs/archive/7ba2551d0f5ffc3053f61338d5e21a98972732d7.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/7ba2551d0f5ffc3053f61338d5e21a98972732d7#python312Packages.oci

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/lfbinvagxpcdb1923dh7fzhikk4g6ka7-python3.12-oci-2.133.0
ls -la /nix/store/lfbinvagxpcdb1923dh7fzhikk4g6ka7-python3.12-oci-2.133.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

19 packages built:

certbot-full
lexicon (python312Packages.dns-lexicon)
lexicon.dist (python312Packages.dns-lexicon.dist)
oci-cli
oci-cli.dist
python311Packages.certbot-dns-ovh
python311Packages.certbot-dns-ovh.dist
python311Packages.dns-lexicon
python311Packages.dns-lexicon.dist
python311Packages.oci
python311Packages.oci.dist
python311Packages.ocifs
python311Packages.ocifs.dist
python312Packages.certbot-dns-ovh
python312Packages.certbot-dns-ovh.dist
python312Packages.oci
python312Packages.oci.dist
python312Packages.ocifs
python312Packages.ocifs.dist

Maintainer pings
cc @ilian for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.


class SUBSTANCE

We never spoke about this class. Could you advise whether the following examples are right? :
- pellets of <ENAMEX type="ARTIFACT">Zyklon-B</ENAMEX> were dropped into the chambers
 through vents in the side walls, releasing toxic <ENAMEX type="SUBSTANCE">HCN</ENAMEX>,
or <ENAMEX type="SUBSTANCE">hydrogen cyanide</ENAMEX>

- <ENAMEX type="SUBSTANCE">gold</ENAMEX> fillings in their teeth

- limit the use of <ENAMEX type="SUBSTANCE">asbestos</ENAMEX> or <ENAMEX
type="PLANT">tobacco</ENAMEX>

Thanks!

I see nothing wrong with it. I think they are correct: Zyklon B seems to be more an artifact, as it's the combination of different substances.


CodeBlock and Draft

Just want to know where to set this "process.env.HLX_TKN"

Just want to know where to set this "process.env.HLX_TKN"

If you are running locally you will need to add HLX_TKN as an environment variable with the token value to your terminal/command prompt profile.  If you are running it in GitHub Actions/CI you will need to add a Repository Secret named HLX_TKN with the value.  @ancysr will have the token value.


Homebrew distribution doesn't get tagged correctly when installing from HEAD

Description of the problem or steps to reproduce
When installing micro from homebrew, using the --HEAD flag (installing from latest off the "master" branch of this repo). The version number won't get tagged as the git clone is shallow, not complete. Thus none of the tags will get matched and will be installed as 0.0.0-unknown, rather the correct version number 1.1.4-dev.55. I don't think this is a big deal outside of the plugin compatibility checks, but it still impacts distribution. I don't know if other package managers allow this sort of installation method but it might be a problem there too.

It works fine for me. I used brew install --HEAD micro and I got the right version (1.1.4-dev.55). Perhaps you were seeing your GOPATH version instead of checking the binary in /usr/local/bin?

This is what happens when I install it: https://gist.github.com/samdmarshall/aaaba3b4d90b68a5db8892da3a8939ae

i'm running the same version. I'll have to do some debugging with it tomorrow but the only tag i see in the checkout that homebrew does is "nightly".

Could you please double check this on your end? I replicated the commands that homebrew performs and I still am unable to get a version number as part installing from --HEAD.

I'm closing this as this got fixed on the homebrew side with a formula change. https://github.com/Homebrew/homebrew-core/pull/8732


Joe/yorickvp/ci/joe/lang 218 investigate tps performance degradation

This PR:

Adds mock_cog_triton/, which includes a mocked cog-triton predict.py that emits tokens at a specified rate. This is useful for performance testing cog. This new directory also includes a version of test_perf.py that works with the mock server and emits useful information, such as server-side (within predict.py) vs. client-side performance metrics

Updates our Triton inference implementation to use /ensemble/ instead of /tensorrt_llm_bls/ as the request entrypoint. We do this because BLS performance is unreliable and seems to randomly introduce substantial latency to requests. To do this, this PR:

Move token accumulation and decoding into predict.py
Updates the preproc model.py in triton_templates and triton_model_repo so that it returns output_ids instead of strings
Updates output type expectations in template and model repo configs for ensemble and preprocessor.

Adds an envar option LOG_PERFORMANCE_METRICS such that, if True, server-side performance metrics will be logged after all tokens are generated.

Updates scripts/test_perf.py to parse request logs, extract optional server-side metrics, and include them in the generated performance report.

Feel free to merge!