Less build machines

I at least see no benefit to build same artifacts on two different operation systems. It's produced only more load on CI.
I choose Ubuntu, because I observed a faster build time

The idea is to catch build errors all environments, because MacOs has different version of sed and possibly other commandline utilities compared to Ubuntu ('sed -i path' does not work, you need to use 'sed -i.tmp path')

I see, you want to test local build with macOS too.
I'm only interested in binary build artifact, and I'm not interested which os build it.
The main motivation to do here some pull requests was this: I'm not able to build on my macOS, so I grab the artifacts from CI
Do you see what's wrong here ? I'm not, because I have a $NDK_ROOT

![image](https://user-images.githubusercontent.com/3314607/93694847-ed72e580-fb10-11ea-901c-4faae0b99836.png)


Add note on callback render efficiency to react usage docs

Refer to #1207 for more context.

It would be more correct to say:

There are two problems in this example: function allocations on every render, and the fact that props are shallowly unequal because of that.
Shallow inequality of props is not a problem by itself, but it makes optimizations harder because you can no longer use PureRenderMixin or shallowCompare to optimize components you know to be pure.
If you wrap component into connect() from React Redux and bind the action creators using its second argument, it will make sure to only bind them when necessary, and implement a strict shouldComponentUpdate() on the wrapper component.

Right now it's a bit unclear in this PR how these several factors come into play so I would appreciate a clarification.

Closing in favor of #1285


Avoid deletion of production storage

Note: If your issue/bug is regarding the AWS Amplify Console service, please log it in the
Amplify Console GitHub Issue Tracker

Describe the bug
cli shows deletion of current s3 and creation of a new one for my current production environment. I have active users in production environment. Is there a way to manually avoid this conflict?
Amplify CLI Version
4.31.0
To Reproduce
I created a new devel environment and deleted an S3 trigger. In order to delete the s3 trigger I needed to delete the storage and added it again. When I went to prod environment, it showed to delete current storage and add a new one. The current storage has files that cannot be deleted or replaced. How do I avoid this conflict while still working on development of the app.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. Mac/Windows/Ubuntu]
macos
Node Version. You can use node -v to check the node version on your system
v12.13.1

Additional context
Add any other context about the problem here.
I am working on the development environment. Eventually I want to merge my changes but I am afraid to do so at this point. I also had a screen showing

just to add to the issue, my production app is based on the old graphql schema setup. The new code I m writing is in the development environment that I will push into production later

@rush86999 To remove the S3 trigger - you could've just run amplify update storage -> remove trigger instead of removing the storage resource completely. When you checkout from say 'dev' env to 'prod' env - the changes from dev - carry-froward to prod. To avoid that you can use the --restore flag when checking out prod - amplify env checkout prod --restore or after checking out prod, you can also run amplify env pull --restore.


Better logging and error tracing for GCS Client

My team and I are currently facing an issue where our edr monitor send-report is failing while attempting to upload our elementary report to a GCS bucket.
For context, we are executing the edr monitor send-report command from the elementary-data/run-elementary-action@v1 GH Action.
      - name: Run Elementary CI
        uses: elementary-data/run-elementary-action@v1
        with:
          warehouse-type: ${{ env.WAREHOUSE_TYPE }} # Type of warehouse to use (bigquery, snowflake, redshift, etc.)
          profiles-yml: ${{ secrets.ELEMENTARY_CI_PROFILES_YML }} # Content of ~/.dbt/profiles.yml, should have an `elementary` profile.
          bigquery-keyfile: ${{ secrets.OUR_JSON_KEYFILE }} # If using BigQuery, the content of its keyfile.
          gcs-keyfile: ${{ secrets.OUR_JSON_KEYFILE }} # If using BigQuery, the content of its keyfile.
          # Read from the test results table, generate the Elementary UI and send it to GCS:
          edr-command: |
            edr monitor --slack-token ${{ secrets.SLACK_TOKEN }} --slack-channel-name ${{ secrets.SLACK_CHANNEL_NAME }}
            edr monitor send-report --slack-token ${{ secrets.SLACK_TOKEN }} \
              --slack-channel-name ${{ secrets.SLACK_CHANNEL_NAME }} \
              --google-service-account-path /tmp/gcs_keyfile.json \
              --gcs-bucket-name gs://our-report-bucket \
              --update-bucket-website false
              #--gcs-bucket-name our-report-bucket
              # --bucket-file-path data-team/PR_${{ env.GH_PR_NUMBER }}_report.html

Note: We have tried to pass different variations of the GCS related parameters (i.e. different prefixes and paths) but still we're getting the same exception at the end of each action.
The error message that we receive is not very helpful and does not give us a lot of information to find the source of the issue:
2022-09-27 12:56:18,218 — elementary.clients.gcs.client — ERROR — Failed to upload report to GCS.

After digging into the code, I noticed that there is not a lot of logging in the GCS Client class:
https://github.com/elementary-data/elementary/blob/1a85cb8da955d99c78a50e044d92f229c11bc2b9/elementary/clients/gcs/client.py#L26-L38
We would like to add more verbose logging here to allow for more granular error traces.
I have opened a PR to add more verbose logging to the GCS Client class:
https://github.com/elementary-data/elementary/pull/331

Hi @seanglynn-thrive ! thanks so much for opening an issue and PR about this. I'll review the PR right now and we could merge it with our planned release tomorrow. If you are open to it we can also debug it together over a call or over Slack in our community

Hi @oravi,
Sure that sounds great! I will open a slack thread shortly on the elementary slack channel.


Update kubevirt storage provisioner on Wiki

Hi guys, this is not a bug, but there is no way to update the wiki.
Some links are broken and some apis are stable now without "beta".
These minor changes worked on CRC 2.0.1+bf3b1a6.
oc apply -f 'https://raw.githubusercontent.com/kubevirt/hostpath-provisioner/main/deploy/kubevirt-hostpath-security-constraints.yaml'

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: kubevirt-hostpath-provisioner
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kubevirt-hostpath-provisioner
provisioner: kubevirt.io/hostpath-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubevirt-hostpath-provisioner
subjects:
- kind: ServiceAccount
  name: kubevirt-hostpath-provisioner-admin
  namespace: kubevirt-hostpath-provisioner
roleRef:
  kind: ClusterRole
  name: kubevirt-hostpath-provisioner
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubevirt-hostpath-provisioner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]

  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubevirt-hostpath-provisioner-admin
  namespace: kubevirt-hostpath-provisioner
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kubevirt-hostpath-provisioner
  labels:
    k8s-app: kubevirt-hostpath-provisioner
  namespace: kubevirt-hostpath-provisioner
spec:
  selector:
    matchLabels:
      k8s-app: kubevirt-hostpath-provisioner
  template:
    metadata:
      labels:
        k8s-app: kubevirt-hostpath-provisioner
    spec:
      serviceAccountName: kubevirt-hostpath-provisioner-admin
      containers:
        - name: kubevirt-hostpath-provisioner
          image: quay.io/kubevirt/hostpath-provisioner
          imagePullPolicy: Always
          env:
            - name: USE_NAMING_PREFIX
              value: "false" # change to true, to have the name of the pvc be part of the directory
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: PV_DIR
              value: /var/hpvolumes
          volumeMounts:
            - name: pv-volume # root dir where your bind mounts will be on the node
              mountPath: /var/hpvolumes
              #nodeSelector:
              #- name: xxxxxx
      volumes:
        - name: pv-volume
          hostPath:
            path: /mnt/pv-data
EOF

Updated the wiki, thanks @hector-vido


Support of Ext.widget

I tried to use a TreeList inside of a Container:
<Container ......>
	<TreeList ..... />
</Container>

This results is the following error message:

Uncaught Error: Could not render child item: [object Object]

After looking at the code, it got clear that reactor does not support rendering of any Ext.widget right now.
https://github.com/sencha/extjs-reactor/blob/master/packages/reactor/src/ExtJSComponent.js#L175
Not sure about the implications, but I tried this:
if (item instanceof Ext.Component
    || item instanceof Ext.Widget) {
	(item.dock ? dockedItems : items).push(item);

My TreeList does now render and seems to work.
Will there be any problems with this?

Good catch!  There are a few other places where instanceof Ext.Component appear.  I'll change them to Ext.Base in the next update.

Should be fixed now on master.


Added Execution Time Finder

Pull Request Description
Execution Time Finder: [Find the execution time of your Python program]
Description
[Execution time finder which helps to understand your program execution time and optimize it accordingly]
Changes Made

[Added Execution time finder snippet]

[X] Fixed Bug/issues

[X] Added snippets/questions_answers/docs

[X] This pull request is all my own work -- I have not plagiarized it.

[X] All filenames are in PascalCase.

[X] All functions and variable names follow Python naming conventions.

[X] All test cases have passed.

[ ] All new algorithms have a URL in their comments that points to Wikipedia or other similar explanations.

checked all boxes properly and updated code to support longer times

Its not a snippet, it's very small and can be done easily.


Feat: Split Users DB tables into a core data table and a meta data table

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.

Project coverage is 83.75%. Comparing base (0458eae) to head (a8b8d7a).

Files
Patch %
Lines

src/db/schema/UserMeta.ts
87.50%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #53      +/-   ##
==========================================
+ Coverage   83.43%   83.75%   +0.32%
==========================================
  Files          62       64       +2
  Lines         851      862      +11
  Branches      115      111       -4
==========================================
+ Hits          710      722      +12
- Misses        116      117       +1
+ Partials       25       23       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feature/prepare for release 3.0.0

Coverage remained the same at 85.573% when pulling fa2fc13046d89ff00db1e6b55c413fd92d2df8eb on feature/prepare-for-release-3.0.0 into c35b61690892a2bdf5327106bb8e67241cd65dd4 on master.

Coverage increased (+0.03%) to 85.607% when pulling b5db0662d4c68e0f1a7384102b86796433ed38b2 on feature/prepare-for-release-3.0.0 into c35b61690892a2bdf5327106bb8e67241cd65dd4 on master.

Coverage increased (+0.03%) to 85.607% when pulling b5db0662d4c68e0f1a7384102b86796433ed38b2 on feature/prepare-for-release-3.0.0 into c35b61690892a2bdf5327106bb8e67241cd65dd4 on master.


Vv 201 contract test for update manager status by UUID

What
Why
Screenshots
Notes

Tests are all good. Worth noting that the actual gateway might need a little looking at. There's no try catch block. I know this isn't part of this PR.

I have implemented it such that rejected promises / any other errors thrown in the gateways are being caught in the outer catch block in the use cases. I hope this is okay.


Umple Generates Broken Code for Unique Immutable Attributes

Description
When setting an attribute as 'unique immutable', the code generated by Umple fails to compile as it tries to use functions it has not defined. I have failed to find documentation about Umple explicitly not supporting setting an attribute as 'unique immutable'.
I'm aware that the umple documentation suggests using 'before' statements to prevent setting values (similar to immutable), but I'm inclined to think that the 'unique immutable' keyword pair should still work.
similar to :
https://github.com/umple/umple/issues/400  (but in this issue the problem is that the code compiles and has functional bugs during runtime)
Reproduce
Setup:
Bug encountered on Windows 10 Pro Education V.22H2
using Java openjdk <IP_ADDRESS> 2023-08-24 LTS,
OpenJDK Runtime Environment Microsoft-8297089 (build <IP_ADDRESS>+1-LTS),
OpenJDK 64-Bit Server VM Microsoft-8297089 (build <IP_ADDRESS>+1-LTS, mixed mode, sharing),
coding on Microsoft Visual Studio Code V.1.72.2
with Umple plugin for Visual Studio Code, V1.2.1
Here is the directory structure of my setup:
/unique_immutable/src/java/ contains:
UniqueImmutable.ump
(these two get automatically generated)
MyClass.java
MyClass.class
Steps

Add the following code to UniqueImmutable.ump:

class MyClass{
    unique immutable myUniqueImmutableAttribute;
}

Generate the .java code with the >Umple: Generate Code from Umple command (ctrl+p to open the command palette). This step should succeed.
Attempt to compile the code with the >Umple: Compile Code from Umple.

Expected results
a message showing success of compilation
Actual Result
The following error was obtained:
UniqueImmutable.ump:23: error: cannot find symbol
    if (!setMyUniqueImmutableAttribute(aMyUniqueImmutableAttribute))
         ^
  symbol:   method setMyUniqueImmutableAttribute(String)
  location: class MyClass
1 error

My understanding is that the 'immutable' keyword prevents a setter from being generated (setMyUniqueImmutableAttribute()) while the 'unique' keyword uses the setter to check for uniqueness, causing the problem.

I have replicated this issue in Eclipse Version: 2023-06 (4.28.0), Build id: 20230608-1333
I was generating the code using the Umple plugin from http://cruise.eecs.uottawa.ca/org.cruise.umple.eclipse.plugin.update.site and compiling using
javac MyClass.java

from the /unique_immutable/src/java/ directory.
Here again, I was able to make this work if I only put either of 'unique' or 'immutable'.

Thanks for this. I will assign it for fixing right away.
I confirm that if I paste your code into UmpleOnline, and hit 'Execute it', the same result appears.
Tijm

Design option 1: In the generator, detect the combined case of immutable+unique and put the relevant code from the set method in the constructor in ONLY that case. This may make the generation template a bit more complex. I think this option should be tried first. New tests will have to be generated, but this should not break many (if any) existing tests.
Design option 2: For ALL unique attributes, instead of calling the set method, put the relevant code from the set method in the constructor. Drawback: Would slightly bloat the code of cases where the attribute is unique but not immutable, which may be common. This change will break all existing tests for generation of unique; those tests would have to be changed, which involves some risk that they are changed incorrectly.
Option 3. Disallow unique immutable with an error. This would not be reasonable.

We discovered that if one uses 'unique lazy immutable' the code generated is much better ... the only issue is a missing argument to the constructor. So the suggested design approach (we can now call option 4) is this:
All 'lazy' does when combined with 'immutable' is that it gives one chance to set an immutable attribute.
With three keywords together the lazy create a setter method, even when the attribute is immutable, but that setter can only be used once. And the constructor in the case of unique would make that call. ... except that the constructor does not currently have the aruemnt

In the parser, identify unique and immutable both being used together and set the relevant flag as though 'lazy' had been present, even if it was not.

The relevant template is
UmpleToJava/UmpleTLTemplates/constructor_AttributeAssignUnique.ump

We need to ensure the template generates the argument for the unique immutable attribute is adding an argument in the following code <<=gen.translate("parameterOne", av)>> Normally this will generate an argument, but it seems it is not doing so perhaps because of the presence of 'lazy' (which if used alone suppresses the presence of an argument on the constructor to allow it to be set later). We need to work out the logic to ensure the argument is not suppressed ... perhaps we need to add a new variable for this special sort of 'lazy' status.

The we need to add tests to make sure this works properly.

This was solved with PR #2055

Solved by PR #2055


docs: Migrate from Docusaurus v1 to v2

Fix: #209

Working on the failings and continuing the migration

@ErikQQY plz fix checks:

This PR is too messy, so I close this PR and I will try to implement the migration in these days.


TFLite NCHW data format issue and Open Questions

TFLite Version used - 2.4.1
Does TFlite Models can be created using NCHW format ?
When I tried to create , I was having issues ?
But the error i was getting was incorrect, Is this expected ?
In documentation it is not mentioned that NCHW is not supported ? When is this expected to support ?
Few open questions i found regarding this :-
https://stackoverflow.com/questions/66671358/which-format-is-preferable-for-tflite-model-nchw-or-nhwc
Is it possible to create tflite nchw models ? When TFLite NCHW format is supported ? Facing issues in the attached TFLite script
Open Questions :-
1.  Does TFLite supports NCHW ?
2.  What is the default format of TFLite Models ? It is always N,H,W,C ?
3.  What is the default format of TFLite Models LSTM models ? It is always N,S,D ? Does TFLite LSTM support S,N,D ?
4.  What are the input formats other than  N,H,W, C  (or) N,S,D ?
5.  How is ideally LSTM's and CNN's differentiated in TFLite ?
6.  When will be NCHW supported for TFLite if its to be supported ?
Facing issues in the below code :-
import numpy as np
import tensorflow as tf
input_shape = (1, 3, 3, 227)
x = tf.random.normal(input_shape)

model = tf.keras.Sequential([
  tf.keras.layers.InputLayer(input_shape=( 1, 3,227,227)),
  tf.keras.layers.Conv2D(96, 11, activation='relu', strides=(4,4), dilation_rate=(1,1), groups=1, data_format = 'channels_first')])
model.save('my_conv_model')

converter = tf.lite.TFLiteConverter.from_keras_model(model)

tflite_model = converter.convert()

with open('convmodel.tflite', 'wb') as f:
f.write(tflite_model)

I am unable to generate the TF model. Is it possible to fix this code? For questions about TensorFlow post a TensorFlow bug issue instead. We can then help you debug the conversion part.

Could you provide the solution to resolve this issue .
Please provide me the answers for the questions i asked.

Does TFLite supports NCHW ?

No.

What is the default format of TFLite Models ? It is always N,H,W,C ?

NHWC. Yes.

What is the default format of TFLite Models LSTM models ? It is always N,S,D ? Does TFLite LSTM support S,N,D ?
Could you point to an example that trains TF models with either of these formats?
What are the input formats other than N,H,W, C (or) N,S,D ?
Only NHWC is supported.
How is ideally LSTM's and CNN's differentiated in TFLite ?
It depends on the operator or layer type specified in the TF model.
When will be NCHW supported for TFLite if its to be supported ?

This is not in our roadmap yet as there is no requirement.

I cannot provide a solution until I can first create a TensorFlow model. If you can provide a Tensorflow model along with the code, I can help with the conversion.

Yes i will point an example of LSTM which takes N(Batch Size), S(Sequence length) and D(features ) as input. As in previous reply, You are sayiong only N,H,W, C are supported , But for LSTM's it takes N,S,D. Is there any API to tell what kind of format input is ?

The below is the example of TFLite LSTM model
`import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential()
model.add(layers.LSTM(input_shape=(None,20),units= 128, return_sequences = True))
model.add(layers.Dense(10))
model.summary()
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']);
x_test = np.zeros([3000,4,20], dtype=np.float32);
x_train = np.zeros([3000,4,20], dtype=np.float32);
y_test =  np.zeros([3000, 4,], dtype=np.float32);
y_train = np.zeros([3000, 4,], dtype=np.float32);
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
run_model = tf.function(lambda x: model(x))
BATCH_SIZE = 1
STEPS = 2
INPUT_SIZE = 20
concrete_func = run_model.get_concrete_function(
tf.TensorSpec([BATCH_SIZE, STEPS, INPUT_SIZE], model.inputs[0].dtype))
MODEL_DIR = "keras_lstm_new"
model.save(MODEL_DIR, save_format="tf", signatures=concrete_func)
converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_DIR)
tflite_model = converter.convert()
with open('lstmmodelnew.tflite', 'wb') as f:
f.write(tflite_model)
`

From the documentation here and example NSD is supported. You can view the model and you'll notice that NSD is used. As the API also seems to use NSD  (see the Args section.. all tensors use [batch_size, time, units]), it looks like only NSD is currently supported.
NHWC was for CNN/image models.

Created issue in stack overflow - https://stackoverflow.com/questions/67857868/unable-to-create-tflite-model-with-time-major-format
Not sure, If we can create TFLite Models in [S,N,D] (time_major) format.

Any update ?

@renjie-liu

Yes, you can create time-majored lstm model.


Add support for posix_spawn on OpenBSD

Could you squash commits into one?

Done

Thanks! @bors r+

:pushpin: Commit e004210089f3d16af0916a305574c29b72db5163 has been approved by JohnTitor
It is now in the queue for this repository.

:hourglass: Testing commit e004210089f3d16af0916a305574c29b72db5163 with merge c204453a7ef2448aab42988fa86898bb93e29751...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-13, checks-cirrus-freebsd-14
Approved by: JohnTitor
Pushing c204453a7ef2448aab42988fa86898bb93e29751 to main...


Use case name for temp dirs in server unit test

Having a separate temp directory for each test case help isolate them.

lgtm


[Model][Misc] Add e5-mistral-7b-instruct and Embedding API

Key Features of This Pull Request
This PR adds the e5-mistral-7b-instruct model and enables an E2E embedding vector generation.
There are a few open PRs that add support for embedding models. Our PR uniquely addresses the following key issues:
Integration of the OpenAI Server Front End
This PR includes comprehensive end-to-end (E2E) embedding functionality, spanning from the OpenAI Front End to the Back End.
Turn off KV cache with embedding models
This PR introduces the capability to turn off KV cache when operating in embedding mode, which includes the block_tables and cache_engine.
High Level Design
The embedding model can essentially be considered a special type of generative model with max_token=1 from an inference perspective. Both embedding and generative models (with max_token=1) require a single feedforward calculation without the need for generating any subsequent new tokens. The primary differences are:
The embedding model returns the hidden state, while the generative model takes an extra step to sample and return the first output token. In this sense, the embedding model is a subset of the generative model in terms of the calculations performed during the single feedforward operation on GPUs. They differ in their API specifications concerning request and output formats.
As a consequence, the serving of embedding models is:

Independent of any technologies implemented for subsequent token-by-token generations; and
They share the same technologies as generative models for prompt processing.

Therefore, in this PR, our current implementation focuses on bypassing vLLM’s components for subsequent token generations, such as KV cache and CUDA Graph to avoid unnecessary GPU memory, which in turn is useful to optimize the performance for embedding serving. If these technologies (e.g., CUDA Graph) are enhanced to improve prompt processing, they can be directly applied to embedding models. At a high level, our PR is independent of such crucial feature tensor parallelism, quantization, etc., which are addressed in Milestone 1 below. We have tested it with tensor parallelism > 1 and it functions effectively. We are willing to work together to test/improve these crucial features where needed.
Benchmarking Results
On 1 H100 GPU, vLLM’s serving of the E5 embedding model reaches a max throughput up to 36.6k tokens/s and remains consistent across sequence lengths from 128 to 32k (when gpu_memory_utilization is 0.8).
On 1 A100 GPU, it reaches up to 15.3k tokens/second and  also remains consistent across sequence lengths. As a comparison, in a test of the latest ONNX, the speed is up to 12.6k tokens/sec when sequence length is low (256 tokens), is reduced to 8k tokens/sec when sequence length is 2k, and shows a general trend of getting worse when the sequent length is longer.
See the figure for more details of our testing results on H100 and A100
Note: The throughput is measured as the "number of sequences in a batch" * "sequence length" / "end-to-end latency"

Design and Implementation Details
Add Embedding generation
Add Embedding API to entrypoints/openai

Add EmbeddingRequest and EmbeddingResponse to protocol.py
Add serving_embedding.py
Add OpenAIServingEmbedding to api_server
Make llm.py work with embedding

Add Embedding in outputs and sequence

Make an abstract base class of RequestOutput and SequenceGroupOutput.
Add separate Completion*Output and Embedding*Output.
Add EmbeddingOutput, EmbeddingRequestOutput, RequestOutputFactory and
EmbeddingSequenceGroupOutput to support processing the embedding output
sequence.
Update process output and sequence in *llm_engine to use embedding

Add MistralModel and embedding generation in llama.py

Add llama.embedding() and load_weights in LlamaModel to support forward
and embedding vector generation
Adapted from code examples in https://huggingface.co/intfloat/e5-mistral-7b-instruct
Mistral uses LlamaModel.
Use embedding when embedding_mode is True in model_runner
Add load_weights in llama.py to support embedding models

Disable KV Cache for Embedding serving
Skip slot_mapping with embedding mode

slot_mapping is only used in model_executor/layers/attention.py when
kv_cache is not None. In embedding mode, we pass None kv_cache. So no
need to process slot_mapping

Turn off KV cache for embedding mode
The goal is to disable the block_table and cache_engine completely, so
we don't consider allocating blocks for KV cache for embedding mode

Add embedding_mode to ModelConfig and SchedulerConfig
Add a BlockSpaceManagerProxy to control the block management behavior
for embedding mode

Update parameters for max batching

Add profile_max_batched_tokens_for_embedding to profile the
max_num_batched_tokens for embedding server mode
Return max_batch_size as each Ray worker runs the profiling once
Enable embedding profiling in ray_gpu_executor.py to support tensor parallelism > 1

Notes
Overlaps with other PRs
This PR overlaps with the following Milestones in Supporting embedding models #3187
Milestone 1: Basic functionality using Hugging Face models [Partially completed]
Note: Instead of using LLM.encode(), this PR currently adds embedding() to LlamaModel.
G) Update parameters for max batching [Completed]
We introduced profile_max_batched_tokens_for_embedding in gpu_executor.py to support the maximum number of tokens the GPU can take in one batch.
Milestone 2: Implement the model properly [Completed]
This PR focuses on adding the e5-mistral-7b-instruct model, which can utilize llama.py. So it already uses the vLLM layer primitives.
Milestone 4: Add the OpenAI Server Front End [Completed]
This PR has implemented the Embedding API to entrypoints/openai.
Discrepancies with other discussions
This PR didn't implement the following in this discussion in Supporting embedding models #3187
Move finish sequence logic to check_stop
Currently, the logic is in llm_engine._process_model_output()
The logic should be in llm_engine._check_stop
Automatically detect that the model is an embedding model
The user should not have to specify that it is an embedding model
Somewhere in the vllm code, create a registry that selects which models are embedding and which models are decoders
F) Update Pass EmbeddingParams around instead of SamplingParams
note: this is going to require a lot of work
Note: This PR passes SamplingParams() to the LLMEngine and disabled the use of it in embedding mode. As separating EmbeddingParams and SamplingParams requires changes to the UX, it would be easier to discuss and review in a following PR.
Discussion Points and Considerations
Simplifying the Workflow

Evaluate the possibility of consolidating embedding-related workflows into existing structures with minimal branching logic. This could involve using if-else statements (specifically in model_runner.py) or integrating embedding as a subset of generation needs.

Soundness of Profiling

Evaluate if the current profile_max_batched_tokens_for_embedding is sufficient to support the maximum number of tokens in one batch without causing CUDA OOM.

Hi there! I just noticed your PR from the main repository and this is fantastic! Do you mind also taking a look at https://github.com/vllm-project/vllm/pull/3187 and see if some efforts can be consolidated? Thank you so much for working on this!

Excellent work, and thank you very much. Could you please provide an example of getting the embedding of a text with your embedding branch?

Is this working now, are we able to host this embedding model over vLLM.


What are these training configurations of downstream tasks in SUPERB paper?

The training configurations of downstream tasks were not reported in the original SUPERB paper. If they are provided, it can help followers step into the SUPERB benchmark.
I got into trouble while reproducing HuBERT SID by a batch size of 32 and a learning rate of 1e-4. I found that using the different batch sizes and learning rates can result in significantly different performances. For a fair comparison, it can be helpful if the training configurations used in the SUPERB paper are released. WavLM also reported different batch sizes and learning rates.
Thank you.

Hi!
Thanks for the great suggestion!
This will really help people more easily reproduce our results.
I would document about how to get our finetuning config for reproducing.
In SUPERB, we use the exactly same config in each downstream folder eg. downstream/voxceleb1/config.yaml, including batch_size, except the learning rate. So, we did not tune the batch size for each upstream and keep it the same.
We tune the learning rate across log-scale, and here are some results for HuBERT Base:

learning rate
ACC

1e-1
0.814204335

1e-2
0.810083627

1e-3
0.7008847

1e-4
0.673736512

Hence, I think for lr 1e-4 we really can get good results on hubert-SID.
As for the other upstream/tasks, since I think people might be more interested in reproducing wav2vec2/hubert results, so I have released the finetuned artifacts as the reference.
Despite each downstream/upstream/ folder in the artifacts server includes only the checkpoints and the Tensorboard events, you can extract the finetuning config from the checkpoint and see how we finetune for a specific upstream-downstream pair.
Specifically,
#!/bin/bash

s3prl_path="/home/leo/d/Benchmarking/s3prl/s3prl"
working_dir=$(pwd)

wget -r http://<IP_ADDRESS>:8000/sid/hubert_1e-1/
cd $s3prl_path
python3 utility/extract_settings.py --ckpt_pth $working_dir/"<IP_ADDRESS>:8000/sid/hubert_1e-1/dev-best.ckpt" --save_dir $working_dir

Then, you can see the $working_dir/args.yaml and $working_dir/config.yaml files which contains all the finetuning configs.
For the learning rate, you can see the optimizer field in the config.yaml file.
I will put the above procedure in the SUPERB documentation. Thanks for the suggestion!

Or, you can directly infer the best learning rate as shown by the folder name in the artifact server lol
I believe the folder name is correct about the learning rate, but if it is not consistent with the configs saved in the checkpoints, you should treat the checkpoint configs as the truth. (Please help report the inconsistency if you fine one. Thanks!)

The way that extracts configuration from checkpoints is good for me. Thank you for your help. It, indeed, helps us obtain good results.
Have a nice day.

Hello,
I am trying to access the the superb artefacts server but the link is not working. Also, trying to reproduce the results from from the default downstream task configs but getting slightly better or worse results. Could you please help with that?


What is purpose of snapd-native, and it uses old format for RDEPENDS, anyway.

Not sure what the point of snapd-native is, but it's broken in that it uses the old format of "RDEPENDS_${PN}". So I'm not going to touch it and let someone else deal with that.
rday

It's purpose is to enabled the use of snap command to create system seeds. Must have missed it when updating for kirkstone. I'll push an update.

OK, I think this can be closed.


Publish the extension to OpenVSX

Hi, I'm a VSCodium user, a FLOSS version of VS Code. It's the same code but without Microsoft's telemetry, branding and licensing.
I would like to request, if possible, to add the change-case for VSCode extension to the open-vsx.org market as it is unclear if it's legal to use the official Microsoft extension marketplace on VS Code forks, see here.
Additional information on how to publish an extension to Open VSX can be obtained here.

Hi there,
We decided to remove this extension from the https://github.com/open-vsx/publish-extensions repository, since it has not had any activity for a while and we try to make the list of published extensions shorter (but published the extension as well, so at least one version is out there!).
This means that new versions of the extension will not be available on the OpenVSX registry, but feel free to enable automatic publishing yourselves, we made a pretty easy guide in our repo [1].

 What is OpenVSX?
OpenVSX is a vendor neutral alternative to the MS marketplace used by most other derivatives of VS Code like VSCodium, Gitpod, OpenVSCode, Theia based IDEs, and so on.

Feel free to close this, just wanted to let you know :)


Behavior at jump discontinuities

Consider the plot <Plot.OfX y={(x: number) => math.mod(x, 1)}/>. This of course has a number of jump discontinuities, but by default mafs joins discontinuities with a vertical line. Personally, I find this behavior unexpected and undesirable and would like to at least have the option to configure the discontinutiy behavior. Something like Mathematica's Exclusions and ExclusionsStyle seem appropriate for the existing Mafs API. I.e., the user provides Plot.OfX with a parameter which tells it where the discontinuities are and how they should be drawn.
To my mind, this should either be:

an array of x values of discontinuities
a function (x: number): boolean

I think the "jump" style that I'm hoping for can be accomplished by modifying the onPoint callback in sampleParametric to output an M command rather than an L command if the t value of the point is a discontiuity.
Let me know what you think! I love the library, and I'm using it to make companion interactive visualizations for my dissertation :)

Totally agreed. I think the function approach might be required for cases where there are infinite discontinuities, but instead of accepting a number and returning a boolean, it should accept a range and return a list (with the option to pass a static list, no function).
Mafs could use those values and:

Evaluate continuity (or the user can explicitly specify)
Ensure additional points are plotted near these important points (whether continuous or not)

In other words, a way to indicate important regions to plot.

Agreed also on the approach for jump discontinuities.
For holes, the best approach would probably just be a two <circle>s: one to mask off the function, and one to show the classic circular outline.
This means adding a mask to some functions which may have an effect on perf, but I doubt it.

Got a nice API for holes I think (not jumps yet, because the sample function scares me after not working with it for over a year).

I think JavaScript's stupidity will prevent a lot of ergonomics here. But I'm okay with devs having to do the math to tell Mafs where to display these indicators.

Wow thanks for starting working on this so quickly, this looks great! I think your modifications for the "jump" api make sense, and if you're open to doing code review I would be happy to take a crack at implementing it.
As a mathematician, I swear Javascript makes some truly odd choices for built in operators that have caused some surprising and sneaky bugs for my viz. I have certainly not come to expect ergonomics haha.


support gpt4o

Problem Description
as in title
Solution Description
add option for new model
Alternatives Considered
No response
Additional Context
No response

@jiangying000  : could you please translate into English? Thank you

@jiangying000 : could you please translate into English? Thank you

@jiangying000 : could you please translate into English? Thank you

@jiangying000 : could you please translate into English? Thank you

done thanks

https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web/pull/4674

Merged


Not able to connect with Azure china keyvault using azure-spring-boot-2.3.2.jar

Hello,
our application connects with key vault on startup and it uses following dependencies.
azure-spring-boot-2.3.2.jar
azure-identity-1.0.4.jar
msal4j-0.5.0-preview.jar
We are hosting the application in china and that will connect with keyvault in azure china. Since the azure-spring-boot library has authority url hard coded  as 'https://login.microsoftonline.com/', application is not able get oauth token.
could you please help us to fix the issue, so that we can get token from https://login.chinacloudapi.cn/{tenant-id}/oauth2/token
Below is the configuration class:
@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(
entityManagerFactoryRef = "entMgrFactory",
basePackages = {"xxx.yyy.zzz"}
)
public class MyDataSourceConfig extends HikariConfig {
@Bean("myDataSourceProperties")
@Primary
@ConfigurationProperties("spring.datasource")
public DataSourceProperties myDataSourceProperties() {
return new DataSourceProperties();
}
}
keyvault related properties in application-local.yml
azure:
keyvault:
uri: {key_vault-uri}
client-id: {client_id}
client-key: {client-key}
tenant-id: {tenant-id}
Error received on application statup:
Caused by: com.microsoft.aad.msal4j.MsalServiceException: AADSTS90002: Tenant {tenant-id} not found. This may happen if there are no active subscriptions for the tenant. Check to make sure you have the correct tenant ID. Check with your subscription administrator.
Thanks in Advance!!!

Hi @sans30 To use Azure China, have you added such entry in the application.properties?
azure.activedirectory.environment=cn

Hi @yiliuTo ,
Yes, I tried adding azure.activedirectory.environment=cn in application.properties, but it didn't work.

Hi @sans30 , to pass the OAuth2 authentication in Azure China, I think you could reset the endpoint manually in application.properties by replacing login.microsoftonline.com with login.chinacloudapi.cn, and change common to your tenant-id if needed.
spring.security.oauth2.client.provider.azure.authorization-uri=https://login.microsoftonline.com/common/oauth2/authorize
spring.security.oauth2.client.provider.azure.token-uri=https://login.microsoftonline.com/common/oauth2/token
spring.security.oauth2.client.provider.azure.user-info-uri=https://login.microsoftonline.com/common/openid/userinfo
spring.security.oauth2.client.provider.azure.jwk-set-uri=https://login.microsoftonline.com/common/discovery/keys

Hi @yiliuTo , spring.security.oauth2 properties also didn't work. Problem is when application starts up the framework is trying to connect with keyvault to get keys. Since the IdentityClientOptions.java is referencing to DEFAULT_AUTHORITY_HOST = "https://login.microsoftonline.com/", my application fails. I was able to debug the library and the execution flows like below.
KeyVaultEnvironmentPostProcessor.postProcessEnvironment()->helper.addKeyVaultPropertySource("")->KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource()->getCredentials(normalizedName)->ClientSecretCredentialBuilder.build()->new ClientSecretCredential(tenantId, clientId, clientSecret, identityClientOptions)
could you please generate spring boot project with above listed dependencies and try to debug the issue.

Hi @sans30 , sorry that I misunderstood your case as a problem of application login error at first. Besides thanks to your clear and detalied debug info, we will investigate and fix this as soon.

Hi @yiliuTo , Do you have any timeline to fix this issue? Also I see there is a beta version (1.1.0-beta.3) which has a fix to configure azure authority host. I would like to know the relase plan of the actual version.

Hi @sans30 , to fix this issue is added into our recent working schedule, but the release plan is not sure yet. Besides, could you provide a link to the beta version or the associated github fixing issue you mensioned? Thanks~

Hi @yiliuTo you can find it from release history.
Change log for azure-identity: https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/identity/azure-identity/CHANGELOG.md
Release number: 1.1.0-beta.3 (2020-04-07)
Please let me know if you need more information.

Hi @sans30 , thanks to your help. We have tested with 1.1.0-beta.3 version of Azure-Identity and it works well to specify the authority host. However, we cannot update the dependency so far because it's a beta version, and we will perform the update once it releases a GA. Thanks again.

Hi @yiliuTo , I see that azure-identity 1.1.0 has been released, does this have fix for Azure Authority Host? and can you please let me know if this can be used with Spring-2.3?

Hi @sans30 sorry for the late response, yes the 1.1.0 has fixed authority host issue. And I have tested it with key vault starter 2.3.2 which can work normally.

FYI, our key vault starter also has a property to set the authority host now, check it here https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/spring/azure-spring-boot-starter-keyvault-secrets#custom-settings.
Closing this issue now.

FYI, our key vault starter also has a property to set the authority host now, check it here https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/spring/azure-spring-boot-starter-keyvault-secrets#custom-settings.
Closing this issue now.


[Feature request] Tiered conditionality for form fields.

Is your feature request related to a problem? Please describe.
Right now, conditional fields are based on a single condition. This causes issues with nested conditions, whereby fields who's conditions were previously met and which have been displayed do not have their conditions updated when those nested conditions are no longer met.
Here is an example:
https://github.com/user-attachments/assets/da57aa45-0e38-4210-bd66-826ba2534611

Domain is the top level Field with the values of "Personal" and "Professional".
Personal Areas is conditional on Domain: Personal
Professional Areas is conditional on Domain: Professional
The video demonstrates a further nested condition. Personal/Areas/Interests, which has two select Options: Interest 1, and Interest 2.
When the Domain is subsequently changed to Professional while the Personal Area (Interest) is selected, the field for Interest remains selected, despite one of its parent conditions no longer being met (Domain: Personal would be the first condition in this chain, and so subsequent conditionals should become invalid.

Describe the solution you'd like
To solve this issue, I would like the ability to add arbitrary levels of conditions to for any given field, and in this way manage all conditional requirements needed to avoid these invalid fields being present.
Alternatively, all conditionals could check for nested conditions further down the chain, and mark them as "unmet" when a parent condition is no longer valid.
I think ultimately both of these options would serve the plugin immensely.

In addition, dependent child fields that were previously selected and set, and then had parent fields modified, and subsequent child fields set are maintained, erroneously. In the following example:
Domain is set to Personal
Personal Area is set to Interest
An interest is chosen: Interest 2
The domain is set to Professional, yet The interests field remains, and Interest 2 remains assigned (despite it being a child of Personal Domain/Interests Area)
The Domain is set back to Personal
The Personal Area is set to Health
The Domain is set to Professional
The Professional Area is Set to Writing
The Writing Type is set to blog post
The form is submitted.
The Form Results of this submission are:
Domain: Professional (Correctly set)
Professional Areas: Writing (Correct set)
Personal Areas: Health (Correctly set, but should be absent from the form result as Domain is Professional)
Interest: Interest 2 (Correctly Set, but Interest should be absent as it is a child field of "Personal Areas", and "Personal Areas" is a child of the "Personal Domain", which is no longer present.
Writing Type: Blog post (Correctly Set)
https://github.com/user-attachments/assets/f1c838ef-8f17-43b7-b6ff-7f70a4b3e7fd
I hope this clarifies the behavior I am attempting to describe, and a path forward for correcting the issue, and implementing the child dependencies / adding arbitrary number of conditions to the Field Conditionality feature of Modal Plugins.

I have noted now that Naming the Fields the same Name, and Labelling them separately does overcome one of the issues with regards to tiered conditionality. Now when the Fields are named the same, they will overwrite the preview level of fields set based on that conditionality. For example:
https://github.com/user-attachments/assets/4b17dffa-7c9b-498b-9c00-db5eb908cdd8
It seems that Fields will be overwritten by conditionally dependent fields when they are the same label, which is expected behavior. But subsequent child dependencies are not cleared and removed, which would also be expected behavior.
Since my use case would employ different names for child fields, this would be I will always assign unnecessary frontmatter properties to files if I was indecisive when using the form to create them.
I think this can be overcome by simply clearing child fields from the proposed form composition when their parents are no longer present.

It seems that Fields will be overwritten by conditionally dependent fields when they are the same label, which is expected behavior.

That's a neat trick!

It seems that Fields will be overwritten by conditionally dependent fields when they are the same label, which is expected behavior.

That's a neat trick!

I'm sorry, I got that backwards!
Labels are independent, but if the fields have the same NAME they will overwrite one another. So you can have two of the same named fields, with different LABELS and make them conditionally dependant on higher, separate fields, and overwrite them based on different conditions.

I added this feature with the most possible basic feature set, and left everything else to be based on feedback (check here for reasoning).
This was something I was not sure about. To me, it feels natural that, if a field that you depend on are not rendered, yours is not either, but I was not sure about it, so I decided to just leave it as is. I would like to collect more user feedback before I take a decission, but if nobody else chimes in, I think I will do what I consider most obvious.
This may feel natural, or logical, but please understand that such things vary a lot from person to person, and what one considers logical it is unexpected for others. A great example is your "abuse" of field names. I consider to have several fields with the same name a bug, but it was never an issue until this feature came out. Now, you found it and are taking advantage of it 😂 . What will happen if I now fix that bug? I will be breaking your workflow. This is just a demonstration of how every new feature or every bug fix may break the workflow of someone.

indeed the idea is, if I make this "cascading conditions" to also make the values not visible not appear in the result. As I said, I started this as a very minimal, because it was not a feature I was needing that much (although, I admit I had a little desire for it) so I decided to not do a drastic change. Specially because, if anything went wrong, it will be harder to know what part of the new very complex feature it was.
That said, it's very hard for me to imagine a scenario where you want hidden fields to appear in the result, so omitting them seems like the sensible thing to do to me


preserve input order for gene list search

We don't know if this will be feasible, but we've had users request it, so:
Option for the gene name and systematic ID filters, to have results returned with genes in the same order as in the input list.
User requests in https://rt.sanger.ac.uk/Ticket/Display.html?id=303908 & https://rt.sanger.ac.uk/Ticket/Display.html?id=456638

moved to "gene list" upload ticket


Try to update to Git 2.19

The CURL_OPENSSL_3 (https://github.com/theia-ide/dugite-extra/issues/21)  issue might have been resolved, cf. https://github.com/desktop/dugite-native/issues/109#issuecomment-433048020.
We should try to use dugite 1.79.0 on Ubuntu 18.04 and pull / push works, it should be fine to update.

I will do an update soon. Does it make sense to bump it up?

Before bumping the version, I'd rather have a test like what we did last time.
On newer Ubuntu (docker should be easiest)

install latest dugite
cd into directory of embedded git
try out git: pull and push should work (they use curl lib)

Is this still relevant, @AlexTugarev?

I think this is outdated, as we're no longer using the embedded git. Sorry for leaving the mess around.


Implement SE-0219 Package Manager Dependency Mirroring

rdar://problem/42511642 [SR-8328]: Implement SE-0219 Package Manager Dependency Mirroring
https://bugs.swift.org/browse/SR-8328

@swift-ci smoke test

@swift-ci test with toolchain

@swift-ci smoke test macOS

@swift-ci smoke test OS X

Maybe this is too late, anyways I want to point to a scenario.
When the user attempts to set a mirror-url for given package url like such:
$ spm package config set-mirror --package-url pkg1 --mirror-url pkg2  (1)
$ spm package config get-mirror --package-url pkg1
// pkg2

$ spm package config set-mirror --package-url pkg2 --mirror-url pkg3 (2)
/// pkg1 --mirror--> pkg2
/// pkg2 --mirror--> pkg3

1 we dont verify if the provided package-url is present in the list of dependencies user specified in Package.swift. Thereby it kind of works like setting values without much validation. I'm not sure which is better but personally (atm) I think we can verify if the package was specified.
2 This is partly a side effect of not having check as described in point 1.  The main question is A -> B and B -> C. Is package C mirroring A then?
These are my opinion. Let me know what you think.

It is indeed possible to misuse the mirroring feature which could lead to weird behaviors. However, we think that the majority of users will use the feature as intended. It would be nice if we could do some sort of validation but there are several complications in doing so.
Validation of mirrored repository:
We thought about possibly verifying that the mirror is indeed a mirror of the same package and not something else. But one of the use-case of mirrors is to get access in an env. where you don't have access to the original url.
Checking the presence of dependency:
This has several issues. The package graph might have not been resolved yet. It is completely valid to have mirror entries of dependencies that are not in your graph yet. In that case, we would end up showing an error or warning, which doesn't seem correct.
We can always make changes to a feature by the swift evolution process. Let's see the response from users and we can figure out how we can improve it if they do run into weird things.


Taggle div loses label identifier

Here's our taggle-based form field:

If I clicked on the "Keywords" label, I'd expect the form field to get focused, but it does not.
Here's the HTML that's generated:

  KeywordsType a comma, Return, or Enter to finish a keyword
  <input class="form-control form-control-full-width js-keyword-picker" type="text" name="curated_link[keywords]" id="curated_link_keywords" style="display: none;">

<input type="text" class="taggle_input" tabindex="0" style="padding-left: 0px; padding-right: 0px;">

Taggle applies display: none; to our curated_link_keywords input and replaces it with its own, without the original id="curated_link_keywords" attribute.
Hope that makes sense?

Since taggle uses it's own input, can you use something like:
var input = instance.getInput();

input.id = 'curated_link_keywords';

Thanks 🙇  , we'll try that out.

@garethrees did that solve things for you?

The project is in maintenance mode at the moment, so we haven't had chance to give this a try. We've linked to this issue upstream, so we'll get to it when we can and open a new issue if we can't get it fixed. Sorry for the super slow reply.


⚠ St. Lawrence Global Observatory CKAN Sitemap has degraded performance

In 77ceabd, St. Lawrence Global Observatory CKAN Sitemap (https://catalogue.ogsl.ca/sitemap/sitemap-1.xml) experienced degraded performance:

HTTP code: 200
Response time: 5114 ms

Resolved: St. Lawrence Global Observatory CKAN Sitemap performance has improved in 6522248 after 5 minutes.


Add performance test scripts

Signed-off-by: Kunal Kotwani<EMAIL_ADDRESS>Description

Add Jenkins job and groovy scripts for performance tests

Issues Resolved

https://github.com/opensearch-project/opensearch-build/issues/129

Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1671 (7affa4d) into main (a80e17a) will increase coverage by 5.30%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #1671      +/-   ##
=============================================
+ Coverage     94.69%   100.00%   +5.30%
=============================================
  Files           163         4     -159
  Lines          3447        65    -3382
  Branches         21        14       -7
=============================================
- Hits           3264        65    -3199
+ Misses          180         0     -180
+ Partials          3         0       -3

Impacted Files
Coverage Δ

src/run_perf_test.py

src/test_workflow/perf_test/perf_test_suite.py

tests/jenkins/jobs/UploadToS3_Jenkinsfile

src/assemble_workflow/bundle_url_location.py

src/ci_workflow/ci_manifest.py

src/system/temporary_directory.py

src/ci_workflow/ci_check_npm_package_version.py

src/ci_workflow/ci_check_package.py

src/test_workflow/test_component.py

src/test_workflow/perf_test/perf_test_cluster.py

... and 149 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a80e17a...7affa4d. Read the comment docs.

@kotwanikunal Looks like there are failing tests. Can you fix those?
Thanks!

The tests succeed on my machine locally -
(opensearch-build) kkotwani@3c22fbb2c92e opensearch-build % git commit -m "Add parallel stages for performance tests, add tests" -s
isort....................................................................Passed
flake8...................................................................Passed
mypy.....................................................................Passed
pytest...................................................................Passed
yamllint.................................................................Passed

Looks to be an import issue on the GHA distros -
Run pipenv run flake[8](https://github.com/opensearch-project/opensearch-build/runs/5597195180?check_suite_focus=true#step:6:8) .
Traceback (most recent call last):
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/bin/flake8", line 5, in <module>
    from flake8.main.cli import main
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/lib/python3.7/site-packages/flake8/main/cli.py", line 5, in <module>
    from flake8.main import application
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/lib/python3.7/site-packages/flake8/main/application.py", line [11](https://github.com/opensearch-project/opensearch-build/runs/5597195180?check_suite_focus=true#step:6:11), in <module>
    from flake8 import checker
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/lib/python3.7/site-packages/flake8/checker.py", line 18, in <module>
    from flake8 import processor
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/lib/python3.7/site-packages/flake8/processor.py", line 13, in <module>
    from flake8 import utils
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/lib/python3.7/site-packages/flake8/utils.py", line 16, in <module>
    from flake8._compat import lru_cache
  File "/home/runner/.local/share/virtualenvs/opensearch-build-ucxYjMk8/lib/python3.7/site-packages/flake8/_compat.py", line [12](https://github.com/opensearch-project/opensearch-build/runs/5597195180?check_suite_focus=true#step:6:12), in <module>
    import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metadata'

Python version mismatch? https://github.com/python-poetry/poetry/issues/1487

Python version mismatch? Is yours passing on 3.8 vs. 3.7? python-poetry/poetry#1487

Thanks!
That seemed to be it. My venv was on 3.7 but it picked up the OS default which was 3.8.
And since the Pipfile.lock was updated, it was breaking with 3.7 on the build envs.

Can you also add a test case for the jenkins job? Thank you!

@kotwanikunal Can you please take a look at the comments and provide an update?

Can you also add a test case for the jenkins job?
Refer this
Thank you!

There are test for both the scenarios now. The dependency on downloading the manifest file was eliminated and instead I have used hard coded values, which makes it on par with the actual test script.

@kotwanikunal Can you please take a look at the comments and provide an update?

All the pending change requests have been resolved.

Can you also add a test case for the jenkins job?
Refer this
Thank you!

There are test for both the scenarios now. The dependency on downloading the manifest file was eliminated and instead I have used hard coded values, which makes it on par with the actual test script.

We need a test for the jenkinsfile, to make sure that it fails if there are any changes to the job. It would also make sure that the job calls the libraries correctly.

Can you also add a test case for the jenkins job?
Refer this
Thank you!

There are test for both the scenarios now. The dependency on downloading the manifest file was eliminated and instead I have used hard coded values, which makes it on par with the actual test script.

We would still need a test for the jenkinsfile, to make sure that it fails if there are any changes to the job. It would also make sure that the job calls the libraries correctly. Example SignStandaloneArtifactsJob, DataPrepperJob

I'll add and update the PR shortly.

Let me know whenever you add it. Thanks!
cc: @bbarani

Can you also add a test case for the jenkins job?
Refer this
Thank you!

There are test for both the scenarios now. The dependency on downloading the manifest file was eliminated and instead I have used hard coded values, which makes it on par with the actual test script.

We would still need a test for the jenkinsfile, to make sure that it fails if there are any changes to the job. It would also make sure that the job calls the libraries correctly. Example SignStandaloneArtifactsJob, DataPrepperJob

I'll add and update the PR shortly.

Let me know whenever you add it. Thanks!
cc: @bbarani

Took longer than expected to get all the cases in. All the cases and jobs should have relevant tests now.

Can you also add a test case for the jenkins job?
Refer this
Thank you!

There are test for both the scenarios now. The dependency on downloading the manifest file was eliminated and instead I have used hard coded values, which makes it on par with the actual test script.

We would still need a test for the jenkinsfile, to make sure that it fails if there are any changes to the job. It would also make sure that the job calls the libraries correctly. Example SignStandaloneArtifactsJob, DataPrepperJob

I'll add and update the PR shortly.

Let me know whenever you add it. Thanks!
cc: @bbarani

Took longer than expected to get all the cases in. All the cases and jobs should have relevant tests now.

Thanks for adding the tests. However, we don't want to create a copy for the job, but rather use the original jenkinsfile to make sure that works. If the test is written on the copy, then any change on the original job will not fail the test and therefore, defeat it's purpose. Let's connect offline so we can close this soon.
Thanks!

Can you also add a test case for the jenkins job?
Refer this
Thank you!

There are test for both the scenarios now. The dependency on downloading the manifest file was eliminated and instead I have used hard coded values, which makes it on par with the actual test script.

We would still need a test for the jenkinsfile, to make sure that it fails if there are any changes to the job. It would also make sure that the job calls the libraries correctly. Example SignStandaloneArtifactsJob, DataPrepperJob

I'll add and update the PR shortly.

Let me know whenever you add it. Thanks!
cc: @bbarani

Took longer than expected to get all the cases in. All the cases and jobs should have relevant tests now.

Thanks for adding the tests. However, we don't want to create a copy for the job, but rather use the original jenkinsfile to make sure that works (as in the examples and readme). If the test is written on the copy, then any change on the original job will not fail the test and therefore, defeat it's purpose. Let's connect offline so we can close this soon.
Thanks!

Sorry took me a while to realize that and then get the actual script to have all the testable stubs and mocks. Jenkins/Python newbie here. 🙂
Let me know if this resolves your concerns.
Thanks!

Merge on green.

GA Run:
62 tests completed, 2 failed, 1 skipped

Tests running successfully locally.
Can someone from @opensearch-project/engineering-effectiveness help with this? 🙂

The success message in TestRunPerfTestScript > testRunPerfTestScript_Pipeline looks different between expected and not, probably a leaking ENV variable across tests that causes different output.
Diff the failed test output in the log:
<                           publishNotification.sh(curl -XPOST --header "Content-Type: application/json" --data '{"result_text":":white_check_mark: perf-test [1236] Performance Tests Successful
<      Build: test://artifact.url
---
>                           publishNotification.sh(curl -XPOST --header "Content-Type: application/json" --data '{"result_text":":white_check_mark:
>      JOB_NAME=perf-test
>      BUILD_NUMBER=[1236]
>      MESSAGE=Performance Tests Successful
>      BUILD_URL: test://artifact.url
>      MANIFEST: null
146,148d149
<                     perf-test.postCleanup()
<                        postCleanup.cleanWs({disableDeferredWipeout=true, deleteDirs=true})
<      "


procedures/713310 returns an internal server error

Describe the bug
http://aind-metadata-service/procedures/713310 returns an internal server error
To Reproduce
Steps to reproduce the behavior:

Go to http://aind-metadata-service/procedures/713310
See error

Expected behavior
Data should be returned
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Pulling this into this sprint because I accidentally found this bug at the same time and resolved it in this PR #244

This bug was fixed by #244


Negative exponents appear positive

Describe the bug
Negative exponents do not appear in a profiling report
To Reproduce
import numpy as np
import pandas as pd
from pandas_profiling import ProfileReport

data = { 'some_numbers' : (0.0001, 0.00001, 0.00000001, 0.002, 0.0002, 0.00003) * 100}
df = pd.DataFrame(data)
profile = ProfileReport(df, 'No Negative Exponents')
profile.to_file('NoNegativeExponents.html')

Minimum should be 1 x 10-8 rather than 1 x 108. The issue also arises for Mean and Maximum.
Version information:
Python 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Additional context

Good catch. Feel free to make a PR with a fix (relevant code), otherwise I'll pick it up later.


Idea: dedicated website for the docs.

I think it would be a good idea to have a dedicated website for the docs, as to make it easier to navigate it. The docs are quite substantial, so in my opinion it would be a lot easier to find what you are searching for (instead of crawling through the readme).
Any thoughts on this? (I understand this is most likely not a priority though)

I saw there were dedicated docs on the Laravel website for Stripe and Paddle. Why not for Mollie also, it's a located at the Laravel Github account, so documentations should be available there too?

I've recently had a discussion about this with Taylor. As Cashier Mollie is not built by the core Laravel dev team, we'll have to find another place for the docs. For me it would also really hard to maintain the docs there without full control over the PR/build process.
We're looking into setting up a simple VuePress website now, which may allow us to include a bit more detailed and rich code samples.

Sad to hear. Especially because the repository is inside the Laravel project I would suspect that it's on the mainsite or there's a link to the repo. I only found this repository through the Mollie website because I didn't see on the official docs that there was a package for it. I thought first it was by Mollie or 3rd party developer but then I saw it was on the Laravel project. Therefore the confusion about the docs.

I hear you but there's not much we can do to host it there I'm afraid.
What we can - and will - do is make sure we reach more Laravel developers ourselves.

V2 will launch later this year with a dedicated documentation website :tada:


Connect Admin HeatMap to backend endpoint

/cib connect-admin-heatmap

Branch 3-connect-admin-heatmap created!


0.2.9 broke the bos/eos/sys handling for chat sequences

recent changes (0.2.7 -> 0.2.9.) to the chat framework broke things (this is vicuna 7b)
first call always returns the prompt:
[INST] <<SYS>> You are a helpful assistant. <|endoftext|>

second call works:
Once upon a time in a faraway land, there lived a silly frog named Freddy who loved to hop around on lily pads and croak loudly.

third and continuing calls get odder, and show the bos tokens, etc:
[INST] <<SYS>>
You are a helpful assistant.
<</SYS>>

The silly frog hops around on the lily pad, croaking loudly.

trying different versions.

0.2.7 fixes it
0.2.9 breaks it
0.2.11 still broken

its a token handling issue.   gonna try to figure out what broke between 0.2.7 and 0.2.9
looks like it was this:
"+- Add configurable chat formats by @abetlen in #711"

this PR does not fully fix the issue, leaving most users locked to version 0.2.7
https://github.com/abetlen/llama-cpp-python/pull/808

easy to reproduce, just specify a longer chat conversation.   at least 3 user and 2 assistant messages result in weird results

@earonesty just confirming that #850 didn't resolve this?

correct I tried that branch and while it fixes the first request it does not fix subsequent requests
It's really about subsequent requests with a persistent running instance

tried again today with v 0.2.13:
$ python example.py
This is an instruction to write a sentence about a silly frog in the next message.
(venv)
erik@LAPTOP-1K8B9NQ8 MINGW64 ~/chainer (main)
$ python example.py
You have been asked to write a description of an absurd scenario, and the only mental image you can conjure is that of a silly-looking frog. It jumps around in your mind, its skin green and
bumpy like a piece of broccoli, and it hops along with a goofy grin on its face.  Maybe it would croak out a silly joke
 or do a funny dance. Or perhaps it would just sit there, looking completely bewildered by the world around it.
(venv)
erik@LAPTOP-1K8B9NQ8 MINGW64 ~/chainer (main)
$ python example.py
[INST]Write one paragraph about why the frog is silly. [/INST]

you can see that it starts hallucinating system messages and prompts, instead of generating responses to the "write one sentence about a silly frog".   this is vicuna.   version 0.2.7 just works every time.

hmm, looks like maybe this is a problem a little deeper.   i can see the engine is formatting the input correctly:
<<SYS>>
You are a helpful assistant.
<</SYS>>

[INST]Write one sentence about a silly frog. [/INST]

some oddness with lots of extra spaces

fixed it.
you now need to specify the chat_format correctly ... it won't guess anymore!


 When I've done this in the past using gpg, I've always had a .asc file, but I can't see one for EPS...

Using the above procedure I have added signatures to the v0.1.1 release(!)
See https://github.com/chris-belcher/electrum-personal-server/releases/tag/v0.1.1


Fix secret file mode bits example on JSON

I tested the example on YAML and, wrongly, asumed that JSON supported it too.
Fixes: https://github.com/kubernetes/kubernetes/issues/33162

This change is

@devin-donnelly PTAL (sorry, I haven't used this new review interface. I'm not sure I'm using it correctly :))

I won't have a chance to review this until later, but I think @rata is awesome for doing this :-)

Looks good, thanks!

LGTM


Does the vscode-shellcheck extension fully support zsh scripting code?

Does the vscode-shellcheck extension fully support zsh scripting code?

No, the extension uses the shellcheck to do its work, which in turn does not support ZSH for now.
https://github.com/koalaman/shellcheck/wiki/SC1071