Auth and make requests as user based on Authorization header

When Supabase's client supports it, we'll be able to parse the JWT inside of the Authorization header and use it to authenticate "as" a user inside of this API.
https://github.com/supabase/gotrue-js/pull/80

I believe it's already supported. You can pass options to createClient().
Something like:
const headers = { authorization: accessToken }
const supabase = createClient(SUPABASE_URL, SUPABASE_API_KEY, {headers})

ah, g2k! my impression from this thread is that this might not work, even though it does support that options param, it doesn't get picked up by the Supabase client when it's running in a non-browser context. very interesting!


Add init_by_shape method usage example.

I added the module usage example to make the doc more understandable.(This commit just a starter commit, I will continue after I understand where to add on correct way)
Will docstring be used in the future ? So should I add the examples there or to a different place?

@MostafaDehghani should I continue like that type
Args: ........
Returns:.......
Examples:..........

or do you recommend different style for usage example?

Thank you @us.
Your proposed format is pretty good (and I think it's a standard).
The only thing that can make it a bit nicer is to wrap the code in the Example section by backticks.
e.g. In your commit:
...
Returns:
      A pair consisting of the model output and the initialized parameters	      A pair consisting of the model output and the initialized parameters
    Example:
     ```
      input_shape = (batch_size, image_size, image_size, 3)
      model_output, initial_params = model.init_by_shape(jax.random.PRNGKey(0),
                                                         input_specs=[(input_shape, jnp.float32)])
     ```

And, yes! You are more than welcome to send PRs that add Example section to docstrings! :)

Thanks - also when you edit it make sure it fits in the 80char limit to match our style.

Okey @levskaya thank you I will now !

@MostafaDehghani it's didn't look right. How can I fix that. Any resource about that style of doc?


Problem handling final \n char from rbw

Looks like rofi-rbw trips on the final \n character at the end of the password text from rbw:
Traceback (most recent call last):
  File "/home/cstrahan/.local/bin/rofi-rbw", line 8, in <module>
    sys.exit(main())
  File "/home/cstrahan/.local/lib/python3.8/site-packages/rofi_rbw/rofi_rbw.py", line 187, in main
    RofiRbw().main()
  File "/home/cstrahan/.local/lib/python3.8/site-packages/rofi_rbw/rofi_rbw.py", line 134, in main
    data = self.get_credentials(selected_entry.strip(), selected_folder.strip())
  File "/home/cstrahan/.local/lib/python3.8/site-packages/rofi_rbw/rofi_rbw.py", line 183, in get_credentials
    return Credentials(result)
  File "/home/cstrahan/.local/lib/python3.8/site-packages/rofi_rbw/credentials.py", line 13, in __init__
    key, value = line.rsplit(": ", 1)
ValueError: not enough values to unpack (expected 2, got 1)

Thanks for noticing that! Should be fixed on main now 🙂

@fdw could you make a hotfix release for this? The AUR package is broken right now since it follows stable releases.

I assume this means that the fix works. Release is on its way :)

Thank you, the fix does indeed work.


Deprecation warning with Gradle 4.0-rc-1

The following deprecation warning is generated with Gradle 4.0-rc-1:
Using a single directory for all classes from a source set. This behaviour has been deprecated and is scheduled to be removed in Gradle 5.0
	at edu.sc.seis.macAppBundle.MacAppBundlePlugin.apply(MacAppBundlePlugin.groovy:84)
	at edu.sc.seis.macAppBundle.MacAppBundlePlugin.apply(MacAppBundlePlugin.groovy)

Output above is generated with --stacktrace option, grepping the stacktrace elements on edu.
Otherwise the given location is the place in build.gradle where the plugin is applied.

Can't reproduce this with 2.1.6 running in gradle 4.0.1.
What version of macAppBundle are you using?

Closing as can't reproduce. Please reopen if you have more information.

First of all: sorry for the crappy original bug report. I should have at least mentioned that I was using 2.1.6 of the plugin.
Secondly, I really should have checked that this issue is easily reproducible, which it isn't. I'm still unable to reproduce this problem in a small project. Sorry again.
Thirdly, I still see the problem in my own build but I now I'm pretty sure that issue #120 of forbidden-apis is the likely root cause of it.
Shown below is another cut-down part of the deprecation warning stack trace:
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
at plugin-init$_run_closure3$_closure5$_closure6$_closure9.doCall(plugin-init.groovy:49)
at de.thetaphi.forbiddenapis.gradle.CheckForbiddenApis_Decorated.getClassesDir(Unknown Source)
at plugin-init$_run_closure3$_closure5$_closure7.doCall(plugin-init.groovy:55)
at org.gradle.api.internal.project.DefaultProject.allprojects(DefaultProject.java:620)
at org.gradle.api.internal.project.DefaultProject.getTasksByName(DefaultProject.java:795)
at org.gradle.api.Project$getTasksByName$3.call(Unknown Source)
at edu.sc.seis.macAppBundle.MacAppBundlePlugin.apply(MacAppBundlePlugin.groovy:84)
at edu.sc.seis.macAppBundle.MacAppBundlePlugin.apply(MacAppBundlePlugin.groovy)
at build_4cibqupx9n0pz46b4w4zdy069$_run_closure10.doCall([..]/lilith/build.gradle:348)
at build_4cibqupx9n0pz46b4w4zdy069.run([..]/lilith/build.gradle:345)

build.gradle:348 contains the apply plugin: 'edu.sc.seis.macAppBundle' line in my build file so I assumed the warning was caused by applying your plugin, providing the (what I thought) relevant parts of the stack trace.
Unfortunately, I missed the line de.thetaphi.forbiddenapis.gradle.CheckForbiddenApis_Decorated.getClassesDir(Unknown Source)  above those which I only just found while trying to produce a small project showing the issue at hand. Because of this, I'll stop searching for a reproducible testcase and will instead wait and see if this issue vanishes when the forbidden-apis plugin has been fixed - which I'm quite sure it will.
I just wanted to let you know all this so it won't occupy any of your neurons unnecessarily.
Cheers and thanks for taking a look at this!

I have a pull request for forbidden apis already: https://github.com/policeman-tools/forbidden-apis/pull/124


Failed to connect to VSCode

Failed to connect to VSCode !data.ok
Machine ID: 7424abfc-640a-44e7-8aaf-6c1271d69a6a

[!NOTE]
Book a call

key
value

id
f51k7m4q0z1719596449261

url
http://localhost:5173/signin

wifi
4g

cpu
4

mem
8

gpu
ANGLE (Intel, Intel(R) HD Graphics 630 (0x0000591B) Direct3D11 vs_5_0 ps_5_0, D3D11)

time
1719596452528

items
18

bytes
638

Hi @hamzamohamed1911, we’ve just deployed a fix for you! Can you try running the setup script npx million@latest. Once it's done, please restart your VSCode/Cursor. Let me know if that works for you!


Added axis parameter to tf.losses.softmax_cross_entropy. Fixes #20866

The internal implementation of tf.losses.softmax_cross_entropy, uses nn.softmax_cross_entropy_with_logits_v2 which has a dim argument with a default value of -1. The new argument axis=-1 wraps the dim argument of nn.softmax_cross_entropy_with_logits_v2. Documentations also updated.
This will fix #20866

Nagging Assignee @aaroey: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aaroey: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aaroey: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aaroey: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aaroey: It has been 74 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @aaroey: It has been 89 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I had to force push because I did a rebase to upstream. Hope that doesn't cause any issues. I am running the bazel build tensorflow/tools/compatibility/update:generate_v2_reorders_map once done i will push the changes.

I still have an issue running bazel-bin/tensorflow/tools/compatibility/update/generate_v2_reorders_map
It not only fails on my run but even the tensorflow/master but it fails on tensorflow/master as well. (I tried out of curiosity) with the following error :
Traceback (most recent call last):
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/tools/compatibility/update/gen
erate_v2_reorders_map.py", line 141, in <module>
    app.run(main=main)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/python/platform/app.py", line
40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tol
erate_undef)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/absl_py/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/absl_py/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/tools/compatibility/update/gen
erate_v2_reorders_map.py", line 137, in main
    update_reorders_v2(_OUTPUT_FILE_PATH)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/tools/compatibility/update/gen
erate_v2_reorders_map.py", line 133, in update_reorders_v2
    file_io.write_string_to_file(output_file_path, renames_f
ile_text)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/python/lib/io/file_io.py", lin
e 351, in write_string_to_file
    f.write(file_content)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/python/lib/io/file_io.py", lin
e 108, in write
    self._prewrite_check()
File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/python/lib/io/file_io.py", lin
e 94, in _prewrite_check
    compat.as_bytes(self.__name), compat.as_bytes(self.__mod
e), status)
  File "/home/ashar786khan/tf2/tensorflow/bazel-bin/tensorfl
ow/tools/compatibility/update/generate_v2_reorders_map.runfi
les/org_tensorflow/tensorflow/python/framework/errors_impl.p
y", line 548, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: third
_party/tensorflow/tools/compatibility/reorders_v2.py; No suc
h file or directory

@coder3101 can you please resolve conflicts

@rthadur Fixed

This change looks fine to me. But my understanding, it might need to be cherrypicked into 1.14 at this point since it changes API (adds an argument in the middle of function signature).
@josh11b @martinwicke Should we accept and cherrypick this change or instead ask author to add axis argument at the end?

My reply above was incorrect.
Actually, all the current losses under tf.losses are deprecated in 2.0. Instead, They are replaced by losses from tf.keras.losses.
axis would need to be added here instead: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/losses.py#L405


Allow passing raw string (without dumping) to ServerSentEvent

🚀 Feature Request
Hey! It would help me if I could pass a string to ServerSentEvent that won't be dumped but will be sent as is [so it won't be wrapped in quotes) - right now my workaround is creating a special class for said strings so I can hardcodingly return a raw string via overriding JsonSettings.dumps.

Can I try contributing a PR to allow this?


Metrics not appearing in Mongo

Using MongoObserver, I ran into an issue where metrics logged via log_scalar did not appear on the run in Mongo. I'm using a local database, the master branch of sacred, and Python 3.6.5.
I pared my experiment down to the following script, which exhibits the issue:
from sacred import Experiment
from sacred.observers import MongoObserver
import uuid

run_name = f"debug-{uuid.uuid4()}"
ex = Experiment(name=run_name)
ex.observers.append(MongoObserver.create())

def run():
    for i in range(2):
        ex.run()

@ex.main
def evaluate_mil(_run):
    _run.log_scalar("foo", 1)
    result = {"a": {"foo": [[0, 1, 2], [2.3, 1.2, .3], [0, 1, 1], 1]},
              "b": {"foo": [[3, 1, 2], [5.3, 1.2, .3], [0, .5, 1], .5]},
              "c": {"foo": [[3, 1, 2], [5.3, 1.2, .3], [0, .5, 1], .5]},
              "d": {"foo": [[3, 1, 2], [5.3, 1.2, .3], [0, .5, 1], .5]},
              "e": {"foo": [[3, 1, 2], [5.3, 1.2, .3], [0, .5, 1], .5]},
              "f": {"foo": [[3, 1, 2], [5.3, 1.2, .3], [0, .5, 1], .5]},
              "g": {"foo": [[3, 1, 2], [5.3, 1.2, .3], [0, .5, 1], .5]}}
    for rater in result:
        for name, metrics in result[rater].items():
            fprs, tprs, thresholds, auc = metrics
            for fpr, tpr, threshold in zip(fprs, tprs, thresholds):
                _run.log_scalar(f"{rater}-{name}-fpr", fpr)
                _run.log_scalar(f"{rater}-{name}-tpr", tpr)
                _run.log_scalar(f"{rater}-{name}-threshold", threshold)
            _run.log_scalar(f"{rater}-{name}-auc", auc)
    # import time
    # time.sleep(20)

if __name__ == "__main__":
    run()

When running this script, the first experiment receives no metrics, and the second is missing some (I expected 29 metrics per run with this setup):

Some (but not the expected 29) metrics for both runs do appear in the metric collection in Mongo, and correctly refer back to runs 27601 or 27602.

Uncommenting the time.sleep() call resolves the issue. As well, reducing the number of entries in the result dictionary resolves the issue.

I think I am encountering the same problem.
All I do is add a MongoObserver and call Experiment.run with different configs a couple of times. Here's the code.
I initially reported an issue with Omniboard where metrics were not showing up in the list of runs. The Omniboard maintainer tracked it down to the references to the metrics collection missing from an affacted run's info.
Meanwhile I've simultaneously been logging to neptune.ai, which has worked flawlessly.


Add into_builder next to builder method

In my code, I set the non-pointer members of Vulkan structs using the struct notation directly, before setting the pointers at the end. Because of the safety advantages, I would like to set pointers using the builder, but I find it unnecessary to set everything else using the builder as well. So I propose adding a method into_builder alongside the existing builder method, that unlike the latter takes self by value and uses it as a basis for creating the new builder. Using this, I could create a struct by directly setting members, convert it into a builder, and use that to set the pointer members. If you think it's appropriate, the into_builder method could set any pointer members to null explicitly so that the lifetimes are correct.

See also #441.

The builders are now gone and the builder helper functions are now implemented directly on the Vulkan structs since #602, so this should work OOTB now :tada:


Support for Dreame Robotic Mower A1

Dreame added new Robotic Mower named A1 to the line-up.
Can you please add support for that? If needed, I would be willing to help testing.
Thanks in advance

It was worth a shot - I thought since that thing is also equipped with a lidar sensor it might be not that time consuming. But thanks for considering.


Add logModuleLoadEvents launch option configuration

Add flag to disable logging module load events to the console. Since everything (including debuggee output) is logged to the debug output window in normal scenarios, module load events can cause the log to become cluttered fairly quickly. Setting this flag to false in launch.json will remove the module load events from the log.

@caslan @gregg-miskelly @chuckries

Instead of doing this, should we have a section for controlling output options in general? i.e. Exception messages, step filtering messages, module load / unload messages, thread exit messages, program output

I'd be fine with that, either way this option needs to be in both sections, launch and attach.

:+1:

:shipit:

NOTE: You might want to hold off actually completing this PR until post escrow when you can insert a new OpenDebugAD7 that has your changes. Though if you really would like to complete it now I will not object as it doesn't break anything - it just will not actually work.

@jacdavis FYI in case you want this in the C++ extension also

LGTM

@gregg-miskelly RE: your note. I'll hold on to this PR for now; it looks like there's a few more logging features planned so it makes sense to wait and add them in one go.

@caslan @gregg-miskelly @chuckries Here's the additional flags.

LGTM

LGTM

Hi @wesrupert, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla2.dotnetfoundation.org.

TTYL, DNFBOT;


Use node 22 via nvm in docker. Enable NODE_COMPILE_CACHE

Fixes #1014
Fixes #1037
@setchy @heubeck Could you kindly test this branch and let me know if you see any difference with node 22? The server mode needs testing as well with the new container image from this branch.

Merged since the fix got tested by another user.


Add HQQ support

Adds support for HQQ quantization without using the hqq lib.
The quantizer in prototype/hqq/core.py can output quantized tensors (W_q, scale, zero) that are compatible with AffineQuantizedTensor by setting raw_output=False (default value).
There's a simple example in prototype/hqq/example.py.
Missing:

Full model quantization and eval
Hqq-mix variant (would a new layout + a new affine quantized tensor)
Tests

Note:
The dequantized output produced by AffineQuantizedTensor is a bit worse than that produced by the hqq-lib. You can check that by setting raw_output=True. The problem has to do with the midpoint used in the dequantization logic by AffineQuantizedTensor which produces zero-point values with very low magnitude.

Llama3.1 8B Instruct lm-eval, the individual scores for each benchmark are a bit different (probably because of the mid-point thing), but the average score is the same

The problem has to do with the midpoint used in the dequantization logic by AffineQuantizedTensor which produces zero-point values with very low magnitude.

this is talking about accuracy? I can see you'd need this conversion for int4 tinygemm, but it's not required for other cases, if this is significant, maybe you can expose raw_output as a flag to the top level quant_api for hqq

@jerryzh168 we can keep it this way for the moment. The other solution would be to modify the alternate minimization and update the quant parameters based ont he mid-point format instead of (Wq - zero)*scale. For the moment, the average lm-eval is about the same as hqq-lib, so I guess it's mostly fine. If there are some performance issues I can get back to it and try to change the steps to improve performance with the mid-point approach which is needed for using the tinygemm kernel.

Sorry for the delay, I was caught in other tasks. I will take a look at your comments now

@mobicham thanks, for tests right now we support both unittest.TestCase and pytest (e.g. https://github.com/pytorch/ao/blob/174e630af2be8cd18bc47c5e530765a82e97f45b/test/dtypes/test_uintx.py#L39), what you have in the file makes sense I think, since you have hardcoded errors for each dtype I feel it's fine to just separate separate them into different tests

@jerryzh168 anything missing for the merge?

IIRC there was some test failure, just retriggered CI if all g ill merge

@msaroufim the tests fail because of triton related to this. I was about to delete torchao.prototype.hqq.core.py since we move the hqq core stuff in quantization, but just saw this pr uses it : https://github.com/pytorch/ao/pull/679
  test/hqq/test_hqq_affine.py:3: in <module>
      from torchao.prototype.hqq.core import HQQQuantizer
  /opt/conda/envs/venv/lib/python3.9/site-packages/torchao/prototype/hqq/__init__.py:1: in <module>
      from .mixed_mm import triton_mixed_mm, pack_2xint4
  /opt/conda/envs/venv/lib/python3.9/site-packages/torchao/prototype/hqq/mixed_mm.py:2: in <module>
      import triton.language as tl
  E   ModuleNotFoundError: No module named 'triton'

I do see some legit looking errors for GPU https://github.com/pytorch/ao/actions/runs/10403091060/job/28816346879?pr=605#step:12:2570
And for CPU not sure I follow the connection to the version guard fix PR, basically we need to ensure HQQ Quantizer doesnt get imported by accident on CPU machines that way we can be sure triton is there

It looks like it's the tensorcore test that is failing not the rest:
  test/hqq/test_hqq_affine.py::TestHQQ4bit::test_hqq_tensorcore_4bit FAILED

It was working fine just 2 days ago I think, let me clone and recheck on an instance

And for CPU not sure I follow the connection to the version guard fix PR, basically we need to ensure HQQ Quantizer doesnt get imported by accident on CPU machines that way we dont crash on the triton import

It's not supposed to be imported, I wanted to delete torchao/prototype/hqq/core.py because we move all the stuff there to torchao/quantization/quant_primitives .py. Will take a look at it


[CMakeLists] Use unique  for check_cxx_source_runs test

What's in this pull request?
CC: @gribozavr
As discussed in https://github.com/apple/swift/pull/2012#discussion_r60973375
CMake check_{c,cxx}_xxxx family macros cache the result associated
with result var name. That means every tests must have unique names.
Apparently, as for now, this is the only code that has this problem.
But in AddSwift.cmake, or SwiftList.cmake, we have code like:
function(_list_add_string_suffix input_list suffix result_var_name)
  set(result)
  ...
  set("${result_var_name}" "${result}" PARENT_SCOPE)
endfunction()

Since set(<var>) does NOT hide the cached value,
If check_working_std_regex success, it would not work, I think.
Test CMakeLists.txt:
cmake_minimum_required(VERSION 2.8.12)

set(result "CACHED_RESULT" CACHE INTERNAL "RESULT")

function(my_func result_var_name)
    set(result)
    list(APPEND result "BAZ!!")
    set("${result_var_name}" "${result}" PARENT_SCOPE)
endfunction()

my_func(MY_FUNC_RESULT)
message("MY_FUNC_RESULT: ${MY_FUNC_RESULT}")

Output:
MY_FUNC_RESULT: CACHED_RESULT;BAZ!!

In theory, to begin with empty value, we should write: set(var_name "") instead of set(var_name).
But it's too annoying...

Before merging this pull request to apple/swift repository:

[ ] Test pull request on Swift continuous integration.

Hmm... in My Ubuntu 15.10 emits different result between
check_cxx_source_runs("${std_regex_test_source}" result) and
check_cxx_source_runs("${std_regex_test_source}" REGEX_TEST)
...
I will look deeper into it.

OK, it turned out that check_cxx_source_runs invokes the compiler with -D<result_var>.
/usr/bin/clang++  ... -Dresult ... CMakeFiles/CMakeTmp/src.cxx

Obviously, it conflicts with other definitions in libstdc++ headers.
It seems that's why check_cxx_source_runs("${std_regex_test_source}" result) fails.

Anyway, the patch is still ready to merge.
And this enables test/IDE/coloring.swift on Linux - at least Ubuntu 15.10.

@swift-ci Please test and merge

It is really bad that CMake is so hostile to abstractions.

Could someone please re-run test and merge this?

@swift-ci Please test and merge

CI issue was unrelated.


Discussion on separation of Event, Monitoring, and Service/header data

The Problem:
The concept of an EventSource has been quite successful and easy to use in ctapipe, since it allows one to write clear loops over events, where the event data is part of an iterable object. However, we now need to think about other data streams that are not tied directly to each Cherenkov event.
When processing data, we don't only need the data for an event, but several other pieces of information

Monitoring data, which includes things like the pointing position of each telescope, or various calibration coefficients that change on ~ 1s timescales.
Service Data

Instrument description, which is fixed during an observation (e.g. the SubarrayDescription)
Observation configuration: the "run header" information associated with an observation
Simulation Configuration: the "mc header" information associated with an observation that comes from a Monte-Carlo simulation, which defines the monte-carlo parameters

These info come at different rates than the events (much more slowly).  However, some form of synchronization to the event data is needed.
Special design considerations
for Monitoring data
The monitoring (MON) data comes rates around 1HZ, but it is not necessarily correct to simply synchronize the "latest" MON data with the current event: normally one would like to do some interpolation, so how the monitoring data is accessed may need some thought.  This means there may be two types of MON data:

the "raw" MON data that is just a measurement each second
interpolated MON data that has the correct value at the exact instance of the event time.

These do not need to be stored in the same way, e.g. one might first read the raw MON data, construct an interpolator, and then use it to fill event.mon info (event-wise interpolated MON data)
Note also that there are also two further types of MON data:

MON  data read from a pre-existing file (which could be simply loaded, and an interpolator function constructed, so that one can generate per-event interpolations)
MON data generated during the event loop (e.g. pedestals), so there is no pre-existing file to read and interpolate -- must be done on-the-fly

For Service Data:
In general these are all static information for a given observation, but in the case of Monte-Carlo files, it is possible for these to change in the middle of a file (e.g. if multiple runs are merged).  The question is whether or not we support these changing for a single EventSource loop, or assume they are header info that is read once at the start.
Parallelization and distributed processing
Right now parallelizing the event loop (e.g. some_parallel_system.map(processing_function, source)) means sending around all of the event, mon, and service data every time an event changes (even though the mon and service info doesn't change).  This is probably very inefficient, since that is a lot of data to transmit.  Separating the streams makes this much cleaner.
Implementation Options
Most of these options involve generalizing EventSources, e.g. to have the EventSource also synchronize MON and SVC data.
This could be done in a few ways:
Option 1
a higher-level DataContainer that contains data.event, data.mon and data.service splits (e.g. see #1040).  One issue with this is that semantically we are no longer looping over "Events", but we could stipulate that the event part is the one to which all others are synched, so it still make some sense, even if it's clunky.
for data in source:
     print(data.event.mc.energy)
     print(data.mon.tel[i].pointing.az) # this is just the  "latest?"
     print(data.event.mon.tel[i].pointing.az) # interpolated
     print(data.service.subarray.tel_coords)

Option 2
we keep only the event part in the iterator loop of the EventSource, and have the svc data accessed from the source itself
for event in source:
    print(event.mc.energy)
    print(source.service.subarray.tel_coords)
    print(source.mon.tel[i].pointing.az(event.time))
    print(event.mon.tel[i].pointing.az) # interpolated to event time

In this case, it's not clear to the user that the monitoring and service data are synced. E.g. the the user needs to re-read the service data if it changes.
Option 3
we always return all three data streams separately:
for event, mon, svc in source:
    print(event.mc.energy)
    print(mon.tel[i].pointing.az)
    print(svc.subarray.tel_coords)

Here, it's clear that you can pass each one to further algorithms, and you might get a pointer to the same info , or new info, and it shouldn't matter.
Option 4
we handle MON and SVC data separately  (probably a bit of an ugly solution, and not clear if this would work for MON data generated during the event loop)
mon_source = MonitoringSource().from_url(x)
event_source =  EventSource().from_url(x)
svc_source = ServiceSource().from_url(x)

Here, it's not quite clear what the event loop(s) should look like.  I guess only EventSource would provide an iterator, and the others would just be interpolators or static info.
Any other ideas?

One option I have is that the EventSource is no longer responsible for creating the DataContainer. Instead it is part of the chain responsible for filling the DataContainer, as are other Components. This makes this option somewhat a combination of option 1 and option 4.... but its probably actually close to something you intended with option 4.
data = DataContainer()  # Create the container
event_source = EventSource().from_url(x)
mon_source = MonitoringSource().from_url(y)
svc_source = ServiceSource().from_url(z)
for _ in event_source:
   event_source.load_event(data)  # Fill the DataContainer with the event
   mon_source.load_event(data)  # Fill the DataContainer with the monitor information corresponding to the event
   svc_source.load_event(data)

In this implementation, the MonitoringSource and ServiceSource requires the event information to be loaded into the data container before the monitor information corresponding can be loaded (+ interpolated) into the DataContainer. Perhaps moving the DataContainer creator outside of the EventSource then does not matter.
I guess the idea here is that the highest rate the DataContainer is updated is defined by the event rate, therefore the iterator should be over the event source, and the other sources act accordingly to the current event.
The advantage of this over option 1 is ONLY that the class responsible for filling the event information is separate to the one responsible for filling the monitor information. Though really the same could be achieved with a higher level class that contains all three of the data-filling classes.

I guess in that example, it might be nicer to have the "load_event" happen automatically in the event_source iterator, but the others be explicit (since otherwise what is the point of the iteration)

I am not a fan of option 2 - accessing monitor information from a component becomes difficult, unless its explicitly passed. And has the potential for unclarity.

Though I don't like that the user has to do so much just to get the data, it should be made as fool-proof and automatic as possible.  However, I do like the idea that filling monitoring info could be just a Component (which could interpolate or otherwise)

I am not a fan of option 2 - accessing monitor information from a component becomes difficult, unless its explicitly passed. And has the potential for unclarity.

agreed - it's also not so easy to allow for changing runs

Though I don't like that the user has to do so much just to get the data, it should be made as fool-proof and automatic as possible. However, I do like the idea that filling monitoring info could be just a Component (which could interpolate or otherwise)

Then it sounds like the way to go with this option should be:
class DataFiller:  # Name TBD
   def __init__(self, event_url, mon_url):
      self.event_source = EventSource(event_url) # Class responsible for only filling the event container
      self.mon_source = MonitorSource.from_url(mon_url) # Class responsible for only filling the monitor container
      # ....other sources

   def __iter__():
       data = DataContainer()
       self.event_source.fill_container(data)
       self.mon_source.fill_container(data)  # Interpolates according to the current event in the container

data_filler = DataFiller(event_url, mon_url)
for data in data_filler:
        print(data.event.mc.energy)
        print(data.mon.tel[i].pointing.az) # this is just the  "latest?"
        print(data.event.mon.tel[i].pointing.az) # interpolated
        print(data.service.subarray.tel_coords)

Alternatively the DataFiller could be defined as:
class DataFiller:  # Name TBD
   def __init__(self, event_url, mon_url):
      self.event_source = EventSource(event_url) # Class responsible for only filling the event container
      self.mon_source = MonitorSource.from_url(mon_url) # Class responsible for only filling the monitor container
      # ....other sources

   def __iter__():
       for data in self.event_source:
           self.mon_source.fill_container(data)  # Interpolates according to the current event in the container
           yield data

As it doesn't seem to make sense to skip the event_source step anyway. Unless we had simulators we may wish to use....

Hi,
it seems to me that  an import element which steers the choice is  the mechanism used to synchronise the event and mon containers (svc is probably not going to change inside a run).
The simpler solution at the moment (where not data base is available) is to connect them by timestamps and to perform the connection at the event generator level: each event must know the time stamp of its correct mon container and in case the present mon container is outdated the generator is simply reading the correct one, that (in absence of a fast random file access mechanism it should be just the next mon event in the file).
So, given the present situation (where archive tools are still rather absent), I would prefer to leave a bigger role to the event source:

it should read the event, mon, svc streams from one or several files  (for example, in LST we are probably going to put everything in the same file, that makes book-keeping easier).
create the general container
update the all the three levels in the event generator in such a way: read a new event always but  read the mon and/or svc containers  only if the timestamps and/or version of the new event do not correspond any more to the values in memory. In the case of mon event interpolation one could think to a more complicated reading mechanism for the mon event, but this is perhaps too early now

So probably this is more option 1?

One thing we can start is by just agreeing what the monitoring data is: is it per-event (e.g. some slower monitoring stream that has been interpolated to be the right value for an event), or does it change slowly.
I kind of prefer the event-wise monitoring, since we can then read the "slow" monitoring stream in any way we like, and just update the event version for each event using a Component.  In that case, we only have event.mon (as now), and it really is different per event.  If we want to write it out again, of course, some down-sampling in time would be necessary.
I could imagine the loop looking like:
# reads a file and just constructs an interp function for each telescope
pointing = PointingInterpolator.from_file("tracking.hdf5")

# Pedestals are computed on-the-fly, not from a file, so each time it is called, it might
# measure some data from the event, update the pedestals, and output the latest per-event version
pedestals = PedestalInterpolator()

for service, event in event_source:
    event.mon.pointing = pointing(event.time)
    event.mon.pedestals = pedestals(event.time, event.dl1)

For things like the pedestals, we also have to think of adding an event buffer of some kind, so it can essentially be n events ahead of the reconstruction (e.g. so that it can already gather enough statistics to compute the pedestals for the first few events).  Or we always do multi-pass processing, where we first compute pedestals and gains, and then in a second pass read them in (as with the pointing example above), and loop a second time to do apply the calibration and do the reconstruction

The on-the-fly approach means we do not need to try and match the calibration to the events, and we instead rely on the latest calculated values. It also avoids having to have two loops through the data stream (assuming the calibration events are interleaved among the cherenkov events).
I believe the workflow Franca has been working on uses the on-the-fly approach... she utilises the concept of interleaved calibration (pedestal & flat-field) events, and updates the calibration coefficients as she progresses through the observation. She uses a flag inside the event container to identify the type of event.
In Franca's design, there is also the possibility of creating a different calibration-calculator subclass that instead reads the correct calibration coefficients from a file (perhaps created with a separate pipeline, i.e. the two-pass approach).
In the case of a on-the-fly calibration, it might make sense that at the start of each observation, a bunch of calibration events are taken, to build up the statistics of the calibration. This means the same, single pipeline can be relied upon. Alternatively, the class loads the latest coefficients to begin with. The great thing about Franca's design is we are not restricted in how the calibration is performed - as long as it obeys the principle of filling the mon container ready to be applied in the event pipeline.
For things like the pedestal and flat-field (and other calibration parameters), I don't think we should force it to be written to the event/data container at the start of the pipeline. This would complicate how we handle the calibration. These coefficients depend on factors such as the charge extraction method that is used. Therefore, it would be better that the coefficients are calculated via following the normal pipeline, up to charge extraction, and then uses the result of the charge extraction to populate the calibration coefficient calculation classes. Ensuring compatibility.
The following is how I imagine a on-the-fly calibration would work:

The calibration events are identifiable with a flag
The calibration events follow the same pipeline as observation events
The last stage of the pipeline for a calibration event is filling the calibration container, after the charge extraction step
If the event is instead an observation event, the calibration coefficients are instead stored into the data container by the calibration calculation class. This event does not contribute to the coefficients itself.
The observation event continues down the pipeline, where a separate class applies the calculated calibration coefficients stored in the event container.

Similarly for a two pass calibration, the calibration events would follow the same pipeline until charge extraction. Then the values are extracted and stored.
This approach ensures that the correct calibration coefficients are used for the specified pipeline configuration. I think it also makes the pipeline easy to follow, and reduces the chance for mistakes.
Regarding the decision between event.mon or data.mon, I am more inclined to the second. It may be that the calibration coefficients are updated at the same rate as event...but only if the telescope chooses to do such a thing as interpolate or utilises part of the waveform without signal. There is a possibility that the telescope chooses to only update the calibration every second. As they have deemed this as enough and see nothing gained with the complication of interpolating between calibration coefficients. There is nothing wrong with the data.mon being updated at the same rate as data.event, but it is not something we should force - we should stick to the original reason for the separation of event and mon, which is that there is the possibility of them being updated at different rates.

Thanks for this precise description @watsonjj, indeed this is what we are planing for LST online calibration. In this discussion I think we should remember that calibration will be an iterative process: the on-fly calibration is meant for feeding the RTA pipeline and producing the first DL0/DL1 data to be written on disk. Most probably, there will be afterwards a more fine calibration based on a multi-pass process and interpolation for a optimised quality. Therefore, we should think to a data model that is suitable for both approaches.
My major concern at this point is that for the prototype data we should be able (now!) to easily connect the calibration data to the event data without having the possibility to inquire an archive system that is not available yet. Therefore, whatever we chose we should think that the connection will be done (at least for a while) at the ctapipe code level. For that, as a first guess I thought to give to the event_souce the role of "connector"  and then option 1 seemed the most natural to be used. But the DataFiller proposed by Jason above (second option?)  will also play the role.
The message here: together with the data model we should think that we need a data level "connector" in order to build something immediately usable by the prototypes.

Hi,
starting again this discussion on the base of the example proposed by Jason above (that seems a good option to me):

class DataFiller:  # Name TBD
   def __init__(self, event_url, mon_url):
      self.event_source = EventSource(event_url) # Class responsible for only filling the event container
      self.mon_source = MonitorSource.from_url(mon_url) # Class responsible for only filling the monitor container
      # ....other sources

   def __iter__():
       data = DataContainer()
       self.event_source.fill_container(data)
       self.mon_source.fill_container(data)  # Interpolates according to the current event in the container

data_filler = DataFiller(event_url, mon_url)
for data in data_filler:
        print(data.event.mc.energy)
        print(data.mon.tel[i].pointing.az) # this is just the  "latest?"
        print(data.event.mon.tel[i].pointing.az) # interpolated
        print(data.service.subarray.tel_coords)

The DataFiller could be a plugin Component as the EventSource class now, so that for raw (or specific) data analysis each telescope can define its own DataFiller and eventually DataContainer (inherited by the general DataContainer), e.g:
class LSTDataContainer(DataContainer):
    """
       Data container including LST information
    """
        lst = Field(LSTContainer(), "LST specific Information")

This would permit an immediate use of this approach with the prototype data.
For MC data and higher data levels a more telescope independent DataFiller should be provided directly in ctapipe. What do you think?

A couple of comments:

I think the "camera event" (be it triggered by a shower, a flatfield event, or a pedestal event) should be the basic step in loops, so I like the "combination of 1 and 4" proposed by Jason.

In general I agree with the on-the-fly calibration approach suggested by Jason, which is actually similar to what we do in MAGIC. Taking a bunch of flatfield and pedestal events for a few seconds before starting with shower events should indeed be the standard data taking procedure.

We should however keep in mind that for real-time analysis we will have to parallelize the processing of the events, sharing it among a few "workers". In that situation the on-the-fly calculation has a problem: if each worker only receives a fraction of all events, including the flatfield and pedestal events, then the precision of the calibration will be worse (or its update rate smaller), and on top of that, the calibration coefficients will be different for each of the workers - a somewhat ugly feature. It would be better that all the workers use a common calibration. Hence I think that the system must allow for the reading of a mon stream with precalculated calibration coefficients, and synchronize it via timestamps (same as e.g. the pointing) with the shower events to be calibrated.

Of course the calibration mon data (or rather, svc) should include everything needed, e.g. the waveform extractor name and parameters, to make sure the coefficients are never wrongly applied with a different combination of those.

I am not sure I understood the question of "data.mon vs. event.mon". Isn't the latter just the event-wise, interpolated version of the former? Probably a bit lighter, keeping only what is strictly needed for the analysis from the given data level onwards. I think that both can coexist.

Franca, what is the part you want to move to a telescope-specific plugin component? Just the generation of the "data.mon"? The interpolation (or any other simpler approach to fill mon.event with reasonable values) should be part of ctapipe, right?

Franca, what is the part you want to move to a telescope-specific plugin component? Just the generation of the "data.mon"? The interpolation (or any other simpler approach to fill mon.event with reasonable values) should be part of ctapipe, right?

@moralejo, I think you mis-understood me.  I will reply to the question explaining which my problem right now:
I would like to treat MC calibration data (flat fields and pedestals) in the same chain of the real data. For that the mon.data containers must not be initialised by the event reader (as it is now in the LST io plugin) because the event reader of the MC data is different from the one of the real data. Therefore we need a DataFiller (or an other solution) that creates and fill the data model choosing the right sources, which depend both from data type (MC or real) and telescope type.  This DataFiller could be a Component class  in ctapipe (and each telescope should probably create its own subclass), or a plugin in the ctapipe_io of each telescope, I don't know what is better...
The point here is: for LST (and NectarCAM, which will take data in June) is urgent to converge rapidly with this discussion and start to implement the solution that we think the best. This will permit also to push in clean environment the monitoring containers in ctapipe and allow their use, test and improvement by other telescopes.

I would like to treat MC calibration data (flat fields and pedestals) in the same chain of the real data. For that the mon.data containers must not be initialised by the event reader (as it is now in the LST io plugin) because the event reader of the MC data is different from the one of the real data. Therefore we need a DataFiller (or an other solution) that creates and fill the data model choosing the right sources, which depend both from data type (MC or real) and telescope type.

Ok, thanks for the clarification. If we follow the same approach as for the "event" data, then the creation and (partial) filling of the standard mon containers for the case of the MC should be done by pyeventio, and not by a telescope-type-specific plugin. Or do you prefer to separate the creation  of the mon containers from the io machinery? Probably for the MC that would be a complication, since the mon info (including calibration mon info, in the case you want to use the one provided by simtelarray instead of calculating it) is read from the simtel files.

This DataFiller could be a Component class in ctapipe (and each telescope should probably create its own subclass), or a plugin in the ctapipe_io of each telescope, I don't know what is better...

For the real data you say that the mon containers are initialized by the LST io plugin, and that should still be the case. What is missing, I guess, is to add the code to fill them (with whatever is not to be calculated during the analysis) - once we will know from where we will retrieve e.g. the LST-1 pointing information etc.

Probably for the MC that would be a complication, since the mon info (including calibration mon info, in the case you want to use the one provided by simtelarray instead of calculating it) is read from the simtel files.

To clarify again: what I want to is to test the LST calibration procedure with MC calibration data (flat-flied and pedestals), therefore I need to access and use the mon containers with  MC calibration data exactly I  do with  real data (even if obviously in the case of MC data the events will be read by pyeventio).

For the real data you say that the mon containers are initialized by the LST io plugin, and that should still be the case. What is missing, I guess, is to add the code to fill them (with whatever is not to be calculated during the analysis) - once we will know from where we will retrieve e.g. the LST-1 pointing information etc.

For the moment the mon containers ARE in the LST plugin, so we don't have other choses.  When they will be moved in ctapipe, we can indeed keep the initialisation in each telescope plugin by mean of a DataFiller plugin or similar. @kosack what do you think? Could we perhaps plan a discussion on this point the next ctapipe meeting (after the next ASWG)?

Comments from #1157
if we want to follow the current data model I presented in Bologna, we should really separate four things:

metadata headers:

simulation (what is in the MCHeaderContainer + other simulation header info, like MCThownShowerDistribution, etc)
observation (right now we don't have this in sim data, but we will need it)
instrument: at least the SubarrayDescription, other things?

event data
monitoring data (things that change far slower than events, but still change)
service data (things that do not change during an observation, but may be needed for analysis)

As discussed a bit in  #1041, at least some of this should be loaded automatically by the EventSource (at least event + the header info, perhaps not the monitoring or service by default since those can be loaded later on demand).
So if you want to implement all that here, let's decide on what it should look like.  i.e.:
source = EventSource(input_url=x)

In fact-tools, we have a Service approach to this. These are basically classes that provide information given some property. E.g. GainService gets a timestamp and returns gains for each pixel.
This would roughly look like this:
mon_service = MonService(data_files)
calibrator = CameraCalibrator(mon_service=mon_service)

for array_event in source:
    for tel_event in event.telescope_events:
        calibrator(event)

And inside __call__, CameraCalibrator would do something like this:
gains = self.monitoring_service.get_gains(telescope_event.telescope_id, telescope_event.timestamp)


Error: 404 Client Error: Not Found for url: http://localhost:11434/api/generate

What is the issue?
While trying to run llama model I am getting Error: 404 Client Error: Not Found for url: http://localhost:11434/api/generate.
I am running this in my windows 11 system with python 3.13.0 version
I have installed llama3.2:latest and it's running.
Code:
response = requests.post('http://localhost:11434/api/generate',
json={
"model": "llama3.2:latest",
"prompt": f"{system_prompt}\n\n{prompt}",
"stream": False
})
Error Output:
Ollama service is running!

Serving Flask app 'app'
Debug mode: on
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on http://<IP_ADDRESS>:3000
INFO:werkzeug:Press CTRL+C to quit
INFO:werkzeug: * Restarting with stat
Checking Ollama service...
Ollama service is running!
WARNING:werkzeug: * Debugger is active!
INFO:werkzeug: * Debugger PIN: 119-213-946
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:08:53] "GET / HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:08:53] "GET /api/history HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:08:53] "GET /favicon.ico HTTP/1.1" 404 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:08:55] "GET /api/health HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:08:55] "GET /api/history HTTP/1.1" 200 -
ERROR:main:Error in Ollama request: 404 Client Error: Not Found for url: http://localhost:11434/api/generate
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:09:01] "POST /api/chat HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:22:37] "GET / HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:22:38] "GET /api/history HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:22:40] "GET /api/health HTTP/1.1" 200 -
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:22:40] "GET /api/history HTTP/1.1" 200 -
ERROR:main:Error in Ollama request: 404 Client Error: Not Found for url: http://localhost:11434/api/generate
INFO:werkzeug:<IP_ADDRESS> - - [15/Nov/2024 23:22:49] "POST /api/chat HTTP/1.1" 200 -

I am able to curl http://localhost:11434 but not able to ping http://localhost:11434/api
curl http://localhost:11434
Ollama is running
What is the issue here? how to resolve this error.
OS
Windows
GPU
Intel
CPU
Intel
Ollama version
llama3.2:latest

What does the following return
curl -D - http://localhost:11434/api/generate -d "{\"model\":\"llama3.2:latest\",\"prompt\":\"hi\",\"stream\":false}"

@rick-github
C:\Users: curl -D - http://localhost:11434/api/generate
HTTP/1.1 404 Not Found
Content-Type: text/plain
Date: Fri, 15 Nov 2024 18:16:56 GMT
Content-Length: 18
404 page not found
C:\Users>curl -D - http://localhost:11434/api/generate -d "{"model":"llama3.2:latest","prompt":"hi","stream":false}"
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Fri, 15 Nov 2024 18:17:37 GMT
Content-Length: 470
{"model":"llama3.2:latest","created_at":"2024-11-15T18:17:37.8303989Z","response":"How can I assist you today?","done":true,"done_reason":"stop","context":[128006,9125,128007,271,38766,1303,33025,2696,25,6790,220,2366,18,271,128009,128006,882,128007,271,6151,128009,128006,78191,128007,271,4438,649,358,7945,499,3432,30],"total_duration":986768100,"load_duration":24535700,"prompt_eval_count":26,"prompt_eval_duration":373000000,"eval_count":8,"eval_duration":587000000}

ollama is working fine, something is wrong with your app or the environment your app is running in.  Are you using docker?  What do the ollama server logs show?

Copied your code into a file:
import requests

system_prompt = "You are an AI assistant"
prompt = "hi"

response = requests.post('http://localhost:11434/api/generate',
json={
"model": "llama3.2:latest",
"prompt": f"{system_prompt}\n\n{prompt}",
"stream": False
})

print(response.text)

Runs fine:
$ python 7689.py
{"model":"llama3.2:latest","created_at":"2024-11-15T18:36:48.900706688Z","response":"Hello! It's nice to meet you. I'm here to help answer any questions, provide information, or just chat with you. How can I assist you today?","done":true,"done_reason":"stop","context":[128006,9125,128007,271,38766,1303,33025,2696,25,6790,220,2366,18,271,128009,128006,882,128007,271,2675,527,459,15592,18328,271,6151,128009,128006,78191,128007,271,9906,0,1102,596,6555,311,3449,499,13,358,2846,1618,311,1520,4320,904,4860,11,3493,2038,11,477,1120,6369,449,499,13,2650,649,358,7945,499,3432,30],"total_duration":1773598547,"load_duration":16260172,"prompt_eval_count":32,"prompt_eval_duration":51000000,"eval_count":35,"eval_duration":1704000000}

@rick-github  yeah I can see  it is working in this way , But I am using this api to generate text to sql with flask web api, there I am getting 404 error.

So far I thought I am not able to connect to /api/generate but it is connecting. Will debug the whole code and get back to you.
I appreciate your quick assistance.

GET /api/generate isn't a valid endpoint, which is why it's returning the 404. Arguably it could return a 405 NoMethod instead of a 404.
The endpoint you're looking for is POST /api/generate. With curl the -d option is implicitly changing the request to a POST, but you can call it with curl -X POST /api/generate (which will of course generate a 400 error since it doesn't have a message body).
I'll go ahead and close the issue.

@pdevine running into this issue as well.  @kiran-kumar-akula seems to be referencing the POST method in their notes NOT the GET method.  Could this be reopened?

@eslynunez This was the command referenced:
curl -D - http://localhost:11434/api/generate

That will call GET /api/generate which will return a 404. Here's what happens if you add -X POST to that same call:
% curl -D - http://localhost:11434/api/generate -X POST
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Date: Tue, 19 Nov 2024 00:14:39 GMT
Content-Length: 32

{"error":"missing request body"}%

which is the expected output.

I missed that you are correct thank you, @pdevine

I think we should probably return a 405 Method Not Allowed error just to be more succinct.


Change LanesAtMost32 to LanesAtMost64 (or possibly remove the lane limit altogether)

We would like to support at least 64-lane vectors (to allow AVX-512 vectors of u8) but there is an aarch64 codegen issue (rust-lang/rust#84020).
Once that issue is solved we can increase the lane limit.

I believe we should consider an alternative to the use of actual numbers in the name if we are going to wind up changing them around based on these codegen vagaries.

since it's a dumb temporary limit just call it LanesAtMost

alternatively we can use a const fn and change the internal implementation without having to remove impls for things, which is still backwards-incompatible but in a different way! :D

I'm not actually positive it's a temporary limit, unless we're planning on requiring all backends to support up to usize::MAX lanes (or okay with monomorphization errors).  Also it's not clear what we're going to do about non-power-of-two lane counts.
But I agree about renaming it.  Maybe something more like ValidLanes?

OTOH, having the count in the name explains what you have to do to fix the error

We discussed this in the meeting and decided that since we really don't want to go below 32, because we consider AVX2 support important, it's fine to leave it with this name as it is.
Also, that the aarch64 codegen problem should be considered as blocking stabilization until we understand why.

Updated the title of this issue to reflect some expressed sentiment that maybe this trait shouldn't exist at all.
We should probably revisit in the future, but it may be appropriate to just produce a monomorphization error.  The compiler currently produces one if the size exceeds some implementation-defined value (65535 lanes, I think), but the error is not very descriptive.  If we can improve the error reporting, maybe we can remove this trait altogether.

@calebzulawski IIRC, monomorphization errors are undesirable. As I had understood it, I had thought rustc was developed in such a way as to totally avoid surfacing monomorphization errors to end users. But I don't have a ton of context on that. It might be wise to ping the compiler team on something like this. (If only to discover whether monomorphization errors are a "hard no" or not.)

Yeah, that would definitely be a good idea.  Realistically it's extremely unlikely to encounter the error in "normal" code, but if it's an absolute no we need to keep the trait.  We could also explore using a lane count type smaller than usize, though that seems hacky.

Actually, I forgot the trait is also preventing sizes that aren't a power of two... so fairly easy to get wrong.  Still worth discussing, I think.

well, technically arrays also have monomorphization errors: [0i32; !0] should fail to compile. I think a monomorphization error is fine for the case of a huge lane count, but for now we should keep LanesAtMost32 and rename it to LanesArePowerOf2 (or similar), and strive to eventually remove it completely (or just leave it as a trait you can use but remove it from all the bounds in core::simd, we aren't the only ones who like powers-of-2).

well, technically arrays also have monomorphization errors: [0i32; !0] should fail to compile

It does fail? https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=03be5c230f87259db3ed325707a24a44
Or are you saying that error is reported at monomorphization time? I can't tell.
In any case, what matters is whether the relevant teams consider that a bug or not (if it's being reported at monomorphization time). I don't know the answer to that question.

I think a monomorphization error is fine for the case of a huge lane count

As I said, the relevant teams (compiler?) should be consulted about this. cc @rust-lang/compiler

well, technically arrays also have monomorphization errors: [0i32; !0] should fail to compile

It does fail? https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=03be5c230f87259db3ed325707a24a44
Or are you saying that error is reported at monomorphization time? I can't tell.

I'm saying the error is reported at monomorphization time (or at least will be if sufficient obfuscating generics are added).

Yes, we shouldn't assume that existing behavior is intended and plow forward. That's not a good strategy for cooperation.

Yes, we shouldn't assume that existing behavior is intended and plow forward. That's not a good strategy for cooperation.

well, for arrays at least, I remember reading that the error on array sizes being more than isize::MAX is intended to be a monomorphization error, though I can't find a reference after some superficial googling.

@lqd is working on improving diagnostics during monomorphization time. They are undesirable though. We would definitely prefer these errors to be handled via bounds, but there is no consensus yet on the various aspects of that. Even with improvements to post monomorphization diagnostics, they have subpar UX. Check builds and IDEs won't show them. And they don't occur in libraries that define them, but on crates that use them.
Ideas for how to represent the bounds that you need and how to handle them in general are very welcome. If you come to the conclusion that post monomorphization errors are preferrable, that is also important to know.

I agree that the trait bound is the "rusty" way to do it.  The trait bound actually never comes up in the API in non-generic code.  In generics the problem is threefold:

The trait must be appropriately named to make it clear what the error is if you are missing the bound.
If we intend on changing the vector length limitations as the library improves, and change the name of the trait to match, the trait must be unstable.
If the trait is unstable, it's extremely hard to work with vectors in generics.

I think if we can come up with a name that is descriptive enough to produce a good error message, but flexible enough to stay the same as we reduce limitations (add non-powers-of-two, increase maximum length, etc), we can just stick with a trait.

SupportedLaneCount?

I mean... ideally you don't want a trait, but a formula, but const_evaluatable_checked is nowhere near ready. Neither impl, design nor consensus wise

Since this is just a true/false situation a trait might not be perfect but it's pretty good.  Once const_evaluatable_checked is done we can use it for a generic trait implementation instead of enumerating each valid type.

@workingjubilee @programmerjake I think we can close this now?

I think while we changed the name, the driving thrust of this ticket was actually only supporting lanes of a certain count? So we should reorient this instead.

Towards the end of the discussion I think we settled on using a trait to limit lane counts, rather than a monomorphisation error.

Closing in favor of #63.  We've settled on the LaneCount<N>: SupportedLaneCount bound (unless const generic bounds are implemented).  Also, 64-lane vectors are now supported.


materialize-css and bootstrap $colors variable collision

Problem

While using materialize-css along with Bootstrap 4 in sass mode, a collision occurs on the sass variable $colors.  Both frameworks have an overridable variable named $colors (Materialize-css have it in : materialize-css/sass/components/_color.scss and Bootstrap in bootstrap/scss/_variables.scss).
Importing the two frameworks together provoke a compilation error :
@import "../node_modules/bootstrap/scss/bootstrap";
@import "../node_modules/materialize-css/sass/materialize";

Compilation error :

$button-background-focus: lighten($secondary-color, 4%) !default;
                         ^
      Argument `$color` of `lighten($color, $amount)` must be a color

Backtrace:
        node_modules/materialize-css/sass/components/_variables.scss:60, in function `lighten`
        node_modules/materialize-css/sass/components/_variables.scss:60

This is due to the fact that $secondary-color: color("teal", "lighten-1") returns a null value since it look for the teal color in the Bootstrap $colors which does not exist.

It would be nice to prefix the name of  $colors variable by mc or something specific to your framework.
Possible Solution

A workaround is to redefine the $colors as below :
@import "../node_modules/bootstrap/scss/bootstrap";

@import "../node_modules/materialize-css/sass/components/color";
$colors: (
        "materialize-red": $materialize-red,
        "red": $red,
        "pink": $pink,
        "purple": $purple,
        "deep-purple": $deep-purple,
        "indigo": $indigo,
        "blue": $blue,
        "light-blue": $light-blue,
        "cyan": $cyan,
        "teal": $teal,
        "green": $green,
        "light-green": $light-green,
        "lime": $lime,
        "yellow": $yellow,
        "amber": $amber,
        "orange": $orange,
        "deep-orange": $deep-orange,
        "brown": $brown,
        "blue-grey": $blue-grey,
        "grey": $grey,
        "shades": $shades
);

@import "../node_modules/materialize-css/sass/materialize";

Note : this works fine of versions above 0.100.1.
Steps to Reproduce

Create a SASS project
Install materialize-css version 0.100.1 or up and bootstrap version 4
Import materialize-css and bootstrap sass files.
Compile your project

Environment

Version used

"bootstrap": "^4.0.0-beta"
"materialize-css": "0.100.2"

Generally it is not meant to mix both frameworks.
@tomscholz what do you think? Should we prefix as many variables as possible? This might be good. Also prefixing / namespacing methods in the JS files.

we're not going to fix compatibility issues when using bootstrap and materialize together.

if you only need  materialize css colors you can use this:
https://github.com/esmaei1/materializecolors


sunovels

Source name
sunovels
Source link
https://sunovels.com/
Language
Arabic
Other details
just add this one it will help alot !!
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a title with source name.
[X] I have checked that the extension does not already exist by searching the GitHub repository and verified it does not appear in the code base.
[X] I will fill out all of the requested information in this form.

Duplicate of #130


UI review 0.12 release

@aniketkatkar97 will take care of all these issues

[x] Change the text styling to regular text since there's no link or clickable.

[x] Update the margin here to 1rem

[x] Spacing not consistent for "Roles" and "Inherited Roles"

[x] Need to move the bar closer to %age

[x] Profiler Page, Profiler & Data quality tab

[x] Service detail page is White keep it consistent to other pages also the table styling is not updated here on either tabs

[ ] Left padding for first column in all the tables should be 16px

[ ] Keep the tag alignment consistent

Keep it like it is in image below

[ ] Remove "Gap" below "Add Rule" ; Update - Bottom padding for title to 1rem, Gap within the following fields to 8px

[ ] Table Styling different on custom properties tab on entity page.

stroke of batch should be same color as card's border color

[ ] No data illustration not updated

[ ] Profiler tab settings button styling incorrect for non admin

[ ] On Profiler tab, non-admin user can not edit owner & tier when on summary page but can edit it when on data quality or profiler sub tab

[ ] centre align the no data illustration

[ ] Inconsistent button height


[1:1] remove pfe-readtime

pfe-readtime will be removed from PFE prior to 2.0.  Will be moved to RHDS as rh-readtime.

Closed by #2229


fix(di): ensure provider creation when imports given an unregistered token from decorator

:tada: This PR is included in version 7.74.1 :tada:
The release is available on:

v7.74.1
GitHub release

Your semantic-release bot :package::rocket:


Automatic installation of core bundle

Hi,
I cannot understand if this plugin provides automatic installation every new build of bunldes. If so, I cannot make this feature work. The bundle is recognized by the plugin but everytime I save a Java file in a bundle (and then the build is run), the automatic deploy is not starting (nothing is printed in the log files, with debug mode switched on).
I am using version 1.0.1 of this plugin.
IntelliJ IDEA 2016.3.4
Build #IC-163.12024.16, built on January 31, 2017
JRE: 1.8.0_112-release-408-b6 x86
JVM: OpenJDK Server VM by JetBrains s.r.o
AEM 6.3
The project bootstrap is done with Adobe Maven Archetype.
Thanks!

This is not quite how it works. When a Java file of a bundle is changed and saved then it will be compiled and hot swapped into the server. In order to deploy the bundle needs to be deploy in the AEM tree.

Sorry, my bad. I misunderstood the way it is supposed to work. I was missing the most important part: the connection must be in debug mode to make the hot swap work. Thanks!


gateone: include SSL_CERT_FILE environment

Additional environment and path requirements to allow SSL authentication with Gateone.

Have you tried it on master ? OpenSSL should now find the right CA without having to pass the environment variable.

@zimbatm, It seems to be fixed. Thanks.


Delayed booking on events with multiple dates

As reported by Alice W, the current flow is that:

We create a dummy event with when the tickets become availble
We attach this to the Prismic event
Publish
Five minutes before Alice W removes the dummy event and attaches the new multi date event

☝ hmmm.

@jamesgorrie @gestchild do you know whether this would be resolved by using the EB widget? Or problematic in a different way?

I thought it might have been something to do with us, but it's not.

NOTE: For events with multiple dates and times, you'll only see the option to set the "Tickets sales end" date for a time that is a specific number of "hours, minutes, or days" before the event either "starts" or "ends.

https://www.eventbrite.co.uk/support/articles/en_US/How_To/how-to-change-the-start-end-date-and-time-for-ticket-sales?lg=en_GB
Let's have a think of how we can help...

Closed in #3147


When SurrealDB encounters a fatal error, it's impossible to know where it failed at (in my own code)

As you see from the logs below, we see a stack trace, but nowhere does my own code appear in that. I can't see which of my functions caused this error, and thus, locating the source of this error is more difficult than it should be.
sourceress-bot-1  | file:///app/node_modules/surrealdb.js/src/index.ts:625
sourceress-bot-1  |                     return reject(new Error(res.error.message));
sourceress-bot-1  |                  ^
sourceress-bot-1  | Error: There was a problem with the database: Specify some SQL code to execute
sourceress-bot-1  |     at Surreal._Surreal_result (file:///app/node_modules/surrealdb.js/src/index.ts:625:18)
sourceress-bot-1  |     at Surreal.<anonymous> (file:///app/node_modules/surrealdb.js/src/index.ts:424:27)
sourceress-bot-1  |     at Surreal.f (file:///app/node_modules/surrealdb.js/src/classes/emitter.ts:26:9)
sourceress-bot-1  |     at file:///app/node_modules/surrealdb.js/src/classes/emitter.ts:33:10
sourceress-bot-1  |     at Array.forEach (<anonymous>)
sourceress-bot-1  |     at Surreal.emit (file:///app/node_modules/surrealdb.js/src/classes/emitter.ts:32:20)
sourceress-bot-1  |     at Socket.<anonymous> (file:///app/node_modules/surrealdb.js/src/index.ts:222:17)
sourceress-bot-1  |     at file:///app/node_modules/surrealdb.js/src/classes/emitter.ts:33:10
sourceress-bot-1  |     at Array.forEach (<anonymous>)
sourceress-bot-1  |     at Socket.emit (file:///app/node_modules/surrealdb.js/src/classes/emitter.ts:32:20)

It is near impossible to change that. This would need a lot mor of tracking etc. that would be a lot of overhead...
Maybe we can add a "debug" flag so you can enable that... (Basicly we would have to save (new Error('')).stack  for each call of a method.

actualy this might be allready better with https://github.com/surrealdb/surrealdb.js/commit/ce949aeddd2b451b3b7b473705e62fbbc58e095b but that is currently not released...

Try running your node app with a higher stack trace limit. The default is 10 lines.
node --stack-trace-limit=50

@LuciferianInk once PR #95 is merged and release this should be a lot easier:
file:///Users/michavries/Development/SurrealDB/surrealdb.js/npm/esm/strategies/websocket.js:169
            throw new Error(res.error.message);
                  ^

Error: There was a problem with the database: Specify some SQL code to execute
    at WebSocketStrategy.query (file:///Users/michavries/Development/SurrealDB/surrealdb.js/npm/esm/strategies/websocket.js:169:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async default (file:///Users/michavries/Development/SurrealDB/surrealdb.js/test/e2e/shared.js:115:2)
    at async file:///Users/michavries/Development/SurrealDB/surrealdb.js/test/e2e/node.js:6:1

In this scenario, the following line is where the .query() function is being executed with an empty query:
at async default (file:///Users/michavries/Development/SurrealDB/surrealdb.js/test/e2e/shared.js:115:2)

Does that solve your problem?

Hi, we have shipped improvements for this a while ago, but haven't gotten a confirmation back from you. Will close this as fixed!


Support for closing system alerts on mobile

When you click on a link that takes you to a new tab in iOS. There is a message that lets you click "allow" or "deny". The alert is generated by iOS, so I can't use Selenium to accept the alert.

You can use switch to NATIVE context, and than work on the native elements. and then return to WEBVIEW context.
You can use DeviceUtils.switchToContext(context) function
or use available steps like:
I switch to native context
I switch to webview context

Help needed


rkt: implement generic no volume policy

This patch implements a generic no volume policy instead of a custom, per
stage1, behavior based on a docker2aci specific app annotation.
This also avoids stage1 having to read the app image manifest but just uses the
pod manifest as its unique source for app setup.
Since the behavior on unspecified volumes is not defined in the appc/spec but
left to the ACE, the idea is to use a pod.RuntimeApp annotation to define the
no volume policy per runtime app (instead of extending the appc PodManifest
with rkt specific fields).
The annotation is called coreos.com/rkt/annotations/novolumepolicy and
can have two possible values:

empty: create an empty, per app, volume.
empty-copy: create an empty, per app, volume and copy image contents.

empty is the default if the annotation doesn't exists.
rkt, in the stage0, will check if the annotation is already provided by the pod
manifest and, in such case, only validate it.
If not provided and the image has been converted using docker2aci then an
annotation with value empty-copy will be defined.
This has the additional effect to let the user define (and override for
docker2aci converted images) the no volume policy for every app (with this
patch only by providing a --pod-manifest, in future, if needed, a per app
command line flag could be added)
Note that only pod.RuntimeApp annotations are checked for
coreos.com/rkt/annotations/novolumepolicy and not an app manifest
annotation (this, to correctly work, requires #3100).
Notes
Before this patch an user provided pod manifest was only validated without
changing it, now, instead, it's also enhanced.
Questions

Currently the behavior of a missing volume for a mount point isn't defined in the spec and left to the ACE. Should the no volume policy behavior be explicitly defined in the appc/spec? I'm not sure about this and where to define it (per app in the pod runtimeApp? per mountpoint in the image manifest?)
Currently rkt fly stage1 is different from other flavors as if a volume isn't specified it just doesn't mount nothing:
Should a new policy none be implemented to define the fly behavior?
Should rkt fly be adapted to also implement the empty and empty-copy behaviors (if it this makes sense).
Should this no volume policy be explicitly defined in the stage1 implementors guide?

This fixes one part of #3098.

Do we actually have usecases for all those cases? If not, I think this could be a good moment for some cleaning and uniforming the behaviors.
The safest option would be to validate the pod manifest and just abort if volumes are not matching. However it seems that Docker is behaving in a different way (described here), which we are trying to mimic. As such, I think it makes sense to clarify in appc that mountpoints not covered by any volumes have the empty-copy behavior.

Also, semaphore is failing because KVM doesn't have the concept of stage1 return value: https://github.com/coreos/rkt/issues/2777

There may be actually more to it. According to appc pod spec:
A Pod Manifest must be fully resolved (reified) before execution.
Specifically, a Pod Manifest must have all mountPoints satisfied by volumes,
and must reference all applications deterministically (by image ID).

@lucab That's an interesting point.
My view, given also the subsequent sentence At runtime, the reified Pod Manifest is exposed to applications through the Metadata Service. makes me think that this reifying job (defining empty kind volumes for unspecified ones and the related runtimeApp.Mounts) should be done in the stage0 instead of the stage1s like currently done.
This change raises a question:

How to handle --pod-manifest: To keep backward compatibility they should also be reified or acurrently working pod manifest will expose a changed behavior (probably no empty volume mounted over mountPoint with no assigned volume). So this will change the "already reified" expectation on a provided podmanifest making it instead a sort of pod manifest template.

Then we should define, as you said above, if just one no volume policy is enough or different policies should be available (error, none, empty, empty-copy).
Related to this point the question is: how to, if needed, make the user choose the no volume policy (per app). Do like in this patch providing a runtimeApp annotation (for user specified podManifests) and a cli option or other solutions?

This reifying job (defining kind=empty volumes for unspecified ones and the related runtimeApp.Mounts) should be done in the stage0 instead of the stage1

I think this should be the way to go to reify incomplete/template manifests.

How to handle --pod-manifest

If k8s is not relying on this implicit behavior, we should stick to the spec and properly validate that all mountpoints are covered by a volume. Otherwise, this should be made explicit somewhere and whatever is producing the runtime manifest for us should be adapted first. /cc @euank for insights

how to, if needed, make the user choose the no volume policy

This looks like a corner case enough, that I would avoid exposing another knob just for it. Just to have the possibility of getting a docker-like behavior we discuss about a kind=guest volume, which works like an empty volume but is pre-populated with all data from guest target path.

@sgotti bumping this to the next release.

ping @jonboulle
It looks like last time we encountered it (https://github.com/coreos/rkt/pull/2315#r59297328) we special-cased it, and it's now coming back. Proposal here is to have another special casing, and I feel this will come back at a later point. Given that this kind of guest-filled volumes are there in the wild and we have to provide compatibility for it, can we capture this semantics in appc?


Updates

Update README

@lhbonifacio is it ready to merge?

@lhbonifacio is it ready to merge?
Yes, it is. Can you approve and merge, please?

Just nudging on this. BTW, it is a common practice that the creator of PR is the one who merges it.


PHOENIX-6501 Use batching when joining data table rows with uncovered…

… index rows

Can we do this for local indexes as well? There is also a significant cost to seeking (even when done locally)

Can we do this for local indexes as well? There is also a significant cost to seeking (even when done locally)
In fact the only difference might be how we get the a table reference.

@lhofhansl, we can definitely do this for local indexes. There are some differences. The first is as you pointed out table vs region. For local indexes, both data and index are accessed via the local region while the global index is accessed remotely so we need to get the table for the global index and then get connections to access the table. The concern of which connection pool to use is not applicable to the local index.  The second difference is that there is one table region to retrieve data rows for the local index. However, for the global index, there can be many. So we discover the table region boundaries and access them in parallel for the global indexes using a thread pool, which is not necessary for the local index. The last difference is handling the row key offset for local indexes, which is not necessary for the global indexes. So, I thought instead of lumping local and global index batching together, we should handle them separately. I suggest having a separate Jira and PR for the local index.


Update tardis test to use Angstroms

https://github.com/bluesky/hklpy/blob/1ce145a4de8c3643a0df224c19eea309a84a8fe1/hkl/tests/test_tardis.py#L60
and
https://github.com/bluesky/hklpy/blob/1ce145a4de8c3643a0df224c19eea309a84a8fe1/hkl/tests/test_tardis.py#L90
Observed it with @cmazzoli while testing the new version of hklpy 0.3.15 at NSLS-II CSX.

Since tardis uses soft X-rays, a wavelength of 1.33 Angstrom (energy of 9.31 keV) is not credible.  Wavelength was specified in nm here.  An energy of 931 eV is more realistic (0.13317314715359827 Angstrom).  Check the lattice parameters for nm or Angstrom.
This change is in response to #17 & #43 which was merged in PR #44.


Not counting use statements

Closes #569

@patrickkusebauch

@patrickkusebauch thanks for contributing so much to deptrac.
Here my 2 cents for this PR. I think not counting use statements are a good idea. But I'm not really happy about the implementation. Putting the configuration through every layer down to FileReferenceBuilder should have been an indicator. The result of the AstRunner can be cached and depending when deptrac was executed and when the option was set we might get an different reporting. My proposal would move up to https://github.com/qossmic/deptrac/blob/main/src/DependencyEmitter/BasicDependencyEmitter.php or https://github.com/qossmic/deptrac/blob/main/src/RulesetEngine.php.


Pin changes

Changes some pin assignments as per Controls Leaderboard revisions.

In PR #158 @NathanielDelgado implemented the same ADC Fixes, but also changed the ADC Input channels from 2 and 3 to channels 9 and 14. Those changes may need to be rolled into this PR as well, so that we can just merge this one and close that one.

These changes were already merged into SendCarCAN, and will soon be merged into master. As such, this branch is no longer needed.


Click options API

Can you provide options like onClick, onDoubleClick, onLongClick on player view?

Couldn't you just wrap it in some other view and register clicks on that view? There's option to delegate click through views and register them on parent layout... Just google it.

I don't intend to add that right now.


On Windows, the vimfiles folder is owned by administrator upon install

Steps to reproduce

Install VIM
Go to the vimfiles folder. Right click -> Properties -> Security -> Advanced. Note that the owner is Administrator

Note that this creates an error if one attempts to initialize a git repo inside vimfiles.
Expected behaviour
The vimfiles folder should be owned by the user. It is in the user folder after all.
Version of Vim
9.0
Environment
Windows 11
Logs and stack traces
No response

What method did you use to install Vim?  If you downloaded an installer: from where?

I've used three installers so far actually (to get python to work): the gvim90.exe download (https://ftp.nluug.nl/pub/vim/pc/gvim90.exe) and the 9.0.0000 32 bit and 64 bit installers from vim-win32-installer. Not sure if the last installer or the first installer caused the problem. I'm guessing something is going wrong with the "create plugin directory" option in the installer since it runs as admin.

Update, after investigating what the installer code is doing, (at least what is in the vim-win32-installer project) I can't see anything obvious where its setting any security details.
It looks to me like the new directories are simply inheriting the security settings from their respective parent directory.   And, I think this is the correct behavior - unless I'm missing something different with your setup!?   I was unable to replicate the problem you described.  I could do a git init  on a fresh install of Windows 11 - even though the owner is the Administrators group.  My user account still has Full Control permissions.    Could you check if your user account doesn't have Full Control, and if the security settings aren't just being inherited from the user's home directory?

It was the admins group for me as well.
Actually, IIRC git init was fine, but subsequently git status failed. There was a file in autoload at this point (from vim plug).
I'm not sure if inheriting home is the reason here. I would more suspect it has to do with the folder being created by the installer while elevated.

ahh, yes, I see it now;
> cd vimfiles
> git status
fatal: detected dubious ownership in repository at 'C:/Users/chris/vimfiles'
'C:/Users/chris/vimfiles' is owned by:
        'S-1-5-32-544'
but the current user is:
        'S-1-5-21-XXXX-YYYYY-ZZZZ-9999'
To add an exception for this directory, call:

        git config --global --add safe.directory C:/Users/chris/vimfiles

It's an interesting issue.  And, sure, it's easy enough to change security ownership back to your user account.  Also, it's easy enough to add the git config as it suggests.    But, I think this seems unnecessary to have to do this, and I think it would be more convenient if the $home/vimfiles directory was created with the users own account ownership.
When I was looking at the source code, for example see src/dosinst.c   I got the impression it was intended to work with old dos and/or windows - meaning it's compatible with the FAT file systems, as well as NTFS.   A solution might involve either going down the path of including adding some headers like this;  <accctrl.h> <aclapi.h> to the c source code.   Or, it might involve adding some features to the NSIS script file.
Your bio says you are a cs major, so perhaps this is this something you'd like to have a go at doing a PR for?

When a user selects to create $HOME/vimfiles, dosinst is called with the -create-directories home option from nsis/gvim.nsi.
A possible solution is to use ShellExecAsUser::ShellExecAsUser in gvim.nsi to create it.

Actually is this an installer bug that should go in that repo instead?

we can test it in the vim/vim-win32-installer repo first but if this is a change in gvim.nsi it should eventually end up here as well.

@k-takata   that sounds better :)  nice, simple, elegant - and now I see that the ShellExecAsUser::ShellExecAsUser plugin is already used with the nsi scripts.
@xsrvmy  maybe :)  @k-takata  and @chrisbra  are across both repos.
@k-takata  and @chrisbra   How might we do a PR to the vim-win32-installer git repo, for code that is in vim git submodue?

I would create a patch (which can reference the vim directory) and place it into the patch directory. that should then be applied during building of the installer.

i've never done a patch file before, I don't mind having a go and trying to do it.

If I recall correctly, you need to go with it like this:
You basically check out the vim-win32-installer repository + the vim submodule. You make your changes in the vim submodule for gvim.nsi.
You can test it, by removing your changes in the vim worktree, going into the vim directory and running git apply /path//to/patch/directory/file.patch. That should apply cleanly, and redo your changes.

Something like this?
--- a/nsis/gvim.nsi
+++ b/nsis/gvim.nsi
@@ -228,6 +228,28 @@ FunctionEnd
 !insertmacro GetParent ""
 !insertmacro GetParent "un."

+# Get home directory
+!macro GetHomeDir un
+Function ${un}GetHomeDir
+  Push $0
+  Push $1
+  ReadEnvStr $0 "HOME"
+  ${If} $0 == ""
+    ReadEnvStr $0 "HOMEDRIVE"
+    ReadEnvStr $1 "HOMEPATH"
+    StrCpy $0 "$0$1"
+    ${If} $0 == ""
+      ReadEnvStr $0 "USERPROFILE"
+    ${EndIf}
+  ${EndIf}
+  Pop $1
+  Exch $0  # put $0 on top of stack, restore $0 to original value
+FunctionEnd
+!macroend
+
+!insertmacro GetHomeDir ""
+!insertmacro GetHomeDir "un."
+
 # Check if Vim is already installed.
 # return: Installed directory. If not found, it will be empty.
 Function CheckOldVim
@@ -520,7 +542,7 @@ SectionGroup $(str_group_plugin) id_grou
 	Section "$(str_section_plugin_home)" id_section_pluginhome
 		SectionIn 1 3

-		StrCpy $1 "$1 -create-directories home"
+		#StrCpy $1 "$1 -create-directories home"
 	SectionEnd

 	Section "$(str_section_plugin_vim)" id_section_pluginvim
@@ -594,6 +616,13 @@ Section -call_install_exe
 	DetailPrint "$(str_msg_registering)"
 	nsExec::Exec "$0\install.exe $1"
 	Pop $3
+
+	${If} ${SectionIsSelected} ${id_section_pluginhome}
+	  ReadEnvStr $3 "COMSPEC"
+	  Call GetHomeDir
+	  Pop $4
+	  ShellExecAsUser::ShellExecAsUser "" "$3" '/c mkdir "$4\vimfiles"' SW_HIDE
+	${EndIf}
 SectionEnd

 ##########################################################
@@ -1042,15 +1071,8 @@ SectionEnd
 SectionGroup "un.$(str_ungroup_plugin)" id_ungroup_plugin
 	Section /o "un.$(str_unsection_plugin_home)" id_unsection_plugin_home
 		# get the home dir
-		ReadEnvStr $0 "HOME"
-		${If} $0 == ""
-		  ReadEnvStr $0 "HOMEDRIVE"
-		  ReadEnvStr $1 "HOMEPATH"
-		  StrCpy $0 "$0$1"
-		  ${If} $0 == ""
-		    ReadEnvStr $0 "USERPROFILE"
-		  ${EndIf}
-		${EndIf}
+		Call un.GetHomeDir
+		Pop $0

 		${If} $0 != ""
 		  !insertmacro RemoveVimfiles $0

Untested. Even not compiled yet.

that looks really cool @k-takata
looks like you guys have this pretty much sorted out :)
I know I asked to have a go at doing it, but I've ran out of time today - and because I gotta bail now for the weekend, so I don't mind if you guys want to wrap it up.   I will have a look on sunday and see if anything remaining I can do to help.

I don't have enough time either. If you (or someone else) want to take it over, it's fine for me.

@xsrvmy  please check the artifact installer from https://github.com/vim/vim-win32-installer/pull/299
https://ci.appveyor.com/project/chrisbra/vim-win32-installer/builds/46068517/job/gbff4bh6qim4yatx/artifacts

No permission issue this time.
Are the subfolders (autoload, etc.) supposed to be created by the installer?

Ah, that should be updated.

Perhaps, the ShellExecAsUser line should be like this:
	  ShellExecAsUser::ShellExecAsUser "" "$3" '/c "cd /d "$4" & mkdir vimfiles & cd vimfiles & mkdir colors compiler doc ftdetect ftplugin indent keymap plugin syntax"' SW_HIDE

Anyone got any ideas why MSVC 2022 takes 2x longer than MSVC 2015 would take?  Might be some compiler settings we could use to get it faster again?
@k-takata patch for the sub-directories is now in the PR.
Anyway, my earlier test results for this issue - with first patch from @k-takata ;  %HOME%\vimfiles directory was correctly owned by the user account on 8.1, 10 and 11 after fresh install.   However, on my Win 7.1 test, the owner still remains as Admin group.   Not sure if this is just because I might have reduced all the acl settings in my Win 7.1 VM earlier when I set it up to test something else!?.   I guess Win7.1 is getting old now, so this is probably a rare edge case - and as this has simple work-arounds anyway - so I don't see it as a problem that needs the installer to deal with right away.

umm - this is new...

Trojan:Script/Wacatac.B!ml

Alert level: Severe
Status: Active
Category: Trojan
Details: This program is dangerous and executes commands from an attacker.

Affected items:
  file: C:\Program Files\Vim\vim90\install.exe

That was a Windows Security alert - quarantined the vim install.exe heper file.!?   I'll report a separate issue.
Apart from that, testing of @k-takata  new patch works fine - creates the set of vimfiles\sub-dirs with correct ownership.

okay, if this works, can someone please create a PR against this repo to change it and then to vim-win32-installer to remove the patch again?

if no-one else does, I can do a pr when i get back home in a few days.


add alunocea.txt

add alunocea.txt

@joseferrao You might miss my previous comment. Please react.

Hello

the school official website URL: https://www.espacoaberto.com.br
the school street address, including city and country: Rua Bento
Albuquerque, 1671 - Cocó, Fortaleza / Ceará
an URL of a page on the official website where the school offers an
IT-related long-term (>1 year) course: My children study at this school,
where in high school, there is an optional science course focused on
programming. It is used to support students in which career to pursue,
being able to choose humanities, exact sciences and social sciences.
a proof, which shows that the school recognizes the domain you are
submitting as an official email domain for the students: The email domain
is separate from the website domain, the email service and all resources
are provided by Google.

[image: image.png]
José F
On Mon, Jun 10, 2024 at 10:02 AM Philip Torchinsky @.***>
wrote:

@joseferrao https://github.com/joseferrao Please provide us with:

the school official website URL
the school street address, including city and country
an URL of a page on the official website where the school offers an
IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are
submitting as an official email domain for the students.

—
Reply to this email directly, view it on GitHub
https://github.com/JetBrains/swot/pull/21121#issuecomment-2158291399,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEJ77KNCESP4WCVZ5ORXMLZGWPWPAVCNFSM6AAAAABJAMDMPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYGI4TCMZZHE
.
You are receiving this because you were mentioned.Message ID:
@.***>

@joseferrao  Thank you for the clarifications. Pull request merged.


Add ATT&CK Technique auto-complete

Adding auto-complete for the technique ID field of Action objects would help immensely, as it not only makes it easier to cite Techniques, but also ensures they're consistent and not subject to typos.
Being able to add multiple Techniques to a single Action object would also be helpful - though I understand if your view is that this isn't in keeping with the design intent of this project.
It would be useful for my particular use case, though. See below for an example:

In visualising a long and complex intrusion, breaking each Technique into its own Action object would result in a sprawling diagram that would be hard to navigate.
My preference is to use the Name field to describe distinct stages of the intrusion, elaborating in the Description field while also inserting the relevant Technique IDs. In doing so I can condense multi-staged attack paths into more compact diagrams, broken up by milestones in the intrusion.

Thank you for suggesting this. It is in our backlog and will be included in a release in the next 2 months.

Hi @mehaase ,
Thanks for your work on such a cool tool! I am also really interested in this feature. Is it still planned for release?
Thanks so much.