TensorFlow 2.0 - ValueError: tf.function-decorated

Hello, I have a code (for MNIST dataset) in which I am doing the following steps:

Train model
Prune model (using "tensorflow_model_optimization" for say p%)
Create a mask of pruned model, so that the sparsity is maintained in subsequent steps
Reset weights of non-pruned model to random initialized weights when model was initialized

I do the following steps iteratively 'n' times.
The code can be found in:
https://github.com/arjun-majumdar/tensorflow_codes/blob/master/Recreating_Error.ipynb
For retraining a pruned model, I use 'GradientTape' along with mask. Now, the first time the model is trained using train_one_step() and test_step() functions which are @tf.function annotated functions, things work fine. But when I try to use them again (in cell 76 of Jupyter notebook), it gives me the error:

ValueError                                Traceback (most recent call last)
 in
13     for x, y in train_dataset:
14         # train_step(x, y)
---> 15         train_one_step(model_gt_stripped, mask_model_stripped, optimizer, x, y)
16
17     for x_t, y_t in test_dataset:
~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in call(self, *args, **kwds)
455
456     tracing_count = self._get_tracing_count()
--> 457     result = self._call(*args, **kwds)
458     if tracing_count == self._get_tracing_count():
459       self._call_counter.called_without_tracing()
~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _call(self, *args, **kwds)
485       # In this case we have created variables on the first call, so we run the
486       # defunned version which is guaranteed to never create variables.
--> 487       return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
488     elif self._stateful_fn is not None:
489       # Release the lock early so that multiple threads can perform the call
~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in call(self, *args, **kwargs)
1820   def call(self, *args, **kwargs):
1821     """Calls a graph function specialized to the inputs."""
-> 1822     graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
1823     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
1824
~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs)
2148         graph_function = self._function_cache.primary.get(cache_key, None)
2149         if graph_function is None:
-> 2150           graph_function = self._create_graph_function(args, kwargs)
2151           self._function_cache.primary[cache_key] = graph_function
2152         return graph_function, args, kwargs
~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
2039             arg_names=arg_names,
2040             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2041             capture_by_value=self._capture_by_value),
2042         self._function_attributes,
2043         # Tell the ConcreteFunction to clean up its graph once it goes out of
~/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
913                                           converted_func)
914
--> 915       func_outputs = python_func(*func_args, **func_kwargs)
916
917       # invariant: func_outputs contains only Tensors, CompositeTensors,
~/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds)
356         # wrapped allows AutoGraph to swap in a converted function. We give
357         # the function a weak reference to itself to avoid a reference cycle.
--> 358         return weak_wrapped_fn().wrapped(*args, **kwds)
359     weak_wrapped_fn = weakref.ref(wrapped_fn)
360
~/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in wrapper(*args, **kwargs)
903           except Exception as e:  # pylint:disable=broad-except
904             if hasattr(e, "ag_error_metadata"):
--> 905               raise e.ag_error_metadata.to_exception(e)
906             else:
907               raise
ValueError: in converted code:
<ipython-input-44-d0ca499a4063>:29 train_one_step  *
    optimizer.apply_gradients(zip(grad_mask_mul, model.trainable_variables))
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:435 apply_gradients
    self._create_slots(var_list)
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/adam.py:146 _create_slots
    self.add_slot(var, 'm')
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:587 add_slot
    initial_value=initial_value)
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py:260 __call__
    return cls._variable_v2_call(*args, **kwargs)
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py:254 _variable_v2_call
    shape=shape)
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py:65 getter
    return captured_getter(captured_previous, **kwargs)
/home/majumdar/.local/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py:413 invalid_creator_scope
    "tf.function-decorated function tried to create "

ValueError: tf.function-decorated function tried to create variables on non-first call.

The only way of avoiding this "ValueError" is by rerunning the train_one_step() and test_step() @tf.function annotated functions!
Why is this happening?
Thanks!

@arjun-majumdar ,
Can you please take a look at the link of Stack-overflow and let me know if it helps.Thanks!

Going through the link and the related tutorial.

Hello, I have the following issue which doesn't make sense. The code that I have is as follows:
@tf.function
def train_step(model, mask_model, optimizer, x, y):
    with tf.GradientTape() as tape:
        y_pred = model(x)
        loss = loss_fn(y, y_pred)

    grads = tape.gradient(loss, model.trainable_variables)

    optimizer.apply_gradients(zip(grads, model.trainable_variables))

    train_loss(loss)
    train_accuracy(y, y_pred)

    return None

@tf.function
def test_step(model, optimizer, data, labels):
    """
    Function to test model performance
    on testing dataset
    """

    predictions = model(data)
    t_loss = loss_fn(labels, predictions)

    test_loss(t_loss)
    test_accuracy(labels, predictions)

    return None

# User input parameters for Early Stopping in manual implementation-
minimum_delta = 0.001
patience = 3

best_val_loss = 1
loc_patience = 0

# Initialize a neural network model-
model_gt = pruned_nn(pruning_params_unpruned)

# Strip model of pruning layers-
model_gt_stripped = sparsity.strip_pruning(model_gt)

for i in range(1, 6):

    print("\n\n\nOuter loop: {0}\n\n".format(i))

    # Initialize parameters for Early Stopping manual implementation-
    best_val_loss = 1
    loc_patience = 0

    for epoch in range(num_epochs):

        if loc_patience >= patience:
            print("\n'EarlyStopping' called!\n")
            break

        # Reset the metrics at the start of the next epoch
        train_loss.reset_states()
        train_accuracy.reset_states()
        test_loss.reset_states()
        test_accuracy.reset_states()

        '''
        # Initialize 'grad_mask_mul' list-
        grad_mask_mul = []

        # Initialize all values to one-
        for wts in mask_model_stripped.trainable_weights:
            grad_mask_mul.append(wts.assign(tf.ones_like(input = wts,dtype = tf.float32)))

        # Convert from Python list to tf.Tensor-
        grad_mask_mul = tf.convert_to_tensor(grad_mask_mul, dtype=tf.float32)

        print("type(grad_mask_mul): {0}".format(type(grad_mask_mul)))
        '''

        for x, y in train_dataset:
            train_step(model_gt_stripped, mask_model_stripped, optimizer, x, y)
            # train_one_step(model_gt_stripped, mask_model, optimizer, x, y, grad_mask_mul)

        for x_t, y_t in test_dataset:
            # test_step(x_t, y_t)
            test_step(model_gt_stripped, optimizer, x_t, y_t)

        template = 'Epoch {0}, Loss: {1:.4f}, Accuracy: {2:.4f}, Test Loss: {3:.4f}, Test Accuracy: {4:4f}'

        print(template.format(epoch + 1,
                              train_loss.result(), train_accuracy.result()*100,
                              test_loss.result(), test_accuracy.result()*100))

        # Count number of non-zero parameters in each layer and in total-
        # print("layer-wise manner model, number of nonzero parameters in each layer are: \n")

        model_sum_params = 0

        for layer in model_gt_stripped.trainable_weights:
            # print(tf.math.count_nonzero(layer, axis = None).numpy())
            model_sum_params += tf.math.count_nonzero(layer, axis = None).numpy()

        print("Total number of trainable parameters = {0}\n".format(model_sum_params))

        # Code for manual Early Stopping:
        if np.abs(test_loss.result() < best_val_loss) >= minimum_delta:
            # update 'best_val_loss' variable to lowest loss encountered so far-
            best_val_loss = test_loss.result()

            # reset 'loc_patience' variable-
            loc_patience = 0

        else:  # there is no improvement in monitored metric 'val_loss'
            loc_patience += 1  # number of epochs without any improvement

If, I re-execute the for i in range(1, 6): block of code again, why do I get the "ValueError: tf.function-decorated function tried to create variables on non-first call." ?
Thanks

@arjun-majumdar
please provide us with simple indented stand alone code with all dependencies, for us to replicate it in our environment and analyse the issue faced by you.
please find the gist of the code shared by you and error faced.

https://github.com/arjun-majumdar/tensorflow_codes/blob/master/Recreating_Error.ipynb

@arjun-majumdar have you seen this related issue #27120 ?
This comment might help you. Try creating a wrapper function for your train_one_step() function and then call separately when you train your different models.

@nikitamaia Let me have a look and get back to you. Thanks!

Hi @arjun-majumdar were you able to get your code working by creating a wrapper function?

Hello @nikitamaia the TF annotated funtion after wrapping function works, however, the performance benefits gained by<EMAIL_ADDRESS>annotation is lost.
Also, why should the graph generated be retraced if the neural network architecture isn't changing and/or the data type of the tensors are also not changing.

Sorry for the late response here. Wanted to provide a quick update that this does seem to be a bug. I know the workaround of wrapping the function is not ideal, but I can update this thread when there's progress on this.

@nikitamaia Thanks for the reply. I will be happy to receive an update if there is a fix/solution found for this bug.

The Better Performance With tf.function guide has now been updated to provide more detail about this error and about using using tf.variables with multiple Keras models or optimizers. Closing this issue now since there is a workaround provided in the docs.

@nikitamaia I'm afraid that isn't the workround, it's just the rule, not fitting the user's demand


一次性添加种子数超过网页高度导致提交按钮无法显示

Yes, this should be fixed. I'll do it later.
Thanks for reporting the issue.


ValueError:nan in latents

My gpu is 2080ti22G, comfyui says the cuda does not support bf16, I have to set model_dtype as fp16, Then I keep getting "nan in lantents" error. the whole log is as follows:
2024-10-12 00:05:30,985 - root - INFO - got prompt
2024-10-12 00:05:35,676 - root - ERROR - !!! Exception during processing !!! nan in latents
2024-10-12 00:05:35,676 - root - ERROR - Traceback (most recent call last):
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\execution.py", line 317, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\execution.py", line 192, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-PyramidFlowWrapper\nodes.py", line 205, in sample
latents = model["model"].generate(
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-PyramidFlowWrapper\pyramid_dit\pyramid_dit_for_video_gen_pipeline.py", line 578, in generate
intermed_latents = self.generate_one_unit(
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "I:\stable_diffusion\ComfyUI-aki-v1.2\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-PyramidFlowWrapper\pyramid_dit\pyramid_dit_for_video_gen_pipeline.py", line 285, in generate_one_unit
raise ValueError("nan in latents")
ValueError: nan in latents
2024-10-12 00:05:35,677 - root - INFO - Prompt executed in 4.63 seconds
## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.

{"last_node_id":27,"last_link_id":38,"nodes":[{"id":8,"type":"PyramidFlowVAEDecode","pos":{"0":1161,"1":873},"size":{"0":315,"1":102},"flags":{},"order":3,"mode":0,"inputs":[{"name":"model","type":"PYRAMIDFLOWMODEL","link":8,"label":"model"},{"name":"samples","type":"LATENT","link":9,"label":"samples"}],"outputs":[{"name":"images","type":"IMAGE","links":[38],"slot_index":0,"label":"images"}],"properties":{"Node name for S&R":"PyramidFlowVAEDecode"},"widgets_values":[256,2]},{"id":9,"type":"PyramidFlowSampler","pos":{"0":1059,"1":497},"size":{"0":411.5168151855469,"1":314},"flags":{},"order":2,"mode":0,"inputs":[{"name":"model","type":"PYRAMIDFLOWMODEL","link":7,"label":"model"},{"name":"prompt_embeds","type":"PYRAMIDFLOWPROMPT","link":31,"label":"prompt_embeds"},{"name":"input_latent","type":"LATENT","link":null,"shape":7,"label":"input_latent"}],"outputs":[{"name":"model","type":"PYRAMIDFLOWMODEL","links":[8],"label":"model"},{"name":"samples","type":"LATENT","links":[9],"slot_index":1,"label":"samples"}],"properties":{"Node name for S&R":"PyramidFlowSampler"},"widgets_values":[1280,768,"20, 20, 20","10, 10, 10",16,7,5,44664248661394,"fixed",""]},{"id":14,"type":"VHS_VideoCombine","pos":{"0":1534,"1":490},"size":[1698.6201171875,310],"flags":{},"order":4,"mode":0,"inputs":[{"name":"images","type":"IMAGE","link":38,"label":"images"},{"name":"audio","type":"AUDIO","link":null,"shape":7,"label":"audio"},{"name":"meta_batch","type":"VHS_BatchManager","link":null,"shape":7,"label":"meta_batch"},{"name":"vae","type":"VAE","link":null,"shape":7,"label":"vae"}],"outputs":[{"name":"Filenames","type":"VHS_FILENAMES","links":null,"label":"Filenames"}],"properties":{"Node name for S&R":"VHS_VideoCombine"},"widgets_values":{"frame_rate":16,"loop_count":0,"filename_prefix":"PyramidFlow","format":"video/h264-mp4","pix_fmt":"yuv420p","crf":19,"save_metadata":true,"pingpong":false,"save_output":true,"videopreview":{"hidden":false,"paused":false,"params":{"filename":"PyramidFlow_00022.mp4","subfolder":"","type":"output","format":"video/h264-mp4","frame_rate":16},"muted":false}}},{"id":22,"type":"PyramidFlowTextEncode","pos":{"0":567,"1":757},"size":{"0":434.50982666015625,"1":227.74803161621094},"flags":{},"order":1,"mode":0,"inputs":[{"name":"model","type":"PYRAMIDFLOWMODEL","link":30,"label":"model"}],"outputs":[{"name":"prompt_embeds","type":"PYRAMIDFLOWPROMPT","links":[31],"label":"prompt_embeds"}],"properties":{"Node name for S&R":"PyramidFlowTextEncode"},"widgets_values":["A campfire burning with flames and embers, gradually increasing in size and intensity before dying down towards the end, hyper quality, Ultra HD, 8K","bad video",false]},{"id":5,"type":"DownloadAndLoadPyramidFlowModel","pos":{"0":575,"1":513},"size":{"0":385.7839050292969,"1":202},"flags":{},"order":0,"mode":0,"inputs":[],"outputs":[{"name":"pyramidflow_model","type":"PYRAMIDFLOWMODEL","links":[7,30],"slot_index":0,"label":"pyramidflow_model"}],"properties":{"Node name for S&R":"DownloadAndLoadPyramidFlowModel"},"widgets_values":["rain1011/pyramid-flow-sd3","diffusion_transformer_768p","fp16","fp16","fp16",false,false]}],"links":[[7,5,0,9,0,"PYRAMIDFLOWMODEL"],[8,9,0,8,0,"PYRAMIDFLOWMODEL"],[9,9,1,8,1,"LATENT"],[30,5,0,22,0,"PYRAMIDFLOWMODEL"],[31,22,0,9,1,"PYRAMIDFLOWPROMPT"],[38,8,0,14,0,"IMAGE"]],"groups":[],"config":{},"extra":{"ds":{"scale":1.708874251124587,"offset":[-739.5461978505967,-454.1771000145869]}},"version":0.4}

I'm aware of this and it was one of the big issues I had getting this running, it's a known issue in the main code tracked here:
https://github.com/jy0205/Pyramid-Flow/issues/25

I'm running a 4090 with a 7950X3D on Win11. I tried both install methods (manual from the repository & thru the ComfyUI Manager). Also made sure to do the Pip install of the requirements.txt.
When I run a Temp = 16 & FPS =16, the GPU consumes 20-22GB VRAM, it runs about 50% of the time. The other 50% of the time, all crashes are "NAN in Latents".
In addition to the repeating error, my other concern is that I can't run a Temp of 31 or FPS of 24, that will crash 100% of the time. Another person I was talking to, that has a 4060 Ti and says that a Temp of 31 and FPS of 24 on his GPU will only consume 10GB VRAM, and runs fine each time. So I'm a little confused why my  4090, at lower settings, is using ~21GB, and his, with a lesser GPU & higher settings is only consuming 10GB and works perfectly???
Any insights would be greatly appreciated. Thank you.
For background, my 4090 runs Flux on Comfy flawlessly.

I'm running a 4090 with a 7950X3D on Win11. I tried both install methods (manual from the repository & thru the ComfyUI Manager). Also made sure to do the Pip install of the requirements.txt.
When I run "I2V" with a Temp = 16 & FPS =16, using all bf16, the GPU consumes 20-22GB VRAM, it runs about 50% of the time. The other 50% of the time, all crashes are "NAN in Latents".
In addition to the repeating error, my other concern is that I can't run a Temp of 31 or FPS of 24 on "I2V", that will crash 100% of the time.
One other data point, a 10 second "I2V" is about 3-4 times (10-16MB) the file size as compared to an "T2V" ouput (3-5MB) ... I'm not sure if that makes sense?
Any insights would be greatly appreciated. Thank you.
For background: my 4090 runs Flux and PF "T2V" on Comfy flawlessly.

What's your pytorch version? It shouldn't even take 10GB when sampling really, so something is very much off indeed.

If I input on Anaconda - pip show torch
Name: torch
Version: 2.4.1
Hope that helps

Is it cu124 version? Because on that it should at least work, as that what I'm using. Other than that I don't know, I'm also using latest Nvidia drivers.

Would you know the Anaconda command to get the Cuda version ?
I'm an old C++ Programmer

Would you know the Anaconda command to get the Cuda version ?
I'm an old C++ Programmer

Should be pip show torch

I had done that before to get Torch 2.4.1
The same command didn't work for Cuda
I checked within the Comfy install, in python_embeded/lib/site-packages and it shows a folder: ~orch-2.4.0+cu121.dist-info
Maybe that's it? Cu121

I had done that before to get Torch 2.4.1
The same command didn't work for Cuda
I checked within the Comfy install, in python_embeded/lib/site-packages and it shows a folder: ~orch-2.4.0+cu121.dist-info
Maybe that's it? Cu121

Yes sorry that's what I mean, it's the version torch was compiled with, it should be listed at end of torch version usually.
121 should be fine...I'm bit confused over your setup though, you use conda env but also portable install?

I'm not sure about the Cuda environment ... When I started using Comfy, used the portable install (only, nothing else) and then Comfy installed all the necessary files. Been using since SD1.5, 2.1, XL, SD3, Flux & now PF
Maybe I overlooked something?
Sorry, don't mean to be a pain. Very grateful that you put this package together 🙏

Does the new model support fp16 now?


WinDbg Linux debugging documentation completely missing?!

Hi there,
I learned not too long ago that WinDbgX understands core files written by .NET on Linux in native/managed debugging scenarios.
For a post-mortem we were trying to tease out information from WinDbg regarding the call stack that led to a crash.
And indeed by setting the symbol path to find the shared object itself, kb would now proclaim to know that one of the stack frames was happening at a method start+0x105.
Alas, even though the file (a native shared object) is not stripped, WinDbg is unable to correlate this to a source line or so.
I have seen this with Windows debugging before whenever symbols were totally unavailable. In this case WinDbg would fall back to referring to the locations by some exported symbol plus and offset.
During my search I also tried looking up more details and came up empty-handed.
Could someone here point me to documentation related to debugging -- or specifically post-mortem debugging scenarios -- for Linux using WinDbgX?!
I am also interested in learning whether the compiler/linker commands should/could be tweaked to yield better results (thinking -ggdb -g3 and friends).
Any pointers to existing documentation would be appreciated. Otherwise, could you please add it? I know it's still called WinDbg, but seemingly it can do a whole lot more by now.
Two doc items I'd definitely like to see:

how to set the symbol path?
how to troubleshoot symbol path related issues?

@exoosh - Thanks for this input on adding more information on the use of Symbols in a Linux environment. We use input such as this, to guide the docs work that we do.
No matter the OS, with the local symbol cache that WinDbg uses, and the many options, there is some complexity in using symbols. These two topics provide general information on that:

Setting Symbol and Executable Image Paths in WinDbg
Symbol path for Windows debuggers

In version  1.0.1908.30002, Support for Open Enclave debugging was added, for more information, see Open Enclave debugging. Also as part of supporting Open Enclave, WinDbg can open ELF core dumps and binaries as well as DWARF symbols (DWARF 5 is not currently supported) from both Enclaves and Linux applications. When opening a core dump from a non-Windows application, basic windows and commands should all work properly, but most extensions and Windows-specific commands will not work. ELF and DWARF files will be downloaded from symbol servers following the key conventions defined here. Enclaves are the only supported scenario.
Starting with version 1.0.2007.01003 of the Windows Debugger, Portable PDB Symbols are supported. This is described in this topic, and it also provides links to related ECMA specifications. Portable PDB Symbols.
Symbols are a great help, but in all settings, you can almost always read registers and disassemble instructions, but that is tedious and not a great option, and assumes some knowledge of assembler.
If you have general feedback on how WinDbg can better support Linux, you can provide that using the WinDbg issues repo - http://aka.ms/dexex
Thanks again for your interest in WinDbg and the input on the docs.

Thanks @DOMARS for the comment and the pointers. This helps a lot already.

When opening a core dump from a non-Windows application, basic windows and commands should all work properly, but most extensions and Windows-specific commands will not work.

Sure, makes sense. I wouldn't expect anything else.

ELF and DWARF files will be downloaded from symbol servers following the key conventions defined here. Enclaves are the only supported scenario.

Perhaps you could clarify a few things.
Enclaves being the only supported scenario precludes other ELF files or means they might (or even should) work, but it's not supported if I'd turn to the Microsoft support?
Is it limited to outright symbol servers, or are local symbol stores fine?

Starting with version 1.0.2007.01003 of the Windows Debugger, Portable PDB Symbols are supported. This is described in this topic, and it also provides links to related ECMA specifications. Portable PDB Symbols.

I was aware of portable PDBs, but the issue we encounter is in native code and that was the blind spot to me.
Reading assembly is no issue, however.
Thanks again for all the valuable pointers.


ATB- 109 Add required infrastructure for DynamoDB

What?
The account management application currently uses Shared-Redis as a session store via Elasticache and we have decided to move to using DynamoDB for session store instead, so have created the required infrastructure for the move to DynamoDB.
Why?
To kickstart the migration from Shared- Redis to DynamoDB for the purpose of decreasing costs overall.
Testing

Hi @MinaMahdi, @domantas-gds & @KG700 - is the table name going to be passed to the Container via env var setting?

Hi @MinaMahdi, @domantas-gds & @KG700 - is the table name going to be passed to the Container via env var setting?

Hi @TaherKapasi , yes as far as I've understood that's something that is going to be passed.

Replaced by new PR for ATB-108


JsFile::getFirstLineToken implementation (for indentation rules)

I've implemented getFirstLineToken to improve indentation rules.
@mikesherov, please take a look.

Coverage decreased (-0.03%) to 98.79% when pulling fbd3745a5f61b0cdca5c24b62129459a40781359 on mdevils.get-first-line-token into d567f914d2461635976d463abe22a13aa0becf5b on master.

Coverage increased (+0.01%) to 98.83% when pulling c6bea2f5214d940c2925fa5db559ac5930092024 on mdevils.get-first-line-token into d567f914d2461635976d463abe22a13aa0becf5b on master.

Looks good except can we name it getFirstTokenOnLine? Makes the purpose clearer in English.

Renamed it :wink:

:+1:

Coverage increased (+0.01%) to 98.83% when pulling 7c4fec055cc3a81109d868f7da992600bc17939f on mdevils.get-first-line-token into d567f914d2461635976d463abe22a13aa0becf5b on master.

Coverage increased (+0.01%) to 98.83% when pulling 2e7ff96ddfab70b96f89e150a74b01df79b3fbd7 on mdevils.get-first-line-token into d567f914d2461635976d463abe22a13aa0becf5b on master.

Ouch, nice! :+1:


License System

We need a proper system for licensing anything (modules, code fragments, images, text, etc), my suggestion is anywhere we license something we also add a file (like a .license.json or something) that contains info that could be used by a license listing generator (for example we could list a summary by owner and have an openable section for all such resources used)
For example a sample license might look like this:
# File included in same location as image
{
    "author": "Randall Munroe",
    "type": "image",
    "original": "http://imgs.xkcd.com/comics/np_complete.png",
    "fragment": "false",
    "modified": "false",
    "usage": "Part of Chapter Complexity and Tractibility",
    "ourCopy": "./xkcd-np-complete-comic.png",
    "licenseURL": "http://xkcd.com/license.html"
}

from which we could then generate some page like:

Licensed Materials
Randal Monroe

"NP Complete"

Usage: "Part of Chapter Complexity and Tractability"
Modified: "False"
License: "http://xkcd.com/license.html"

etc...

I'd be happy to write the generator, but would everyone be on board with using a system like this? (e.g. you'd just throw the file where you have the copyrighted material and the parser finds all .license.json files in the entire field guide and generates a license page).
Also what fields should be included in every license.json file and what additional ones should be considered?

There is also a prototype of managing licenses but Jordan and I have planned to extend the current system. It's something I plan to do when I return, as it does not need to be implemented over Christmas.

Hi Jack,
Thankyou for your response.
It would be helpful if you told us what we should be doing with licence files in the mean time, which will make it easy as possible for you to move them into the new system once it is written.
Cheers!
Heidi

The temporary location of details is the permissions.conf file.

Closed as Jack already has plans to do something else :-)


🐛 [Bug] Running docker-compose up in authentik folder causes trouble

🐛 Bug

Running docker-compose up in authentik folder causes trouble.
We might want to remove the compose files there and make docker run the compose files in the folder above.

The PR https://github.com/dreammall-earth/dreammall.earth/pull/2661 only fixed it partly. I can not estimate if there further things needed or not.


LookupField value handling corrected

LookupField no longer throws away an atomic value.
Solution for https://github.com/silverstripe/silverstripe-framework/issues/8022

Also, if you'd like this in 4.x could you please rebase and change this PR's target branch?

@robbieaverill I rebased the branch on 4.1 and added a unit test. Please review.

Interesting catch. It looks almost to me like like MultiSelectField::stringDecode is expecting to throw an exception in a situation like this:
        // If json deserialisation fails, then fallover to legacy format
        $result = json_decode($value, true);
        if ($result !== false) {
            return $result;
        }

        throw new \InvalidArgumentException("Invalid string encoded value for multi select field");

Given that json_decode returns null when deserialisation fails, this logic seems like it would never execute (the exception thrown).
I think this fix is fine, and looks like a safe change. I also feel like we need to fix MultiSelectField::stringDecode to correctly detect JSON deserialisation failures (I presume there's no unit test for this), but that's another story.
You're also right in that it seems odd for DropdownField > readonly > LookupField to extend from MultiSelectField. I guess it's probably the same case as a field that allows multiple selections going through readonly transformation extending from LookupField, so perhaps is used to handle both cases. In the case of DropdownField > readonly > LookupField, throwing an exception on failure to deserialise a JSON input value is wrong because we don't expect it.
The logic in this PR is good for that case, assuming that the parent logic is currently wrong and not detecting serialisation failures. I suspect if we "fix" MultiSelectField::stringDecode then this logic would no longer work, as an exception would be thrown early. This could be a bit risky in terms of future maintenance.
I also notice a disparity between stringEncode (public) and stringDecode (protected).

I think you've highlighted a valid problem (and fixed it) but I'm a little concerned with fixing it in this way with respect to the longevity of the change.
What do you think @tractorcow?

The code in MultiSelectField is incorrectly checking for false as error condition, when null is actually the error condition.
I think you are right in that sometimes the object's field doesn't contain proper json content, but actually is the selected value to show.
How about this
protected function stringDecode($value)
    {
        // Handle empty case
        if (empty($value)) {
            return [];
        }

        // Attempt to decode json format
        $result = json_decode($value, true);

        // If non-json value, assume single value
        if (json_last_error()) {
            return [ $value ];
        }

        return $result;
    }

Notice we retain the array value return result. ;)
Fix this in MultiSelectField, not just in a subclass.

I also think dropdownfield readonly should just be setReadonly(true), not shifting to a separate form field class. See https://github.com/silverstripe/silverstripe-framework/issues/8002

@robbieaverill @tractorcow I implemented Damian's solution and it seems to be working fine. Please review.

Hmmm... now the behaviour is to treat json deserialisation failures as atomic values rather than throw an exception. This would mean broken json strings are returned literally instead of throwing errors. I suspect this could end up causing data to be displayed to CMS users rather than a system error message

I think the REAL fix here is to stop using LookupField as a readonly dropdown? :P

Yeah totally, but we can't in 4.x, that's be an API change (DropdownField::performReadonlyTransformation no longer returns LookupField). This is a good intermediary until we can simplify the readonly/disabled form field methodology as discussed the other day!

@robbieaverill @tractorcow I agree that we probably don't want to change existing functionality of the MultiSelectField. I added a new SingleLookupField to represent the single value version of LookupField. Please review.

I'm not sure this is better. You have a singleselect extends somefield extends multiselect. Shouldn't it extend SingleSelectField instead?

This is the API of performReadonlyTransformation()
/**
     * Returns a read-only version of this field.
     *
     * @return FormField
     */
    public function performReadonlyTransformation()

It doesn't require a specific subclass of FormField to be returned.

@tractorcow The reason why I decided to extend the LookupField  is that I can re-use functionality already present in that field. It seems to be more efficient than extending the SingleSelectField .

Also, it's more backward compatible because if some code checks for instanceof LookupField, my change will not break such code.

The reason why I decided to extend the LookupField is that I can re-use functionality already present in that field.

I understand that, but it's a mis-use of object inheritance. Inheritance isn't about code saving. There is much more appropriate logic to single-selection of fields in SingleSelectField that should be applied anyway.

@tractorcow Ok, I'll update the code, thanks for the feedback.

@robbieaverill @tractorcow New changes added, please review.

Sorry, I dismissed the wrong review! I'll leave this to @tractorcow =)

Implementation is ok with me, but @robbieaverill do you think we should merge to 4 branch not 4.1? It's a new class, seems like a new API rather than a pure bugfix.

I'm easy. Semver allows for bugfixes that are backwards compatible (this mostly is) as a patch

In that case I'm making the call to make it a minor branch change.

Bughancement


[WIP] Exceptions in @GetSize and finalization callbacks should not fail the bundle directly

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

r: @kennknowles
cc: @lukecwik

r: @kennknowles
cc: @lukecwik

Run Java PreCommit

Run Java PreCommit

Kindly pinging @kennknowles. Please let me know if you think it's not a good idea.

Run Java PreCommit

Run Java PreCommit

Won't a failure in finalization callback mean that messages are not ACKed so they will be replayed?

Won't a failure in finalization callback mean that messages are not ACKed so they will be replayed?

To my knowledge, it depends on the source implementation details. Taking KafkaIO as an example, the finalizeCheckpoint does commit the offset but this commit has no effect on current reading(since the reader will track current position), Within such implementation, the replayed happens when you stop your pipeline then restart it, and you want to read from last committed offset.
I want to discuss more about error handling on bundle finalization. The SDK can choose to fail the bundle directly or SDK can choose to return a response with error to runner, then let the runner to decide whether to fail the bundle. Currently in python sdk, the SDK will return the response with error instead of failing the bundle. But the Java SDK fails the bundle directly. I'm leaning to let the runner make the decision when exception happens(ignore/retry/fail the execution). What do you think?


621 Test EQ Flush

Motivation and Context

[ ] Context Index has been kept up to date

Adds ATs for the new EQ flushing action rule feature. One test stops at just checking the cloud task queue messages sent by case processor, so that it can be run locally, the other test (tagged to run in the cloud only) calls our EQ flushing stub to check the correct API call was made by the invoker.
What has changed

Add EQ flush scenarios
Ignore @cloud_only tag in Makefile run target to avoid running cloud scenarios locally.

How to test?

Build and run all the branches locally, run the ATs branch. Try creating/running an EQ Flush action rule from support tool.
Links

https://trello.com/c/sN2EG6Rh/621-eq-flushing-by-action-rule-with-rate-limiting-13

@davidmort It shouldn't need it, that tag is only used to exclude scenarios when running locally so nothing running in the cloud should need to change.

@davidmort It shouldn't need it, that tag is only used to exclude scenarios when running locally, so nothing running in the cloud should need to change.

fair enough, then all looks good to me 👍


Error while opening multiple

When opening the reader multiple times, the browser restarts and displays a script loading error.

Do you have example code that is causing this issue?

Closing due to lack of activity. If you have more information that will help debug this issue please re-open with it.


Preserve order when using JSON output

Fixes #241

Interesting! I'll need to update the snapshots to make this pass the tests nicely, thanks for this!

@omerbenamram tests should be fixed now!

apologies @omerbenamram all the remaining tests should now be fixed, and it does successfully build for me now

@ac-rn thanks again for this contribution!


Unable to build run locally

I was trying to build the image and run the manual tests locally, but swarm is unable to start the service with the below error
standard_init_linux.go:175: exec user process caused "no such file or directory"

I'm using the golang 1.7 image to build the source (same instruction as in the build.md)
One thing I noticed is that the size of image that I built locally is slightly larger than the latest image from Docker hub
12:06 $ docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
vfarcic/docker-flow-swarm-listener   1.0                 ad3ce6604fac        12 minutes ago      18.1 MB
vfarcic/docker-flow-swarm-listener   beta                ad3ce6604fac        12 minutes ago      18.1 MB
alpine                               3.4                 0766572b4bac        9 days ago          4.8 MB
golang                               1.7                 6639f812dbc7        3 weeks ago         674 MB
vfarcic/docker-flow-swarm-listener   latest              186692b36f6b        3 weeks ago         18 MB

My docker for mac info
Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 9
Server Version: 1.13.0-rc4
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: 64vzjjiwwfqvwq11vvspwmpne
 Is Manager: true
 ClusterID: xv0437qqvi5wx2513wso348fi
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: <IP_ADDRESS>
 Manager Addresses:
  <IP_ADDRESS>:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.8.15-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952 GiB
Name: moby
ID: MI3A:FWVT:5XZZ:55E5:EAFH:DU5W:LERU:H373:REL6:FIFA:7T5O:BLXX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 342
 Goroutines: 719
 System Time: 2017-01-06T18:09:40.431504523Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false

just realized I did not have the CGO_ENABLED=0 set.
Works fine now

Hi,
I'm glad it worked out.
The project was done relatively quickly and it fulfilled its main purpose without any problems. As a result, I did not prioritize writing proper instructions. One thing is that it works on my laptop, the other to make it useful for everyone. I'd appreciate if you could make a pull request with improvements to docs (or any other addition).


add test for more mutation operations

This adds some tests for the remaining mutation operations. I'm not sure whether it makes sense to test all public methods, as some of them simply forward calls to each other.

Looks good to me.
Two observations:-

I would like to see the empty collection also used as a test case for the addition tests, though - given that the removal operations treat attempts to remove non-existent items as a no-operation, could this be added in without breaking the existing tests? The only exception would be it should "be unchanged when removing elements that do not exist", which could be cutover to using newElem instead of using b.head and b.removalAll(elem) as a way of ensuring that the sut doesn't already contain the element being removed.

I'll put in an inline comment about the tests for updating.

@Toromtomtom I guess it's on me now to deliver on the tests for pure-functional '-' tests. :smile:
Leave it with me...

Regarding my comment above about including the empty bag test case in the MutableBagBehaviours behaviour, I've realised that will still break the existing tests for element removal, simply because the assertions do expect an element to be removed, even if removal is a no-operation for the empty bag.
I guess a better way would be split the behaviours into those that would hold for all bags, and those that require the bag to be non-empty. So additive tests go in the first (as does it should "be unchanged when removing elements that do not exist"), whereas the subtractive ones go in the second.

I updated the code and separated the tests into two methods for empty and non-empty mutable bags.

@nicolasstucki If I can, I'll merge this next week unless you want to continue the review...

@sageserpent-open, you can merge it whenever you believe it is good


Make it possible to run badge server on App Engine.

Is it safe for us to remove the GKE version badge server and the memorystore stuff after this change and the deployment?

I think that it would be safe to remove the GKE stuff.


Add more tests for cache invalidation.

I needed these tests to convince myself that
this API works the way I believed it to.
This is just the unit tests commit part of https://github.com/systemjs/builder/pull/374, since we're not ready to add documentation to the feature yet.

Thanks, I will review this when I can.


getMessageNo() is empty?

When looping through a series of messages in a folder, I can get the UUID, subject, etc. but when I access the 'getMessageNo()' it's empty.
Is there any reason for this or a bug perhaps?

Hi @AndyDunn ,
please provide some more information or perhaps some code samples.
I can't reproduce this behavior.
Best regards

Hi @Webklex, this is the code I'm using:
`<?
$client = new Client([
'host'          => env('IMAP_HOST'),
'port'          => 993,
'encryption'    => 'ssl',
'validate_cert' => true,
'username'      => env('IMAP_USERNAME'),
'password'      => env('IMAP_PASSWORD'),
]);
        $client->connect();
        $aMailboxes = $client->getFolders();

        foreach($aMailboxes as $oMailbox) {
            if ($oMailbox->name == env('IMAP_FOLDER')) {
                foreach($oMailbox->getMessages() as $message) {
                    Log::info("Number: ".$message->getMessageNo());
                }
            }
        }`

In the log files the number value is empty. Anything else you'd need me to check?


Feature/ldap

I've taken LDAP code started here and tested it out and make some additional changes. This addresses #230

@ammmze , thanks for the contribution, i'll test it and provide some feedback if needed before merging

@ammmze , please resubmit the pull req to https://github.com/pantsel/konga/tree/ldap-integration

@pantsel FYI...this is the docker-compose.yml file I used for testing this:
version: "3"

services:
  ldap:
    image: osixia/openldap:1.2.1
    environment:
      LDAP_ORGANIZATION: Example Inc.
      LDAP_DOMAIN: example.org
      LDAP_ADMIN_PASSWORD: admin

  ldap-ui:
    image: osixia/phpldapadmin:0.7.1
    environment:
      PHPLDAPADMIN_LDAP_HOSTS: ldap
    ports:
      - "${LDAP_UI_PORT:-6443}:443"

  #######################################
  # Postgres: The database used by Kong
  #######################################
  kong-database:
    image: postgres:9.5
    restart: always
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-kong}
      POSTGRES_DB: ${POSTGRES_DB:-kong}
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

  #######################################
  # Kong database migration
  #######################################
  kong-migration:
    image: kong:0.13.1-alpine
    command: "kong migrations up"
    restart: on-failure
    environment:
      KONG_PG_HOST: kong-database
    depends_on:
      - kong-database

  #######################################
  # Kong: The API Gateway
  #######################################
  kong:
    image: kong:0.13.1-alpine
    restart: always
    environment:
      KONG_PG_HOST: kong-database
      KONG_PROXY_LISTEN: <IP_ADDRESS>:8000
      KONG_PROXY_LISTEN_SSL: <IP_ADDRESS>:8443
      KONG_ADMIN_LISTEN: <IP_ADDRESS>:8001
    depends_on:
      - kong-database
    healthcheck:
      test: ["CMD", "curl", "-f", "http://kong:8001"]
      interval: 5s
      timeout: 2s
      retries: 15
    ports:
      - "${KONG_PORT:-8001}:8001"
      - "${API_PORT:-8000}:8000"

  #######################################
  # Konga: Kong GUI
  #######################################
  konga:
    image: pantsel/konga:latest
    build: ./
    restart: always
    environment:
      KONGA_AUTH_PROVIDER: ldap
      KONGA_LDAP_HOST: ldap://ldap:389
      KONGA_LDAP_BIND_DN: cn=admin,dc=example,dc=org
      KONGA_LDAP_BIND_PASSWORD: admin
      KONGA_LDAP_USER_SEARCH_BASE: ou=users,dc=example,dc=org
      KONGA_LDAP_GROUP_SEARCH_BASE: ou=groups,dc=example,dc=org
    depends_on:
      - kong
    ports:
      - "${KONGA_PORT:-1337}:1337"

I just setup the ldap instance via the ui. I setup a users organizational unit and groups organizational unit. Then added a posix user account for John Smith under users. Then added a konga posix group under groups and added jsmith to the group.
In the end it looked something like this:

Hmm...so actually...I just realized the group checking isn't working. Basically it is currently letting any user who has an account login.

Err...never mind...group checking is working. I had assumed that by setting admin=false, they would not be able to login. But without the appropriate group, they just get a read-only view. Perhaps we should add something additional for a user check, so that you can restrict read-only access as well?

@ammmze appreciate if you have a casual minute for a 4 year ago thing yah made!
Any idea what is happening here? I am no MS AD LDAP expert by any means...
debug: ######################################## { user: null }

Was trying to use your LDAP integration logic:
KONGA_AUTH_PROVIDER:
ldap
KONGA_LDAP_HOST:
ldap://ad-ldap.company.com:389
KONGA_LDAP_BIND_DN:
Source: username (konga-ldap-secret)
KONGA_LDAP_BIND_PASSWORD:
Source: password (konga-ldap-secret)
KONGA_LDAP_USER_SEARCH_BASE:
CN=Users,dc=ms,dc=ds,dc=company_value_here,dc=com
KONGA_LDAP_GROUP_SEARCH_BASE:
CN=Users,dc=ms,dc=ds,dc=company_value_here,dc=com
KONGA_ADMIN_GROUP_REG:
^(stargate_self_service_admin|stargate_api_gateway_nonprod)$

Leaving all other vars as default.
For 2 microsoft ad groups to be allowed in as admins(and I know I have those groups as well). Based on the output I am guessing my BIND DN and Pass are good for connecting to the LDAP HOST, my user search base must be working likely too to have gotten by register. My group search base must likely be the missing component here and why my user ended up as just a "basic" non admin user in limbo...

Hey @jeremyjpj0916 ... TBH I'm not much of an LDAP guy either and where I work we never ended up using this. But looking through it, I'm not sure if the group search base is correct. It looks like you have it the same as the user search base. From what it looks like, it looks like it is expecting to do a search in collection of things that returns groups, not users. And then it does adds to the query a filter to filter the groups by the user using the KONGA_LDAP_GROUP_SEARCH_FILTER property mentioned in the readme. In the end, the majority of the ldap stuff is done via the passport-ldapauth plugin, which then in turn passes off stuff to node-ldapauth-fork.

Thanks for your input, yeah I am on the same page I think my group search base is indeed incorrect, will try to get clarification from our internal LDAP management team on what it should look like.

@ammmze  Hmm maybe microsoft ad ldap isn't compatible w this actually.
Takes a query like this(via CLI):
ldapsearch -h ad-ldap.company.com -x -b "CN=USERNAME_HERE,CN=Users,dc=ms,dc=ds,dc=COMANY_HERE,dc=com" -W -D<EMAIL_ADDRESS>Enter LDAP Password:

To get results of a given user details with a memberOf list(microsoft ad groups):
memberOf: CN=GROUP_NAME_HERE,OU=Cloud-Contacts,OU=Messaging,OU=COMPANY_FIELD,OU=COMPANY,DC
 =ms,DC=ds,DC=COMPANY,DC=com
memberOf: CN=GROUP_NAME_HERE2,CN=Users,DC=ms,DC=ds,DC=COMPANY,DC=com
memberOf: CN=GROUP_NAME_HERE3,OU=Cloud-Contacts,OU=Messaging,OU=COMPANY_FIELD,OU=COMPANY,D

If there was then a way per user to get the memberOf list to parse for admin access I imagine the
KONGA_ADMIN_GROUP_REG field would be like:
'^(CN=GROUP_NAME_HERE,OU=Cloud-Contacts,OU=Messaging,OU=COMPANY_FIELD,OU=COMPANY,DC
 =ms,DC=ds,DC=COMPANY,DC=com|CN=GROUP_NAME_HERE2,CN=Users,DC=ms,DC=ds,DC=COMPANY,DC=com)$'

For example giving the first 2 groups in the list access to the thing. I don't think there is a way in ms ad ldap to lookup all groups by user, I think the lookup is by user which will include memberOf groups list. Will confirm this with a team tomorrow. Time to sleep lol.

Well I can search on a specific group by CN Seems:
ldapsearch -h ad-ldap.company.com -x -b "CN=stargate_self_service_admin,CN=Users,dc=ms,dc=ds,dc=COMPANY,dc=com" -W -D<EMAIL_ADDRESS>Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <CN=stargate_self_service_admin,CN=Users,dc=ms,dc=ds,dc=company,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# stargate_self_service_admin, Users, ms.ds.company.com
dn: CN=stargate_self_service_admin,CN=Users,DC=ms,DC=ds,DC=company,DC=com
objectClass: top
objectClass: group
cn: stargate_self_service_admin
description: Admin group to override LDAP validation checks on user with regar
 ds to Stargate self-service.
member: CN=user1,CN=Users,DC=ms,DC=ds,DC=company,DC=com
member: CN=user2,CN=Users,DC=ms,DC=ds,DC=company,DC=com

distinguishedName: CN=stargate_self_service_admin,CN=Users,DC=ms,DC=ds,DC=company,
 DC=com
instanceType: 4
whenCreated:<PHONE_NUMBER>4717.0Z
whenChanged:<PHONE_NUMBER>4205.0Z
uSNCreated: 157601882
uSNChanged: 984032823
name: stargate_self_service_admin
sAMAccountName: stargate_self_service_admin
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=ds,DC=company,DC=com
dSCorePropagationData:<PHONE_NUMBER>1410.0Z

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

Wonder if there is compatibility I could work with here by isolating my group search to a singular group I want members to be a part of without having to change your code impl...

If anyone else uses MS AD LDAP and needs guidance I found a pattern that works for me so throwing it here incase it helps any weary traveler, caveat being it limits you to 1 group to do admin access control but it works!
KONGA_LDAP_USER_SEARCH_BASE: CN=Users,dc=ms,dc=ds,dc=COMPANY,dc=com
KONGA_LDAP_GROUP_SEARCH_BASE: CN=YOUR_GROUP_HERE,CN=Users,dc=ms,dc=ds,dc=COMPANY,dc=com
KONGA_LDAP_GROUP_SEARCH_FILTER: (member=CN={{uid}},CN=Users,DC=ms,DC=ds,DC=COMPANY,DC=com)
KONGA_LDAP_GROUP_ATTRS: cn,member
KONGA_ADMIN_GROUP_REG: ^.*$

What I think it does is: 1 user search makes sure valid user.

Group search our groups uid being my MS username are under the same Users CN but also adding a extra CN=your_group_here etc paired with the (member=CN={{uid}},CN=Users,DC=ms,DC=ds,DC=uhc,DC=com) will only RETURN data if that user has that YOUR_GROUP_HERE membership.

Then the REGEX is just a match all, meaning if we got proper data of any kind back then user met the criteria, its not pretty but did work for me and distinguish regular users from admin users via a MS AD LDAP group. Phew this took awhile to crack but found a hacky way to make it work for me.


修复 picker 通过 setData 改变数据，列数变化时产生的问题

列数增加: 增加的列对应的 scroll 未实例化无法滚动
列数减少: pickerSelectedVal 保留了变化前选中的值

Please make sure these boxes are checked before submitting your PR, thank you!

[x] Make sure you follow DiDi's contributing guide.
[x] Make sure you are merging your commits to dev branch.
[x] Add some descriptions and refer relative issues for you PR.

Codecov Report

Merging #116 into dev will decrease coverage by 0.23%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##              dev     #116      +/-   ##
==========================================
- Coverage   88.77%   88.54%   -0.24%
==========================================
  Files          83       83
  Lines        1559     1562       +3
  Branches      220      220
==========================================
- Hits         1384     1383       -1
- Misses         94       97       +3
- Partials       81       82       +1

Impacted Files
Coverage Δ

src/components/picker/picker.vue
85.32% <100%> (+0.41%)
:arrow_up:

src/components/time-picker/time-picker.vue
73.5% <0%> (-2.65%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbdc727...42fa785. Read the comment docs.

再次感谢。之后如果还要修改的话，可以直接在你的 dev 分支就可以了，这边会自动同步哈，不用关了再开一个 👍

好的，添麻烦了，第一次 pr，多多包涵🤐

非常感谢PR，还有点小问题，当时 dirty的时候也依旧会遇到同样的问题， https://github.com/fyerl/cube-ui/blob/9bb4e06eb31cbe221b29681c56b8636e3d90e11b/src/components/picker/picker.vue#L169
。
建议考虑统一处理，这样相同的删除销毁逻辑就没必要在两个地方写了。

我看源码里当 dirty 时, 是将 this.wheels 置空, 所以意思是源码本身这块逻辑也是存在问题的, 应该逐一 destory ? 确认一下😊
https://github.com/fyerl/cube-ui/blob/9bb4e06eb31cbe221b29681c56b8636e3d90e11b/src/components/picker/picker.vue#L171

Travis tests have failed
Hey Fyerl,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
Node.js: 6

npm test

><EMAIL_ADDRESS>test /home/travis/build/didi/cube-ui
> npm run unit && npm run codecov

><EMAIL_ADDRESS>unit /home/travis/build/didi/cube-ui
> cross-env BABEL_ENV=test karma start test/unit/karma.conf.js --single-run

(node:4072) DeprecationWarning: loaderUtils.parseQuery() received a non-string value which can be problematic, see https://github.com/webpack/loader-utils/issues/56
parseQuery() will be replaced with getOptions() in the next major version of loader-utils.
12 03 2018 15:20:50.571:INFO [karma]: Karma v1.7.1 server started at http://<IP_ADDRESS>:9876/
12 03 2018 15:20:50.573:INFO [launcher]: Launching browser PhantomJS_mobile with unlimited concurrency
12 03 2018 15:20:50.586:INFO [launcher]: Starting browser PhantomJS
12 03 2018 15:20:51.258:INFO [PhantomJS 2.1.1 (Linux 0.0.0)]: Connected on socket rOpn4UjEWM9uO1dqAAAA with id 42972563
INFO LOG: 'You are running Vue in development mode.
Make sure to turn on production mode when deploying for production.
See more tips at https://vuejs.org/guide/deployment.html'

  ActionSheet
    ActionSheet.vue
      ✓ use
      ✓ should render correct contents
      ✓ should render correct contents - no data
      ✓ should render correct contents - picker style
      ✓ should trigger events
    ActionSheet API
      ✓ $createActionSheet API

  Button.vue
    ✓ use
    ✓ should render correct contents
    ✓ props
    ✓ events-called
    ✓ events-no-called

  CascadePicker
    ✓ use
    ✓ should render correct contents
    ✓ should render correct contents when use alias
    ✓ should trigger events
    ✓ setData
    ✓ should add warn log when sigle is true

  CheckboxGroup.vue
    ✓ use
    ✓ should render correct contents (horizontal=false)
    ✓ should render correct contents (horizontal=true)
    ✓ should change v-model value

  Checkbox.vue
    ✓ use
    ✓ should render correct contents
    ✓ should toggle v-model value

  Dialog
    Dialog.vue
      ✓ use
      ✓ should render correct contents - icon
      ✓ should render correct contents - title
      ✓ should render correct contents - alert
      ✓ should render correct contents - confirm
      ✓ should render correct contents - confirmBtn/cancelBtn
      ✓ should trigger events
      ✓ should not trigger events when btn is disabled
    Dialog API
      ✓ $createDialog API

  IndexList
    IndexList.vue
      ✓ use
      ✓ should render correct contents
      ✓ should trigger events
      ✗ should fixed title
	AssertionError: expected 'B' to equal 'C' (/home/travis/build/didi/cube-ui/node_modules/chai/chai.js:210)

      ✓ run normal when group or item undefined
      ✓ should handle condition of unexpected param

  Input.vue
    ✓ use
    ✓ should render correct contents
    ✓ should not show clear button when no text
    ✓ should show clear button when has text
    ✓ value should be empty when clear button clicked
    ✓ should support more native props
    ✓ should show password eye and work correctly

  Loading.vue
    ✓ use
    ✓ should render correct contents - no size
    ✓ should render correct contents - with size

  Picker
    ✓ use
    ✓ should render correct contents
    ✓ should render correct contents when use alias
    ✓ should trigger events
    ✓ should refill correct
    ✓ should add warn log when single is true
    ✓ setData when picker is invisible
    ✓ setData when picker is visible

  Popup
    Popup.vue
      ✓ use
      ✓ should render correct contents - no type
      ✓ should render correct contents
      ✓ events
    Popup API
      ✓ $createPopup API

  Radio.vue
    ✓ use
    ✓ should render correct contents
    ✓ should render correct contents - horizontal
    ✓ should toggle v-model value

  Rate.vue
    ✓ use
    ✓ should render correct contents
    ✓ should trigger change event
    ✓ should trigger change event with click

  Scroll
    ✓ use
    ✓ should render correct contents
    ✓ should trigger pullingDown
    ✓ should trigger pullingUp
    ✓ should trigger click

  Select.vue
    ✓ use
    ✓ should render correct contents
    ✓ should trigger events

  Slide.vue
    ✓ use
    ✓ should render correct contents
    ✓ should trigger change event

  Swipe
    ✓ use
    ✓ should render correct contents
    ✓ should shrink if the swipe width not enough
    ✓ should shrink if the last moving direction is right
    ✓ should shrink and not moved if the duration is more than momentumLimitTime and moving distance is less than momentumLimitDistance
    ✓ should shrink and not moved if y-axis distance is more than x-axis distance
    ✓ should grow if the swipe width enough
    ✓ should grow if the duration is less than momentumLimitTime and moving distance is more than momentumLimitDistance
    ✓ should not swipe out of the boundaries when swiping left
    ✓ should not swipe out of the boundaries when swiping right
    ✓ should trigger events
    ✓ should shrink if another swipe item active
    ✓ should do nothing if active the same swipe item
    ✓ should call refresh if the btns prop has changed
    ✓ should use default props if do not pass any prop

  Switch
    ✓ use
    ✓ should render correct contents
    ✓ should toggle v-model value

  Textarea.vue
    ✓ use
    ✓ should render correct contents
    ✓ should not expand when blur
    ✓ should expand when focus, fold when blur
    ✓ should has remain when focus
    ✓ should change value
    ✓ should support more native props

  TimePicker
    ✓ use
    ✓ should render correct contents
    ✓ should render correct contents - no showNow
    ✓ should trigger events
    ✓ should add warn log when sigle is true

  Tip.vue
    ✓ use
    ✓ should render correct contents
    ✓ should render correct contents with offset[X]
    ✓ should be hidden after clicked content
    ✓ should be hidden after clicked close button

  Toast
    Toast.vue
      ✓ use
      ✓ should render correct contents - no type
      ✓ should render correct contents - time 0
      ✓ should render correct contents - with type
    Toast API
      ✓ $createToast API

  Upload.vue
    ✓ use
    ✓ should render correct contents
    ✓ should add files & upload
    ✓ should remove file
    ✓ should start & pause upload
    ✓ should retry upload
    ✓ should trigger events

  validator.vue
    ✓ use
    ✓ should render correct contents - invalid

  rules
    ✓ should be valid - no rule
    ✓ should be valid - unstated rule
    ✓ should be invalid - require: array
    ✓ should be invalid - type: string
    ✓ should be valid - type: string
    ✓ should be invalid - type: number
    ✓ should be valid - type: number
    ✓ should be invalid - type: array
    ✓ should be valid - type: array
    ✓ should be invalid - type: date
    ✓ should be valid - type: date
    ✓ should be invalid - type: email
    ✓ should be valid - type: email
    ✓ should be invalid - type: tel
    ✓ should be valid - type: tel
    ✓ should be invalid - type: url
    ✓ should be valid - type: url
    ✓ should be invalid - min: string
    ✓ should be valid - min: number
    ✓ should be valid - max: string
    ✓ should be invalid - min: number
    ✓ should be valid - min: string
    ✓ should be invalid - min: number
    ✓ should be invalid - notWhitespace
    ✓ should be invalid - pattern
    ✓ should be invalid - custom

  methods
    ✓ should add customized rule and message
    ✓ should rewrite raw type
    ✓ should change the language of default message

PhantomJS 2.1.1 (Linux 0.0.0): Executed 159 of 159 (1 FAILED) (24.417 secs / NaN secs)
TOTAL: 1 FAILED, 158 SUCCESS

1) should fixed title
     IndexList IndexList.vue
     AssertionError: expected 'B' to equal 'C' (/home/travis/build/didi/cube-ui/node_modules/chai/chai.js:210)

=============================== Coverage summary ===============================
Statements   : 93.77% ( 1476/1574 )
Branches     : 76.41% ( 554/725 )
Functions    : 94.38% ( 151/160 )
Lines        : 93.77% ( 1476/1574 )
================================================================================

npm ERR! Linux 4.14.12-041412-generic
npm ERR! argv "/home/travis/.nvm/versions/node/v6.13.1/bin/node" "/home/travis/.nvm/versions/node/v6.13.1/bin/npm" "run" "unit"
npm ERR! node v6.13.1
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>unit: `cross-env BABEL_ENV=test karma start test/unit/karma.conf.js --single-run`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>unit script 'cross-env BABEL_ENV=test karma start test/unit/karma.conf.js --single-run'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the cube-ui package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     cross-env BABEL_ENV=test karma start test/unit/karma.conf.js --single-run
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs cube-ui
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls cube-ui
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/travis/build/didi/cube-ui/npm-debug.log
npm ERR! Test failed.  See above for more details.

再次感谢，你的代码非常棒。我们会在下个版本1.7.0发布这次优化。Thank you very much!

第一次 pr 也谢谢二位的包含和 review


kerberos: authentication between client and broker

Fixes #3652
** Motivation **
Currently both Zookeeper and BookKeeper could be secured by using Kerberos, It would be good to support Kerberos in Pulsar Broker and Client.
This is the sub-task for issue in #3491 to support Kerberos in Pulsar Broker and Client.
Will add proxy and web resource support in following prs.
The Kerberos authentication is similar to that in Zookeeper and BookKeeper, which leverage SASL and GSSAPI, so reused some of the code there.
PR #3658 is the first version of PR before #3677 .
** Changes **
provide both client and broker side support for authentication api;
add unit test.

@eolivelli  Thanks for the comments in #3658. This is the second version.

@jiazhai I will take a look


feat: version 3.0

How to use this beta
To install the beta version of node-llama-cpp, run this command inside of you project:
npm install node-llama-cpp@beta

The interface of node-llama-cpp will change multiple times before a new stable version is released, so the documentation of the new version will be updated only a bit before the stable version release.
If you'd like to use this beta, you can visit this PR for an updated example of how to have a simple chat session with a model.
How you can help

Try the beta and provide feedback here
Like this issue to help make it resolve sooner, or open a PR for it: https://github.com/ggerganov/llama.cpp/issues/4315

Included in this beta

feat: automatic batching (#104)
Automatic chat wrapper detection based on the model file name, metadata, and BOS string representation
Fix the n_tokens <= n_batch error
Basic context shifting support, will improve in subsequent betas

Planned changes before release

[ ] Use the smart context shifting support in LlamaChatSession - in progress
[ ] Function calling support (#101) - in progress
[ ] Better contextSize and batchSize defaults that depend on the current machine hardware (depends on https://github.com/ggerganov/llama.cpp/issues/4315)
[ ] Add contexts manager to automatically create more contexts as needed
[ ] Improve grammar support
[ ] minP support (#119)
[ ] Getting embedding for text support (#123)
[ ] Try to disable llama.cpp logs by default
[ ] Add more tests - in progress
[ ] Update documentation
[ ] Add migration guide from v2 to v3

Usage example

Relevant for the 3.0.0-beta.1 version

import {fileURLToPath} from "url";
import path from "path";
import {LlamaModel, LlamaContext, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const model = new LlamaModel({
    modelPath: path.join(__dirname, "models", "dolphin-2.1-mistral-7b.Q4_K_M.gguf")
});
const context = new LlamaContext({
    model,
    contextSize: Math.min(4096, model.trainContextSize)
});
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

const q1 = "Hi there, how are you?";
console.log("User: " + q1);

const a1 = await session.prompt(q1);
console.log("AI: " + a1);

const q2 = "Summerize what you said";
console.log("User: " + q2);

const a2 = await session.prompt(q2);
console.log("AI: " + a2);

Pull-Request Checklist

[ ] Code is up-to-date with the master branch
[ ] npm run format to apply eslint formatting
[ ] npm run test passes with this change
[ ] This pull request links relevant issues as Fixes #0000
[ ] There are new or updated unit tests validating the change
[ ] Documentation has been updated to reflect this change
[ ] The new commits and pull request title follow conventions explained in pull request guidelines (PRs that do not follow this convention will not be merged)

Hey, I have switched to the beta due to the infamous n_tokens <= n_batch error, and I saw that it is now possible to automatically detect the correct context size. However, there is a problem with that: I have been trying this out with Mistral's OpenOrca 7b in the Q4_K_M quantized size, and the issue is that the training context is 2^15 (32,768), but the quantized version reduces this context to 2,048. With your code example, this will immediately crash the entire server when using your code, since contextSize: Math.min(4096, model.trainContextSize) will in this case resolve to contextSize: Math.min(4096, 32768) and then contextSize: 4096 which is > 2048.
I know that it's not always possible to detect the correct context length, but it would be great if this would not crash the entire app, and instead, e.g., throw an error.
Is it possible to add a mechanism to not crash the module if the provided context size is different from the training context size?

What is the equivalent of context.decode in this new release? I stream responses by passing in an onToken function that calls decode on the tokens that come through and now it's saying this context.decode is undefined

@nathanlesage I'm pretty sure that the reason your app crashes is that larger context size requires more VRAM, and your machine doesn't have enough VRAM for a context length of 4096 but has enough for 2048.
If you try to create a context with a larger size than is supported by the model, it won't crash your app but may cause the model to generate gibberish as it crosses the supported context length size.
Unfortunately, it's not possible to safeguard against this at the moment on node-llama-cpp's side since llama.cpp is the one that crashes the process, and node-llama-cpp is not aware of the available VRAM and memory requirements for creating a context with a specific size.
To mitigate this issue I've created this feature request on llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4315
After this feature is added on llama.cpp I'll be able to improve this situation on node-llama-cpp's side.
If this issue is something you expect to happen frequently in your application lifecycle, you can wrap your code with a worker thread until this is fixed properly.

I thought that at first, but then I tried the same code on a windows computer, also with 16 GB of RAM, and it didn't crash. Then I tried out the most recent llama.cpp "manually" (I.e., pulled and ran main) and it worked even with the larger context sizes. I'm beginning to think that this was a bug in the metal code of llama.cpp -- I'll try out beta.2 that you just released, that should fix the issue hopefully.
And thanks for the tip with the worker, I begin to feel a bit stupid for not realizing this earlier, but I've never worked so closely with native code in node before 🙈

@giladgd  hi, I think the embedding Fn, can you follow the interface?
EmbeddingsInterface here https://github.com/langchain-ai/langchainjs/blob/5df71ccbc734f41b79b486ae89281c86fbb70768/langchain-core/src/embeddings.ts#L9

Quick update regarding the beta.2: I noticed that now llama.cpp defaults to the CPU, despite being on an MBP 14'' M2 Pro, which makes token generation extremely slow. The previous beta.1 was defaulting to the GPU and token generation was much faster.
I am still trying to investigate what the cause is, but I wanted to let you know that it seems to have metal support disabled (…?)

I'm missing the LlamaContext decoder() function for the tokens, when streaming the prompt using the chat session onToken-"event":
 chatSession.prompt(message,    onToken(chunk) {       console.debug(context.decode(chunk)); // ?       //...    } }
Is there a substitution for that ?

@mstankala :
model.detokenize(chunk)

for the issue of @scenaristeur scenaristeur mentioned this pull request Mar 4, 2024
n_tokens <= n_batch
i have tried to migrate to     "node-llama-cpp": "^3.0.0-beta.13", but not i have a crash on my laptop ideapad (https://www.google.com/search?client=firefox-b-lm&q=ideapad+3+15alc6 ) (no GPU, AMD rizen 5000, 16 core CPU / 16GBram)
It worked like a charm with  "node-llama-cpp": "^2.8.8", (i had no issue of memory apart  n_tokens < n_batch with long ConversationHistory) but now it crashes even with a small conversationhistory with "radv/amdgpu: Not enough memory for command submission."
[node-llama-cpp] llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /home/smag/dev/igora/models/vicuna-7b-v1.5-16k.Q2_K.gguf (version GGUF V2)
[node-llama-cpp] llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[node-llama-cpp] llama_model_loader: - kv   0:                       general.architecture str              = llama
[node-llama-cpp] llama_model_loader: - kv   1:                               general.name str              = lmsys_vicuna-7b-v1.5-16k
[node-llama-cpp] llama_model_loader: - kv   2:                       llama.context_length u32              = 16384
[node-llama-cpp] llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
[node-llama-cpp] llama_model_loader: - kv   4:                          llama.block_count u32              = 32
[node-llama-cpp] llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
[node-llama-cpp] llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
[node-llama-cpp] llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
[node-llama-cpp] llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
[node-llama-cpp] llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
[node-llama-cpp] llama_model_loader: - kv  10:                    llama.rope.scale_linear f32              = 4.000000
[node-llama-cpp] llama_model_loader: - kv  11:                          general.file_type u32              = 10
[node-llama-cpp] llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
[node-llama-cpp] llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
[node-llama-cpp] llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
[node-llama-cpp] llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
[node-llama-cpp] llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
[node-llama-cpp] llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
[node-llama-cpp] llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
[node-llama-cpp] llama_model_loader: - kv  19:               general.quantization_version u32              = 2
[node-llama-cpp] llama_model_loader: - type  f32:   65 tensors
[node-llama-cpp] llama_model_loader: - type q2_K:   65 tensors
[node-llama-cpp] llama_model_loader: - type q3_K:  160 tensors
[node-llama-cpp] llama_model_loader: - type q6_K:    1 tensors
[node-llama-cpp] llm_load_vocab: special tokens definition check successful ( 259/32000 ).
[node-llama-cpp] llm_load_print_meta: format           = GGUF V2
[node-llama-cpp] llm_load_print_meta: arch             = llama
[node-llama-cpp] llm_load_print_meta: vocab type       = SPM
[node-llama-cpp] llm_load_print_meta: n_vocab          = 32000
[node-llama-cpp] llm_load_print_meta: n_merges         = 0
[node-llama-cpp] llm_load_print_meta: n_ctx_train      = 16384
[node-llama-cpp] llm_load_print_meta: n_embd           = 4096
[node-llama-cpp] llm_load_print_meta: n_head           = 32
[node-llama-cpp] llm_load_print_meta: n_head_kv        = 32
[node-llama-cpp] llm_load_print_meta: n_layer          = 32
[node-llama-cpp] llm_load_print_meta: n_rot            = 128
[node-llama-cpp] llm_load_print_meta: n_embd_head_k    = 128
[node-llama-cpp] llm_load_print_meta: n_embd_head_v    = 128
[node-llama-cpp] llm_load_print_meta: n_gqa            = 1
[node-llama-cpp] llm_load_print_meta: n_embd_k_gqa     = 4096
[node-llama-cpp] llm_load_print_meta: n_embd_v_gqa     = 4096
[node-llama-cpp] llm_load_print_meta: f_norm_eps       = 0.0e+00
[node-llama-cpp] llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
[node-llama-cpp] llm_load_print_meta: f_clamp_kqv      = 0.0e+00
[node-llama-cpp] llm_load_print_meta: f_max_alibi_bias = 0.0e+00
[node-llama-cpp] llm_load_print_meta: n_ff             = 11008
[node-llama-cpp] llm_load_print_meta: n_expert         = 0
[node-llama-cpp] llm_load_print_meta: n_expert_used    = 0
[node-llama-cpp] llm_load_print_meta: pooling type     = 0
[node-llama-cpp] llm_load_print_meta: rope type        = 0
[node-llama-cpp] llm_load_print_meta: rope scaling     = linear
[node-llama-cpp] llm_load_print_meta: freq_base_train  = 10000.0
[node-llama-cpp] llm_load_print_meta: freq_scale_train = 0.25
[node-llama-cpp] llm_load_print_meta: n_yarn_orig_ctx  = 16384
[node-llama-cpp] llm_load_print_meta: rope_finetuned   = unknown
[node-llama-cpp] llm_load_print_meta: model type       = 7B
[node-llama-cpp] llm_load_print_meta: model ftype      = Q2_K - Medium
[node-llama-cpp] llm_load_print_meta: model params     = 6.74 B
[node-llama-cpp] llm_load_print_meta: model size       = 2.63 GiB (3.35 BPW)
[node-llama-cpp] llm_load_print_meta: general.name     = lmsys_vicuna-7b-v1.5-16k
[node-llama-cpp] llm_load_print_meta: BOS token        = 1 '<s>'
[node-llama-cpp] llm_load_print_meta: EOS token        = 2 '</s>'
[node-llama-cpp] llm_load_print_meta: UNK token        = 0 '<unk>'
[node-llama-cpp] llm_load_print_meta: LF token         = 13 '<0x0A>'
[node-llama-cpp] llm_load_tensors: ggml ctx size =    0.11 MiB
[node-llama-cpp] llm_load_tensors: offloading 0 repeating layers to GPU
[node-llama-cpp] llm_load_tensors: offloaded 0/33 layers to GPU
[node-llama-cpp] llm_load_tensors:        CPU buffer size =  2694.32 MiB

[node-llama-cpp] llama_new_context_with_model: n_ctx      = 4096
[node-llama-cpp] llama_new_context_with_model: freq_base  = 10000.0
[node-llama-cpp] llama_new_context_with_model: freq_scale = 0.25
[node-llama-cpp] llama_kv_cache_init: Vulkan_Host KV buffer size =  2048.00 MiB
[node-llama-cpp] llama_new_context_with_model: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
[node-llama-cpp] llama_new_context_with_model: Vulkan_Host input buffer size   =   192.08 MiB
[node-llama-cpp] llama_new_context_with_model: Vulkan_Host compute buffer size =  2304.00 MiB
[node-llama-cpp] llama_new_context_with_model: graph splits (measure): 1
radv/amdgpu: Not enough memory for command submission.
Segmentation fault (core dumped)

@scenaristeur As you can see from the logs, node-llama-cpp detected that you have Vulkan and used it by default.
It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.
For now, you can either disable the GPU support by passing gpu: false to getLlama:
import {getLlama} from "node-llama-cpp";

const llama = await getLlama({
    gpu: false
});

Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM.
To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu

@scenaristeur As you can see from the logs, node-llama-cpp detected that you have Vulkan and used it by default. It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.
For now, you can either disable the GPU support by passing gpu: false to getLlama:
import {getLlama} from "node-llama-cpp";

const llama = await getLlama({
    gpu: false
});

Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu

it's a 16 core CPU only, no GPU, i'll try getLlama with GPU false . thxs

@scenaristeur As you can see from the logs, node-llama-cpp detected that you have Vulkan and used it by default. It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.
For now, you can either disable the GPU support by passing gpu: false to getLlama:
import {getLlama} from "node-llama-cpp";

const llama = await getLlama({
    gpu: false
});

Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu

thxs ,
Works with GPU false, but i've lost conversationHistory, how to deal with conversationHistory in the beta version ?

@scenaristeur Please open a discussion for your questions; I don't want to notify everyone who watches this PR

@scenaristeur As you can see from the logs, node-llama-cpp detected that you have Vulkan and used it by default. It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.

For now, you can either disable the GPU support by passing gpu: false to getLlama:

import {getLlama} from "node-llama-cpp";

const llama = await getLlama({

gpu: false

});

Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:

npx --no node-llama-cpp inspect gpu

thxs ,
Works with "gpu:false", but i've lost conversationHistory, how to deal with conversationHistory in the beta version ? I'm working on a server where there can be multiple sessions, with multiple history, in what format should history be injected to a session ? to which class ?

If your application is GPL 3.0 compliant, feel free to inspire yourself here as to how that can go: https://github.com/nathanlesage/local-chat

@scenaristeur Please open a discussion for your questions; I don't want to spam with notifications everyone who watches this PR

transfered to https://github.com/withcatai/node-llama-cpp/discussions/176

switching from beta 13 to beta 14 gives me
file:///home/smag/dev/igora/node_modules/node-llama-cpp/dist/evaluator/LlamaModel.js:22
    constructor({ modelPath, gpuLayers, vocabOnly, useMmap, useMlock, onLoadProgress, loadSignal }, { _llama }) {
                                                                                ^

TypeError: Cannot destructure property '_llama' of 'undefined' as it is undefined.
    at new LlamaModel (file:///home/smag/dev/igora/node_modules/node-llama-cpp/dist/evaluator/LlamaModel.js:22:103)
    at new McConnector (file:///home/smag/dev/igora/src/mcConnector/index.js:30:13)
    at new Worker (file:///home/smag/dev/igora/src/worker/index.js:14:24)
    at file:///home/smag/dev/igora/index.js:28:16

Node.js v20.10.0

I am using beta version and the code below still after asking question (giving prompt) i don't get any output its totally blank can anyone please help as I am stuck here badly
import {fileURLToPath} from "url";
import path from "path";
import {getLlama, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const llama = await getLlama({
    gpu: false
});
const model = await llama.loadModel({
    modelPath: path.join(__dirname, "models", "phi-2-orange.Q2_K.gguf")
});
const context = await model.createContext({
    contextSize: Math.min(4096, model.trainContextSize)
});
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

const q1 = "Hi there, how are you?";
console.log("User: " + q1);

const a1 = await session.prompt(q1);
console.log("AI: " + a1);

const q2 = "Summerize what you said";
console.log("User: " + q2);

const a2 = await session.prompt(q2);
console.log("AI: " + a2);

@juned-adenwalla You can ask it in Discussion.
I'm now using it, works fine with GPU. You might want to look at #199 about __dirname issue. Or try different model like https://huggingface.co/Qwen/Qwen1.5-32B-Chat-GGUF

switching from beta 13 to beta 14 gives me
file:///home/smag/dev/igora/node_modules/node-llama-cpp/dist/evaluator/LlamaModel.js:22
    constructor({ modelPath, gpuLayers, vocabOnly, useMmap, useMlock, onLoadProgress, loadSignal }, { _llama }) {
                                                                                ^

TypeError: Cannot destructure property '_llama' of 'undefined' as it is undefined.
    at new LlamaModel (file:///home/smag/dev/igora/node_modules/node-llama-cpp/dist/evaluator/LlamaModel.js:22:103)
    at new McConnector (file:///home/smag/dev/igora/src/mcConnector/index.js:30:13)
    at new Worker (file:///home/smag/dev/igora/src/worker/index.js:14:24)
    at file:///home/smag/dev/igora/index.js:28:16

Node.js v20.10.0

was this addresses? I'm still seeing this in the beta branch today.

I have currently two issues preventing me from updating from beta.13 to beta.17.

Beginning with beta.14, loading models always fails with defaults that have worked in beta.13 with the error message that allegedly the settings require more VRAM than I have (given that the same settings work in beta.13, I believe this to be incorrect). I unfortunately don't know if the bug originates with this library or with llama.cpp
During development, the library itself will load fine, but after packed it will tell me that it requires the (old) ggml-meta.metal library which, as far as I can see, has been replaced with default.metallib – do you have an idea what might cause this? I am a bit confused that the library would complain about a missing library that appears to have been consciously replaced with a different target…?

It would be great if you could give me some pointers so that I can debug it!
(Also: You've mentioned in the changelog to beta.17 that the lib now supports Llama3, but I can confirm that beta.13 works fine with quantized Llama3-models!)

@nathanlesage Have you seen my response for your message in the feedback discussion?
I'd like to resolve your issues before I release version 3 as stable.

For anyone who sees this, please share your feedback on the version 3 beta feedback discussion and not on this PR.

{
  librarySearchPaths: [ 'C:\\Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5' ]
}
{ hasNvidiaDriver: true, hasCudaRuntime: false }
OS: Windows_NT 10.0.19045 (x64)
Node: 18.20.1 (x64)
TypeScript: 5.4.3
node-llama-cpp: 3.0.0-beta.22

CUDA: NVIDIA driver is installed, but CUDA runtime is not
Vulkan: available

{ gpu: 'vulkan' }
Vulkan device: NVIDIA GeForce RTX 3080
Vulkan used VRAM: 0% (0B/9.82GB)
Vulkan free VRAM: 100% (9.82GB/9.82GB)

CPU model: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
Used RAM: 33.18% (10.58GB/31.89GB)
Free RAM: 66.81% (21.31GB/31.89GB)

I have CUDA installed, but node-llama-cpp isn't finding it.
It fails here in detectCudaSupport:
```
asyncEvery([
asyncSome([
hasFileInPath("libcuda.so", librarySearchPaths),
hasFileInPath("libcuda.so.1", librarySearchPaths)
]),
asyncSome([
hasFileInPath("libcudart.so", librarySearchPaths),
hasFileInPath("libcudart.so.11", librarySearchPaths),
hasFileInPath("libcudart.so.12", librarySearchPaths),
hasFileInPath("libcudart.so.13", librarySearchPaths) // for when the next version comes out
]),
asyncSome([
hasFileInPath("libcublas.so", librarySearchPaths),
hasFileInPath("libcublas.so.11", librarySearchPaths),
hasFileInPath("libcublas.so.12", librarySearchPaths),
hasFileInPath("libcublas.so.13", librarySearchPaths) // for when the next version comes out
]),
asyncSome([
hasFileInPath("libcublasLt.so", librarySearchPaths),
hasFileInPath("libcublasLt.so.11", librarySearchPaths),
hasFileInPath("libcublasLt.so.12", librarySearchPaths),
hasFileInPath("libcublasLt.so.13", librarySearchPaths) // for when the next version comes out
])
])
because the files are actually in `'C:\\Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/bin'`, not `C:\\Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5`


Automated cherry pick of #8357

Cherry pick of #8357 on release-5.37.
/cc  @amyblais

/release-note-none


Aget to goog.object

tl;dr aget should be used on arrays and not on objects. goog.object/get should be used instead.
I've done this as per advice by Mike Fikes in his blog post here:
http://blog.fikesfarm.com/posts/2017-11-09-avoid-converting-javascript-objects.html
and a longer write up here:
https://clojurescript.org/news/2017-07-14-checked-array-access
It was also brought up in the issue here: https://github.com/seantempesta/cljs-react-navigation/issues/11

This looks great!  Thanks!


[WelcomeUX] Wrongly “unbound” shortcuts in playground

Hi,
VSCode 1.9 has a new editor playground that explains the different shortcuts in a prose:

On the above screenshot, I'm not sure if the sentence “press unbound or unbound” is the expected text in case there are no shortcuts bound for that particular actions or if it is a bug, but that certainly feels weird.
The second point is that I happen to have shortcuts defined for these actions (“add a new cursor above or below”), so there is certainly a bug here.
I'm on Ubuntu, with a French keyboard layout, if that may be of any relevance.

Related to https://github.com/Microsoft/vscode/issues/18902

@mquandalle Have you configured keybindings other than the default ones? If so, could you post that part of your keybindings.json?

Having the same issue, these are my keybindings:
[
    {
        "key": "ctrl+d",
        "command": "editor.action.deleteLines",
        "when": "editorTextFocus"
    },
    {
        "key": "ctrl+right",
        "command": "cursorWordStartRight",
        "when": "editorTextFocus"
    },
    {
        "key": "ctrl+shift+right",
        "command": "cursorWordStartRightSelect",
        "when": "editorTextFocus"
    },
    {
        "key": "ctrl+left",
        "command": "cursorWordStartLeft",
        "when": "editorTextFocus"
    },
    {
        "key": "ctrl+shift+left",
        "command": "cursorWordStartLeftSelect",
        "when": "editorTextFocus"
    },
    {
        "key": "ctrl+shift+s",
        "command": "workbench.action.files.saveAll"
    },
    {
        "key": "alt+shift+l",
        "command": "workbench.files.action.showActiveFileInExplorer"
    }
]

Sorry, deleted my previous comment, the issue was that I had Ctrl+Alt+UpArrow and Ctrl+Alt+DownArrow redefined.

@chrmarti, no I haven't customized these keybindings, though if I remember correctly the shortcut was specific to my particular keyboard layout. Weirdly I currently can't find these particular settings (add a cursor above or below) in the left panel of new settings UI.

@borekb Are you on Linux?

Yes, Ubuntu 16.04

I reported the missing setting entries in #19857.

Hum, I restarted VSCode, and the problem is now fixed:

Closing as not reproducible. Please reopen if you see this again and append your keybindings.json. Thanks.

Oh, I actually restarted code with --disable-extensions a few hours ago without realizing that this was the reason for the non-reproducibility.
So to be clear, I can reproduce the “press unbound and unbound” message again when starting code normally, with the extensions enable, and the problem doesn't appear if I launch code with --disable-extensions.
Do you have some tool to identify which extensions is causing the bug? Or do I have to switch them on and off one by one manually?

Filed Microsoft/vscode-sublime-keybindings#41.


Profermance Improvement with

Your codes really help me a lot, appreciate.
And I found a performance issue maybe can improve it:
The tridiagonal linear equation, I think you figure it out with Gaussian elimination algorithm. It is O(n^3) time complexity. While with the Thomas algorithm, the complexity can be reduced to O(n), Check the wikipedia here: https://en.wikipedia.org/wiki/Tridiagonal_matrix_algorithm

Thanks! In my case, I worked with 10 - 15 points at most, and I need to solve the equations at most 60 times per second, so the performance is more than sufficient.


Searching in an artist's albums

There seems to be no way to find an album from specific artist in spotify api endpoint, For example i want to find the Pink album from Ferry Corsten, We have two choice, Grab all album from the artist then search into it Or search album and get All results (for that album there is 4885 result item!), So is there a proper way to do this by this repo?

Hey!
It should be possible to use the SpotifyWebAPI::search() method and combining what Spotify calls "field filters" into a query. Take a look at this example and their docs.
The code could for example look like:
$api->search('album:pink artist:"Ferry Corsten"', 'album');


for...in loops cause errors

https://github.com/y-js/y-map/blob/master/src/Map.js#L22
Seems to be the same issue as in
https://github.com/y-js/yjs/issues/46

Dooone

sure? :)

:sweat_smile: Sorry. Now it's really done

Confirming, it works now! :+1: Cool!

:sparkles: :sparkles: :sparkles:


Autorest Regen Preview to 3.0.0-alpha.20231102.2 by Mingzhe Huang from refs/pull/3898/head

Triggered from https://github.com/Azure/autorest.csharp/pull/3898

Close since it's covered in #39526


Authorization lambda

This is an effort to make this into a multi-user POP server.
First, if you specify the name of an authorization-lambda, any "name" and "password" in the config.yaml file are ignored.
Second, after USER and PASS have been entered, the named lambda is invoked, and if it returns "OK", the user is considered authenticated.
Third, the user name is appended to the S3 Prefix, so that only their own files are visible.
The assumption here is that there can be a common Prefix for all users, and then files are differentiated by each users' name. E.g. assuming that the configured "aws-s3-prefix" is Inbox/mydomain.org, user John Doh will be able to see this email,
"Inbox/mydomain.org/John Doh/email.eml"
but not this one
"Inbox/mydomain.org/Jon d'Eaux/email.eml"

Closing this, so I can delete my local branch. I'll submit another one instead.


switch around meaning of high and low byte of crc as per implementation guide

test for crc.value() is also swapped, maybe a byte slice base interface such as hash.Sum(b []byte) []byte is better to express byte ordering.

Thank you.


Drag items from inbox into draft and remove form index

Seriously. Please stop trying to use any of this work


Better error message for missing member in tslib

@amcasey brought this up while upgrading Yammer to 3.6/3.7.
This syntax requires an imported helper named '__spreadArrays', but module 'tslib' has no exported member '__spreadArrays'.

The error message tells us something that is technically correct, but which is not clearly actionable. The user has no idea why __spreadArrays gets imported during an upgrade.
We should indicate that the user should try upgrading tslib.
This syntax requires an imported helper named '__spreadArrays' which does not exist in 'tslib'. Consider upgrading your version of 'tslib'.

@DanielRosenwasser did upgrading tslib fix the error?

@m-henderson yep.

@amcasey @DanielRosenwasser it looks like there are a lot of places that the message is referenced:

Are we going to want to recommend a tslib upgraded on every instance of the message or only when tslib as no exported member of _spreadArrays ?

It should just be a matter of updating the message in diagnosticMessages.json, running gulp generate-diagnostics and updating the place where the error is issued in src.

@DanielRosenwasser I opened a PR. Just let me know if any changes need to be made to it and I will take care of it.
https://github.com/microsoft/TypeScript/pull/33625

Thanks @m-henderson!

@DanielRosenwasser no problem! thank you for helping me out with the detailed comment :

It should just be a matter of updating the message in diagnosticMessages.json, running gulp generate-diagnostics, updating the place where the error is issued in src, and accepting new error message baselines. Please check out our CONTRIBUTING.md at the root of the repo.

Thanks!
Upgraded to 1.10.0 and all ok.
My error had been for ionic / cordova and the android logcat error was:
"Uncaught TypeError: tslib.__spreadArrays is not a function"