gae-pytz breaks if memcache API stub is missing

I ran into a scenario where using gae-pytz threw an exception because it expects the memcache API to be available.  My script utilizes the Remote API and is run from the command-line. Due to the order of our import chain through appengine_confg.py, importing webapp2_extras.i18n has a side-effect of calling pytz.  This all happens before we're able to initialize the API proxy stubs and it's a bit messy to change.
Relevant code is here.
It'd be great if gae-pytz were more of a drop-in replacement for pytz, and it could use environment variables to detect when it should use the Memcache API and other optimizations.
13:09:13   File "/app/src/www/lib/monkey.py", line 16, in <module>
13:09:13     from webapp2_extras import i18n as w2_i18n
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2_extras/i18n.py", line 17, in <module>
13:09:13     from babel import dates
13:09:13   File "/app/src/www/zip_imports/Babel-1.3.zip/babel/dates.py", line 28, in <module>
13:09:13   File "/app/src/www/zip_imports/Babel-1.3.zip/babel/util.py", line 278, in <module>
13:09:13   File "/app/src/www/zip_imports/Babel-1.3.zip/babel/localtime/__init__.py", line 74, in <module>
13:09:13   File "/app/src/www/zip_imports/Babel-1.3.zip/babel/localtime/__init__.py", line 70, in get_localzone
13:09:13   File "/app/src/www/zip_imports/Babel-1.3.zip/babel/localtime/_unix.py", line 96, in _get_localzone
13:09:13     return pytz.timezone(etctz.replace(' ', '_'))
13:09:13   File "/app/src/www/vendor/pytz/__init__.py", line 136, in timezone
13:09:13     _tzinfo_cache[zone] = build_tzinfo(zone, open_resource(zone))
13:09:13   File "/app/src/www/vendor/pytz/__init__.py", line 51, in open_resource
13:09:13     return loader.open_resource(name)
13:09:13   File "/app/src/www/vendor/pytz/gae.py", line 57, in open_resource
13:09:13     zonedata = memcache.get(cache_key)
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.py", line 559, in get
13:09:13     rpc = self.get_multi_async([key], namespace=namespace, for_cas=for_cas)
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.py", line 612, in get_multi_async
13:09:13     self.__get_hook, user_key)
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.py", line 381, in _make_async_call
13:09:13     rpc = create_rpc()
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/memcache/__init__.py", line 295, in create_rpc
13:09:13     return apiproxy_stub_map.UserRPC('memcache', deadline, callback)
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 414, in __init__
13:09:13     self.__rpc = CreateRPC(service, stubmap)
13:09:13   File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 68, in CreateRPC
13:09:13     assert stub, 'No api proxy found for service "%s"' % service
13:09:13 AssertionError: No api proxy found for service "memcache"

Use environment variables? Or just skip trying to get/set things in memcache if from google.appengine.api import memcache raises ImportError?

@davidwtbuxton I think either would be fine, more of an implementation detail.

@erichiggins no idea if that actually works, but I think it does what you want.
Please checkout the latest from this fork https://github.com/davidwtbuxton/gae-pytz/tree/works-without-memcache and let us know how it goes.


return to Main menu button

The program is needed to restart after plotting graph for new plotting in graph.
It should be returned to the main menu without the need for restart.

wont fix

wont fix


Fixed A werid indent and added a exeption handle

Added exception handling to Chuck Norris
Fixed #5
Fixed a weird indent

Thank you for the PR :)


Add yaml to explicitly create our webhook configs.

This imports knative/pkg#802 and explicitly creates the
webhook configurations that updates, and handles the breaking struct changes.
Related: knative/serving#5845

/lgtm


Estimate work needed

As preparation for the next presentation, add an estimate of the work needed to be done until this is able to be included in the MVP and create the necessary issues 🫡

🫡


Optimize release and benchmark builds

I've added some options in Cargo.toml to optimize the performance of release and benchmark builds.

lto or link time optimizations, allows optimizations across codegen units and even languages (for example when we are using v8). It also inlines most of the functions. See this and this
codegen-units Disallows the compiler to split code in multiple codegen-units, which results in a bit better performence.
opt-level Level at which the compiler optimizes things. (e.g unrolling loops, ...)

This of course causes the build to take much longer. From my measurements a full release build takes about 23.3s instead of 10.8s on my machine. A rebuild (with some dependencies already built) takes 17.3s instead of 2.8s.
Development builds are untouched, so they only take 6.2s with or without these options.
The benchmark results go up ~10-11%.

 Benchmark results
TreeIterator - wikipedia

TreeIterator - stackoverflow

TreeConstructor

Tokenizer

pr author check fail is fine. This probably needs finetuning or we might need to get rid of it if it doesn't work out well enough..

pr author check fail is fine. This probably needs finetuning or we might need to get rid of it if it doesn't work out well enough..

Probably because I didn't add myself to the file.

maybe, but that check is still broken :-)


HP Gen8 servers hang when using "localboot 0"

Hi,
Due to a BIOS bug on some HP servers (mostly Gen8) they hang when PXE boot try to do a "localboot 0".
Instead we need boot directly from the HD:
label localboot
com32 chain.c32
append hdX (where X is your booting harddrive).
The chain.c32 file must be available from the tftp server for the server to download it.

The previous commit will ease what you suggest here. We should have done that before. Thanks.


Hide green line and labels when mouse isn't over

In this PR we are just drawing

Green line in the TimeGraph
Labels in Variable/PageFault/Memory Tracks
MouseLabel in the Timeline

only when the mouse is over the TimeGraph and also inside the visible
time range. This last case is to avoid the situation of having the mouse on
top of the right-margin or the vertical slider resulting on partially visible
labels:
http://screenshot/7M7dd7q4wTJe49P
Bugs:
http://b/231160028
http://b/233325318
Screenshots:
Mouse over TimeGraph: http://screenshot/9aUnjp8NqfPduyA
Mouse not over TimeGraph: http://screenshot/37JoH6YRGBVwPPM
Test: Load a capture, move the mouse around Orbit.

Could this break E2E tests?

Yes, green light :)

Are there some low hanging fruits that could be unit tested?

Good question. Ideally we should have:

TimelineUI unit tests
GraphTrack unit tests
TimeGraph unit tests to check the mouse_tick is in between ranges or nullopt.
TimeGraph integration tests about mouse inside or outside TimeGraph and number of elements drawn in TimelineUI and GraphTrack

I added test for the first one. The second one is easy but since there is no Track unit tests I won't start them. The third one we should include it when we add drawing tests to the TimeGraph (ideally soon). The forth one I think it's not a priority for now.


Blocky "username" translates to user instead of chatID

I am not sure if this is intended or if user has a function at all, but shouldn't "username" be the chatId so we can ad a text field to Username in blocky in order to text a specific person?
So far this only works when we use chatId: 'XXX' instead of user in JS.
Thanks in advance!

Username is username, chatID is chatID..
ChatID and Username are linked together.. I'm not sure what exactly you
mean with that..?
It is possible to send a message to a specific user by using the Username
or the ChatID..

for example if I put in block the username 123456789 it will be like this in written code, but it wont send anything to anyone, most importantly nothing to the user specified.
sendTo("telegram.0", "send", {         text: 'Test Message',         user: '123456789'     });
But if i change user with chatId then it will send the message to that specific user
sendTo("telegram.0", "send", {         text: 'Test Message',         chatId: '123456789'     });

By the way.. If that issue is regarding blocky you should open an issue at https://github.com/ioBroker/ioBroker.javascript

Username has to be the username. Not the ChatID!
Check the data point telegram.0.communicate.users there is a list of chatID and the linked username.

Here is how my datapoint is looking like..
{"40452xxxx":"BuZZy","41648xxxx":"otherUser"}
So my telegram username is BuZZy and the chatID is 40452xxxx.

okay before I used the whole thing with the ID and that is why it did not work. So in your case if you put BuZZy as username in blocky it will text you directly.
Is the username unique? Cause I have not chosen any and it basically used my first name to as username, so I am wondering if the user is not unique wouldn't it be more accurate to use the ID? Unless I missed a step where I set my first name as my username (doubt though)
Ill submit this again in iobroker.javascript if users are not unique.
Thanks for the help, much appreciated!

Never mind, sometimes my brain is blocked completely. I just checked the app, I can set a username for myself now and it is unique, but if you never set one the adapter will use your first name as username!
My bad!

Well.. The Telegram Adapter picked my firstname as Uername too.. But i changed it to "BuZZy" in the example that i posted above..
Not sure if it picks the "public username" that you can set in the telegram app on your phone..
I connected the bot to my telegram app before i entered a public username in the telegram app.. :smiley:

P.s.: don't forget to close this issue here on github.. ;)

I did not forget xD, I was testing he username thing and it does not work... Adapter assumes first name is the username. Even after I deleted everything and reinstalled and setup a bot again.
With chatId it is still far more reliable and reacts all the time, when using username (with my first name) to send a message to a specific person it sometimes for some reason does not send the message until a 2nd or 3rd trial.
So second problem is that the adapter takes first name as username instead of actual username.

Issue moved to iobroker.javascript

Uhhmm.. Yeah, but now the issue is clearly related to telegram... So you should close the issue on the Javascript repo and reopen this one here! Haha.. Sorry about that.

Or better: open a new issue here in the telegram repo which describes the "new problem" that you have now.


Cursors

Builds on top of attachments, so the diff will be easier to swallow once that is merged...

The merge conflicts are because I renamed FileResource to Attachment on master. Sorry!


Data Race in watcher reconciler test

Expected Behavior
No data race in unit tests
Actual Behavior
From https://prow.tekton.dev/view/gs/tekton-prow/pr-logs/pull/tektoncd_results/226/pull-tekton-results-unit-tests/1565331909276340224
WARNING: DATA RACE
Read at 0x00c0006d7243 by goroutine 81:
  testing.(*common).logDepth()
      /usr/local/go/src/testing/testing.go:753 +0xa6
  testing.(*common).log()
      /usr/local/go/src/testing/testing.go:746 +0x8f
  testing.(*common).Logf()
      /usr/local/go/src/testing/testing.go:792 +0x21
  testing.(*T).Logf()
      <autogenerated>:1 +0x75
  go.uber.org/zap/zaptest.testingWriter.Write()
      /home/prow/go/pkg/mod/go.uber.org/zap@v1.22.0/zaptest/logger.go:130 +0x153
  go.uber.org/zap/zaptest.(*testingWriter).Write()
      <autogenerated>:1 +0xad
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /home/prow/go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/core.go:90 +0x1e7
10 skipped lines...
  knative.dev/pkg/controller.(*Impl).processNextWorkItem()
      /home/prow/go/pkg/mod/knative.dev/pkg@v0.0.0-20211115071955-517ef0292b53/controller/controller.go:542 +0x72f
  knative.dev/pkg/controller.(*Impl).RunContext.func3()
      /home/prow/go/pkg/mod/knative.dev/pkg@v0.0.0-20211115071955-517ef0292b53/controller/controller.go:478 +0x6c
Previous write at 0x00c0006d7243 by goroutine 35:
  testing.tRunner.func1()
      /usr/local/go/src/testing/testing.go:1180 +0x52c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1197 +0x22a
Goroutine 81 (running) created at:
  knative.dev/pkg/controller.(*Impl).RunContext()
      /home/prow/go/pkg/mod/knative.dev/pkg@v0.0.0-20211115071955-517ef0292b53/controller/controller.go:476 +0x21a
  knative.dev/pkg/controller.StartAll.func1()
      /home/prow/go/pkg/mod/knative.dev/pkg@v0.0.0-20211115071955-517ef0292b53/controller/controller.go:789 +0xaf
Goroutine 35 (finished) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1238 +0x5d7
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1511 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1193 +0x202
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1509 +0x612
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1417 +0x3b3
  main.main()
      _testmain.go:49 +0x236

Steps to Reproduce the Problem

Additional Info

Kubernetes version:
Output of kubectl version:
(paste your output here)

Tekton Pipeline version:
Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

Looks like the issue here is that the controllers and test are using the same logger, and the controllers are running in a separate goroutine. Since this depends on pipeline repo code, it may be hard to untangle.
@dibyom can we convert these to run the controller reconcile functions serially?

@dibyom can we convert these to run the controller reconcile functions serially?

Yeah, not sure how much effort that is but that sounds like a more better way of testing this

Race condition likely arises from here: https://github.com/tektoncd/results/blob/main/pkg/watcher/reconciler/reconciler_test.go#L51-L53
I think to resolve this, we should:

Ensure that unit-level (synchronous) tests check the result annotation is set
Ensure that e2e (asynchronous) tests check the result annotation is set

/assign @enarha

With the e2e tests now enabled and the same functionality covered by the TestTaskRun and TestPipelineRun tests, I see 3 options going forward with this and unblocking the release:

Remove the unit tests.
Keep the unit tests, but skip them. They can be later used as a basis for improvement.
Improve the tests making them synchronous as suggested earlier. This will be the most time consuming and this task is not the highest in my priority list, but I'll eventually get to it.

Opinions?

I am in favor of option 1 - we have the same test as proper E2Es at both the TaskRun and PipelineRun level:

https://github.com/tektoncd/results/blob/main/test/e2e/e2e_test.go#L95-L98
https://github.com/tektoncd/results/blob/main/test/e2e/e2e_test.go#L146-L149

#316 was merged. The issue can be closed.

Faulty test was re-added in https://github.com/tektoncd/results/commit/073b4707f7d3941b0e2932ee96fa11518d6e5249


Update deploy documentation workflow

Update deploy documentation workflow
Summary
The deploy documentation workflow now executes the steps sequentially to deal with the dependencies

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


How to visualize validation images on tensorboard?

in your tensorboard_hook.py, you have this line
"log_images = runner.outputs.get('log_imgs')"
But this seems to only give the images from training set.

Currently, our codebase can only log images during the training stage. If you'd like to extend the codes to support log images during the validation stage, you should do some hacks in the tensorboard_hook.py. While I have no plan to add this support, you can make a pr to share your implementation!

Thatis the thing. I do not see how to get validation images from tensorboard_hook.py? It always returns the training images regardless if it is validation phase or training phase.

Yes, note that we only add images to loggers when training. Here is what we do now:

First, see this line of codes, we add log images to depth_head's losses for a temporal save.
Then, see this line of codes, we conduct a post-process of these loss terms mixed up with real losses and log images, dividing them into loss terms and images, respectively.
Finally, log images are saved as a key-value pair in training output as shown in this line of codes
Hence, in tensorboard_hook this line of codes, after ensuring this is the training stage, we can get the log images from the runner.outputs.get('log_imgs').

For your case, if you wanna visualize images during validation, you need to do some hacks in eval_hooks.py. Remember the validation is achieved via hooks (This is a feature of MMcodebase). See this line of codes, your logging process is achieved here. So you need to consider how to pass the log images to this function, where you can imitate our solutions for training stage logging.

Close this issue for now.


Can theano be removed totally?

Given theano will not be updated any more ,i wonder wether merlin would drop it in future version. Why there is no LHUC and MDN layer in keras model or tensorflow model ,is run_merlin.py and run_keras_with_merlin_io.py equal  in model level?

If you are considering making some of the above suggested contributions, please feel free to send a pull request and someone will take a look

I'm trying to do this,but it take time.I'll pull request later.

This repo  is what i am writing ,just a demo version and not finished yet.


give analytics its own storage volume

See also: https://github.com/istio/istio/issues/4610

If we provide shared storage for adapters, we need to audit and add tests for the plugin's analytics storage to ensure there won't be potential conflicts between multiple running adapters configured for the same environment as well as those configured for multiple environments.

I think this can be closed and left as an exercise for the deployer if they wish. It should be noted that multiple adapter instances should NEVER share the same volume and AX buffer location.... as on startup, the crash recovery will pick up any files in the temp or staging directory and handle them.


[Bug]: lora base_model.model.lm_head.base_layer.weight is not supported

Your current environment
Collecting environment information...
PyTorch version: 2.2.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version: (Debian 12.2.0-14) 12.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.36

Python version: 3.11.9 (main, Apr 10 2024, 14:54:51) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.36
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3

Nvidia driver version: 550.54.14
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             96
On-line CPU(s) list:                0-95
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Gold 6442Y
CPU family:                         6
Model:                              143
Thread(s) per core:                 2
Core(s) per socket:                 24
Socket(s):                          2
Stepping:                           8
BogoMIPS:                           5200.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          2.3 MiB (48 instances)
L1i cache:                          1.5 MiB (48 instances)
L2 cache:                           96 MiB (48 instances)
L3 cache:                           120 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94
NUMA node1 CPU(s):                  1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] onnxruntime==1.17.3
[pip3] torch==2.2.2
[pip3] torchvision==0.17.2
[pip3] triton==2.2.0
[conda] Could not collectROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV6	NV6	NV6	0,2,4,6,8,10	0		N/A
GPU1	NV6	 X 	NV6	NV6	0,2,4,6,8,10	0		N/A
GPU2	NV6	NV6	 X 	NV6	1,3,5,7,9,11	1		N/A
GPU3	NV6	NV6	NV6	 X 	1,3,5,7,9,11	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks```

### 🐛 Describe the bug

I fine tuned the Mixtral 8x7b  model with lora ,  I included the layer **lm_head** in lora    so in my tensors has this "base_model.model.lm_head.base_layer.weight , in  code vllm/lora/utils.py" : it doesn't support or ignore it .

      def parse_fine_tuned_lora_name(name: str) -> Tuple[str, bool]:
          """Parse the name of lora weights.

          args:
              name: the name of the fine-tuned LoRA, e.g.
                  base_model.model.dense1.weight
          return:
              Tuple(module_name, is_lora_a):
                  module_name: the name of the module, e.g. model.dense1,
                  is_lora_a whether the tensor is lora_a or lora_b.
          """
          parts = name.split(".")
          assert parts[0] == "base_model"
          assert parts[1] == "model"
          if parts[-1] == "weight":
              assert parts[-2] == "lora_A" or parts[-2] == "lora_B"
              return ".".join(parts[2:-2]), parts[-2] == "lora_A"

          if parts[-1] == "lora_embedding_A" or parts[-1] == "lora_embedding_B":
              return ".".join(parts[2:-1]), parts[-1] == "lora_embedding_A"

          raise ValueError(f"{name} is unsupported format")

I am getting this error:
    Traceback (most recent call last):
  File "/workspace/vllm/lora/worker_manager.py", line 139, in _load_lora
    lora = self._lora_model_cls.from_local_checkpoint(
  File "/workspace/vllm/lora/models.py", line 227, in from_local_checkpoint
    return cls.from_lora_tensors(
  File "/workspace/vllm/lora/models.py", line 148, in from_lora_tensors
    module_name, is_lora_a = parse_fine_tuned_lora_name(tensor_name)
  File "/workspace/vllm/lora/utils.py", line 33, in parse_fine_tuned_lora_name
    assert parts[-2] == "lora_A" or parts[-2] == "lora_B"
AssertionError

hello, i have same problem, any thing to fix this?

VLLM is designed to support multiple LORAs on top of a single base model. Your adapter includes lm_head weights, which likely means you have changed the vocabulary size, altering the dimensions of the embedding layer and the prediction head. This change disrupts the concept of one base model supporting multiple LORAs, as it requires multiple layers of lm_head to be stored.
A workaround is to save your base model with the new vocabulary and then load it. Alternatively, a cleaner solution would be to merge your LORA with the base model and then load it using VLLM.

A workaround is to save your base model with the new vocabulary and then load it.

@arshadshk regarding the workaround. What do you mean by 'save your base model with the new vocabulary'?
Is it wrong to expect that if it works with transforms (and peft) smoothly, it should also work in vLLM? if so, possible to throw some light into this?

A workaround is to save your base model with the new vocabulary and then load it. Alternatively, a cleaner solution would be to merge your LORA with the base model and then load it using VLLM.

Can you explain how to do this workaround? I have a need to serve multi-lora without merging that has this issue with modules_to_save

Another case, this issue is because you are finetuning the adapter with special tokens added.
This is not an expected behavior of VLLM, they should fix this.

somebody fix this pls

Has this issue been fixed or is there another PR that addresses this issue?


Import mean from Statistics on 0.7.

I used the same version number as in https://github.com/JuliaArrays/StaticArrays.jl/pull/454.

I'm going to go ahead and tag version 0.7.1, hope you don't mind 🙂.

Not at all - thank you!


[CRITICAL] [VIP-Travel] Create new Trip Room

VIP-Travel Implementation Issue
Tracking Issue: https://github.com/Expensify/Expensify/issues/333471
Project: https://github.com/orgs/Expensify/projects/112/
Relevant Design Doc Section: https://docs.google.com/document/d/122DLLTuCsU4NC3c7rNiy4UF8bZxVlLEWSy3yncqCTcQ/edit#bookmark=id.g77lx534xj1g

A new component, TripDetailsView will be created in src/components/ReportActionItem:

It will display and behave mimicking MoneyReportView’s behavior.
A new chat type, POLICY_TRIP_ROOM, will be added to CONST.ts.
A new helper function, isTripRoom(report) will be added to ReportUtils. It will return true if the provided report type is CONST.REPORT.TYPE.CHAT, and the chat type is the new CONST.REPORT.CHAT_TYPE.POLICY_TRIP_ROOM
It will be displayed from ReportActionItem if isTripRoom returns true.
It will take the associated expense report as parameter and:
Use the new getTripTransactions function to retrieve all transactions with trip data to display.
Display them as shown above using the existing MenuItemWithTopDescription like we do for money reports.
On click, it will open the trip details panel via a new route, ROUTES.TRAVEL_DETAILS

Routes table:
https://docs.google.com/document/d/122DLLTuCsU4NC3c7rNiy4UF8bZxVlLEWSy3yncqCTcQ/edit#bookmark=id.gn6jbl2udlnx

Hey! I’m Wojtek from Software Mansion, the expert agency, and I’d like to work on this issue.

Adding a C+ here for Travel reviews: @shubham1206agra

Only assigned on Friday. We can give them a day here Melvin.

Are we waiting on anything internal here?

Hi! I'm Aleksandra from Software Mansion, the expert agency, and I would like to take it over from Wojtek and work on this issue.

@smelaa only assigned on Friday.

Hi! I have some questions regarding this issue :) I've marked on the screenshot elements related to the questions below.

What should the rows with travel details look like when hovering over these elements? I'm specifically asking about the background color of the icon, I'm also attaching a screenshot showing the current hover style of this element.
I need to know the logic for displaying date text. Could you please give me similar examples as here for different years, months etc?
Which field from this data structure should be used to display this text (record locator)? If it's not known yet, can we adjust it once the backend is ready?

We have a value in our color system for rowHover, which is product300. If you use that, we shouldn't get any collisions with the icon BG color:

@shawnborton @dannymcclain

I need to know the logic for displaying date text. Could you please give me similar examples as here for different years, months etc?

Could you take a look at it when you have time? :)

I would just use the same? Not sure why they would be different. Will definitely let Shawn weigh in too, but I can't think of a reason for them using different date formatting.

Yeah that's my thinking too. I think if you are looking at a date that has a year other than the current year we are in, it should show the year. But if you are looking at a date within the current year, no need to show the year.

Ok, I asked because I had noticed on the designs that the date text also includes the day name, flight time, etc. and I was wondering if I could apply the same rules for date text displaying which was used for TripRoomPreview.

Oh, are you asking about the date here for instance?

Exactly :)

Ok, and when years are different, we'll display just: Friday, Dec 29, 2023 to Tuesday, Jan 2, 2024. Am I right?

I think so, as well as if you are looking at a flight that happened in a previous year.

Which field from this data structure should be used to display this text (record locator)? If it's not known yet, can we adjust it once the backend is ready?

This should just be start.date and the confirmation number in the bottom left is confirmations.value. For the overall trip room there is also a startDate and endDate in the overall report data https://docs.google.com/document/d/122DLLTuCsU4NC3c7rNiy4UF8bZxVlLEWSy3yncqCTcQ/edit#bookmark=id.pa0o1z2qu2nd

Ok, I was thinking about confirmations, but I see on the screen attached in the design doc that it's an array. So, I believe that we can just display the value of the first element in this array. Do you think it's ok?

I was thinking about the confirmation number (look at the rectangle nr 3). I know which fields I should use to display date text, but I was wondering which one should be used to display the confirmation number.

Latest comment: https://expensify.slack.com/archives/C05S5EV2JTX/p1712244259890909?thread_ts=1712164287.109189&cid=C05S5EV2JTX

Still waiting on webhooks.

Waiting on the backend.

Waiting on the backend still

Spotnana thread for backend updates here: https://expensify.slack.com/archives/C066REXJ9D3/p1713220956933639

Chatting about getting the data here: https://expensify.slack.com/archives/C05S5EV2JTX/p1713784714491629?thread_ts=1713545844.230119&cid=C05S5EV2JTX

Updates in https://expensify.slack.com/archives/C05S5EV2JTX/p1714085516706139?thread_ts=1713825695.223409&cid=C05S5EV2JTX

Update here: https://expensify.slack.com/archives/C05S5EV2JTX/p1714431147318139. Waiting on Spotnana.

@rushatgabhane has a PR above.

Updates here - https://expensify.slack.com/archives/C05S5EV2JTX/p1716196862504789?thread_ts=1715723531.626149&cid=C05S5EV2JTX
waiting for car details

@shubham1206agra this is ready for your review. There are a few backend issues  here and here but it shouldn't block anything.

PR hit production 2 days ago.

@shubham1206agra please complete checklist for this one

@twisterdotcom could you please attach payment summary 🙇

Payment Summary: ($500 as originally this was CRITICAL):

@rushatgabhane paid $500 via newDot requests
@shubham1206agra paid $500 here

$500 approved for @rushatgabhane


Import performance (slowness) considerations

The Import can be very slow if you have somewhat larger scans to be imported. I've heard some people complaining about this on Slack and other places.
From a high level there are 2 steps that we can take. They are not mutually exclusive, but I would like to start with step 2.

Make (re-)imports asynchronous.
This seems like a tempting approach because it would make the initial response very fast. But it also has a lot of downsides:

How do we report back to the user the import was finished? What was the result?
What do we do when an import doesn't finish or crash?
What if an import is still running and (another) user tries to import the same file concurrently?
What if the current (cicd) script using the API expects some stats in the response? (nr of issue imported, nr of duplicates, ...)
What about the performance impact of running large imports in the background?
Because there are some relatively easy and big gains to be made in step 2, I think we should go for that first.

Simplify and Optimize (re-)import
We don't want to overoptimize, but looking at the import code there are some big performance gained to be made from simplifying things.
a) finding.save() is called 4 for times for each new finding.
serializer.py calls finding.save() twice. And Finding.save() itself calls super().save() twice.
This causes django-auditlog to have to calculate differences 4 times. And this causes 4-8 extra db queries. django-watson will update its search index 4 times, also 4 extra queries.
Simplifying this to be exactly one save() call will instantly make the import 4 times faster.

b) watson search index is updated syncrhonously during import.
This could be made asynchronously to speed up more. Might need some help from django-watson authors as they are using post_save signals which are synchronous by design
edit: turns out there is a middleware that deduplicates watson updates, which already helps. PR #2066 . Might be an interesting approach for auditlog as well.
c) notifications are created synchronously
By design defect dojo creates a notification for each user in the system. I have over 50 users in my system, resulting in 50 database queries. This could be made asynchronously and should be limited to authorized users for each product.
d) calculate product grade is performed after updating of * each finding*.
This should be done only at the end of the import once for each product. And preferably asynchronously.
I realize that starting this journey (simplifying) will impact some core pieces of code. So I would like to hear some feedback on this before starting. We could consider starting a 2.0 branch where we do stuff like this, including removing api v1, unify UI and api_V2 logic, ...

Is something Matt T & I have been discussing recently. And the UI changes that would be necessary and what additional warnings we'd have to put next to metrics while uploads are being processed.

Too bad no further work/discussion has been done on this issue


Update dist files with module wrapper

Will do this myself when I cut the next release (very soon!)


version is too dynamic

Describe the bug
https://github.com/aws-quickstart/taskcat/blob/develop/setup.py#L5-L7
Looks to me like anyone running this script will end up with a version number that depicts install time, not build/release time.
To Reproduce
python setup.py install > /dev/null ; taskcat --version ; sleep 61 ; python setup.py install > /dev/null ; taskcat --version
Expected behavior
version numbers should reflect build or commit time, not install time

What if we mod setup.py to only do a version bump when build is passed in as an argument to setup.py then edit this line to python setup.py build && python setup.py install
Anyone have a better way to handle this?


Accordion component

Accordion component to be used in the Column Mapping Modal and any other time we might need an :accordion:.
(JIRA ticket) Figma
Description

The content is hidden by default. Clicking on title or chevron with display the hidden content

ARIA
The aria-expanded and aria-controls tags were used for accessibility.  More info can be found here
Screenshots
Closed

Open

Tests

Tests for rendering and non-rendering of hidden content were added

GIFs/Memes
slaps hood This bad boy can hold so much content

Reviewer Checklist
This section is to be filled out by reviewers
Testing

[ ] This code was tested by somebody other than the developer. Do not merge until this has been done.

I agree with @Mrtly, those transitions would be nice to have!

Thanks for the feedback @Mrtly  and @juanfriss !


add loading effect when requesting an email

Add a loading effect when requesting an email to prevent users from clicking multiple times before the request is completed.

Thanks for your interest. Will have a look after the existing PR is merged.

Thanks for your interest. Will have a look after the existing PR is merged.

Thanks for letting me know. Is there a specific procedure for contributing to the repo? Should I first submit an issue before creating a pull request?

No. There is no specific procedure.


Fix metadata file resolution when inferred pattern is **

Fix #6442

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


"Suhaia" is listed in the drop-down menu

Description:
"Suhaia" is listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Suhaia" in the search bar.

Expected results
The location is listed in the drop-down menu as "Sat Suhaia, TELEORMAN (Suhaia)" and "Comuna Suhaia, TELEORMAN".


feature: Improve handling of whitespace in tables with links in their cells

Is your feature request related to a problem? Please describe.
The issue arises when working with a table that contains links of varying size in the markdown code and the user elects to vertically align table columns in the code, such as here:

which then renders as:

This wastes a lot of screen space on white space and makes the table hard to read.
Describe the solution you'd like
It would be preferable if the table rendered as such, with the padding stripped:

Describe alternatives you've considered
The user could elect to not vertically align the pipes of the columns. But this makes the markdown document more difficult to use for other people viewing it in plain text (such as coworkers not using this plugin).

Additional information
The relevant section of the table for your convenience:
| Points To                                                                                | Level   |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| [[TTRPG/WWN/Caeraema/Locations/Pike Cave/_Pike Cave\|Pike Cave]]                                                                                | Low     |
| [[TTRPG/WWN/Caeraema/Locations/Floria's Tower/_Floria's Tower\|Floria's Tower]]                                                                                | Low     |
| [[TTRPG/WWN/Caeraema/Locations/The Sunken Fane/_The Sunken Fane\|The Sunken Fane]]                                                                                | Low     |
| [[TTRPG/WWN/Caeraema/Locations/The Sunken Fane/_The Sunken Fane\|The Sunken Fane]]                                                                                | Low     |
| [[TTRPG/WWN/Caeraema/Locations/The Sunken Fane/_The Sunken Fane\|The Sunken Fane]]                                                                                | Low     |
| [[TTRPG/WWN/Caeraema/Locations/Ruins of Stormwatch/_Ruins of Stormwatch\|Ruins of Stormwatch]]                                                                               | Low     |
| [[_The Haunted Mill\|The Haunted Mill]]                                                                                | Low     |
| [[_The Whispering Hollow]]                                                                                | Low     |
| [[_The Whispering Hollow]]                                                                                | Low     |
| [[_The Logging Venture]]                                                                                | Low     |
| [[_Marauder Cave\|Marauder Cave]]                                                                                | Low     |
| [[_Owl's Roost\|Owl's Roost]]                                                                                | Low     |
| [[_Blighted Orchard]]                                                                                | Low     |
| [[_Gloomwood Asylum\|Gloomwood Asylum]]                                                                                | Low     |
| [[_Witchfire Plains\|Witchfire Plains]]                                                                                | Low/Mid |
| [[_The Bloodpit]]                                                                                | Low/Mid |
| [[_Titan Shrine\|Titan Shrine]]                                                                                | Low     |
| [[_Alatryn's Tower\|Alatryn's Tower]]                                                                                | Low     |
| [[_Passing Forest Chapel\|Passing Forest Chapel]]                                                                                | Low     |
| [[TTRPG/WWN/Caeraema/Locations/Serpent's Isle/_Serpent's Isle\|Serpent's Isle]]                                                                                | Mid     |
| [[TTRPG/WWN/Caeraema/Locations/Serpent's Isle/_Serpent's Isle\|Serpent's Isle]]                                                                                | Mid     |
| [[_Goblin-Kin Caves]]                                                                                | Mid     |
| [[_Jungle Fever Hollow\|Jungle Fever Hollow]]                                                                                | Mid     |
| [[_Necrotic Glade]]                                                                                | Mid     |
| [[_Quicksilver Pool\|Quicksilver Pool]]                                                                                | Mid     |
| [[_Toxic Fungus Hollow\|Toxic Fungus Hollow]]                                                                                | Mid     |
| [[_Mangrove Cove]]                                                                                | Mid     |
| [[_Bog of Eternal Rain]]                                                                                | Mid     |
| [[_Bloodwood Grove]]                                                                                | Mid     |
| [[_Shipwreck Cove]]                                                                                | Mid     |
| [[_Jungle Sinkhole]]                                                                                | Mid     |
| [[_Wyrmfang Bluff]]                                                                                | Mid     |
| [[_Greybeard's Rest]]                                                                                | Any     |
| [[_Soulfire Crag]]                                                                                | Mid     |
| [[_Serpent's Gorge]]                                                                                | Mid     |
| [[_Starfall Observatory]]                                                                                | Any     |
| [[_Bloodfang Caverns]]                                                                                | Mid     |
| [[_Dwarven Bridge Ruins]]                                                                                | Mid     |
| [[_Dwarven Lighthouse]]                                                                                | Mid     |
| [[_Golem Graveyard]]                                                                                | Mid     |
| [[_Thunderpeak Plateau]]                                                                                | Mid     |
| [[_Moonshadow Plateau]]                                                                                | Mid     |
| [[_Bonegrinder Peak]]                                                                                | Mid     |
| [[TTRPG/WWN/Caeraema/Locations/Thaladrimarixtur, The Eternal Blizzard's Lair/_Thaladrimarixtur, The Eternal Blizzard's Lair\|Thaladrimarixtur, The Eternal Blizzard's Lair]] | High    |

I recognize that this issue can be somewhat mitigated by having shorter links and then deleting the whitespace and ---s causing a lot of the padding, but this issue would also exist for http hyperlinks which cannot necessarily be shortened in this way.
Thank you for this terrific plugin and I hope you have a wonderful day <3

I'll see what I can do, will likely be a new cell style if it's not too much of a pain to implement.

Added a new cell value trimmed here: https://github.com/MeanderingProgrammer/render-markdown.nvim/commit/c68697085441d03a20eee15d4d78e2e5a771569a
That should handle this like you describe. Update the plugin and set the configuration like so:j
require('render-markdown').setup({
    pipe_table = { cell = 'trimmed' },
})

Works like a charm
I verified it with the table above, as well as with tables that place the links in the not-first column and found no problems.
Furthermore, I tried to break it by removing part of the whitespace / the header ---s in some lines / unaligning the column pipes; the table rendered correctly each time.
The documentation for the feature looks good / is understandable to me as well.
As far as I am concerned, this feature request is completed.
Thank you very much for the change!
Is there a way to buy you a coffee?
In any case cheers and enjoy your weekend!

Awesome, thanks for stress testing it for me!
Thanks for the offer, is much appreciated. I added a donation link: https://donate.stripe.com/4gw2bSbwA5gw5s48ww.
Hope you enjoy your weekend as well.


s/relavent/relevant/

Fixes a small typo in a modal.

Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! :smile:

I signed the CLA, I closed and opened the push so the test would pass.

Ah cool! Thanks @etcet!


Markdown Code Fence Block Support

Description
I've originally filed an issue on nvim-treesitter here, but upon further inspection, I'm not sure if this is a treesitter issue anymore. It seems that the end pair isn't being indented appropriately, although the indent size should be ok.
Mapping  bug
1.If you report a bug about indent. Please remember that plugin doesn't do anything about indent.
It just trigger the indent of your vim config so if you have wrong indent config then it will do wrong indent.
You can check by select a block of code and press ==
2. provide result of command :verbose imap <cr>.
Steps to reproduce
No response
Minimal config
vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvim/site]]
local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'
local function load_plugins()
  require('packer').startup {
    {
      'wbthomason/packer.nvim',
      {
        'windwp/nvim-autopairs',
      },
      -- ADD PLUGINS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. '/plugin/packer_compiled.lua',
      display = { non_interactive = true },
    },
  }
end
_G.load_config = function()
  require('nvim-autopairs').setup()
end
if vim.fn.isdirectory(install_path) == 0 then
  print("Installing nvim-autopairs and dependencies.")
  vim.fn.system { 'git', 'clone', '--depth=1', 'https://github.com/wbthomason/packer.nvim', install_path }
end
load_plugins()
require('packer').sync()
vim.cmd [[autocmd User PackerComplete ++once echo "Ready!" | lua load_config()]]

Read please.

.If you report a bug about indent. Please remember that plugin doesn't do anything about indent.
It just trigger the indent of your vim config so if you have wrong indent config then it will do wrong indent.
You can check by select a block of code and press ==

Hi @windwp, I think you've misunderstood my issue. I've clarified that this is not due to incorrect indentings, but rather the endwise pair not being placed appropriately :)


fix: comepng-upload-data-duplication

main / 🐍[pytest]lib/dl_api_lib:db (pull_request) Failing after 20m - is it unrelated to the pr?


do not use Buffer in the browser

Remove the usage of Buffer in code executed by both NodeJS and browser.
This removes the need for polyfilling the Browser NodeJS built-in in the browser.

Hello @D4nte,
Thank you for the variety of fixes, that are really useful. A separate PR for each would have made merging easier 😅
The thing I am noticing though is that ./lib.mjs is [supposed to be replace in the browser]https://github.com/martinheidegger/dns-query/blob/82436fc8bb486d90832f87fa62aae04c5529f421/package.json#L22-L25). So I am wondering in which build tool this doesn't happen. As this seems to be the bigger issue?!

Hi @martinheidegger,
Thank you for the quick response. Will separate the typo fix in another PR.
Regarding the usage of Buffer. Indeed I misunderstood the codebase, I need to review. Thank you for pointing this out.

Thank you for the quick response. Will separate the typo fix in another PR.

Oh I see you already did that :+1:

Thank you for the quick response. Will separate the typo fix in another PR.

Oh, I already cherry-picked the commits, if you don't mind 😅

I think the buffer issue was a keyboard-chair one. Other changes have been extracted. Closing PR.

Thank you for the PR still, it is helpful to have response like this! The latest changes have been published as 0.11.2


Add Support for Hugo v0.55+

Hugo v0.55 has breaking changes that break some features and also warns of future deprecation of several variables. See https://github.com/gohugoio/hugo/releases/tag/v0.55.0
Therefore, refactor the code and require Hugo v0.55+ going forward.

[x] Refactor code based on Hugo v0.55 log messages:

WARN Page's .Hugo is deprecated and will be removed in a future release. Use the global hugo function.
WARN Page's .RSSLink is deprecated and will be removed in a future release. Use the Output Format's link, e.g. something like:
    {{ with .OutputFormats.Get "RSS" }}{{ . RelPermalink }}{{ end }}.
WARN Page's .Dir is deprecated and will be removed in a future release. Use .File.Dir.
WARN Page's .UniqueID is deprecated and will be removed in a future release. Use .File.UniqueID.
WARN Page's .URL is deprecated and will be removed in a future release. Use .Permalink or .RelPermalink. If what you want is the front matter URL value, use .Params.url.

 The following may need refactoring to work with Hugo v0.55+:

[ ] The Publication "Cite" feature is now broken

https://discourse.gohugo.io/t/v0-55-404-error-resource-in-bundle-is-found-but-link-is-invalid/18056

[x] Shortcodes no longer work properly > Behaviour has changed for Shortcodes using the {{% construct

"If you want the old behavior, you can put the following line in the start of your shortcode template: {{ $_hugo_config := { "version": 1 } }}"

[x] links to authors without profiles no longer works?
[x] preserveTaxonomyNamesconfiguration option is removed. Use .Page.Title.
[x] We no longer add XML headers to Output Formats that output XML (<?xml version="1.0" encoding="utf-8" standalone="yes" ?>). This header is moved to the templates. If you have custom RSS or sitemap templates you may want to add the XML declaration to these. Since they, by default, is handled by Go's HTML template package, you must do something like this to make sure it's preserved: {{ printf "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"yes\" ?>" | safeHTML }}

upgrade rss.xml, remove sitemap.xml to revert to Hugo's latest internal tpl

[x] The taxonomy nodes now have a .Page accessor which makes it much simpler to get a proper .Title etc.

upgrade tag_cloud.html

Awesome work @gcushen, super appreciated, this issue saved me quite a few of headaches today wondering why a fresh new site from scratch was not working.
Is there a way to know which Hugo version is actually supported on master? I'm getting this message when running<EMAIL_ADDRESS>(after reading the very helpful breaking changes page):

ERROR 2019/04/12 14:57:29 HUGO-ACADEMIC theme does not support Hugo version 0.49.2. Minimum version required is 0.53

But then when I downgrade to 0.53 via:
$ brew unlink hugo
$ brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/b285e5d4827f86ec6ad72bec2b881e4ce6fe931d/Formula/hugo.rb
$ brew list hugo
/usr/local/Cellar/hugo/0.53/bin/hugo
/usr/local/Cellar/hugo/0.53/etc/bash_completion.d/hugo.sh
/usr/local/Cellar/hugo/0.53/share/man/ (25 files)
$ hugo version
Hugo Static Site Generator v0.53/extended darwin/amd64 BuildDate: unknown

I'm getting this instead:

ERROR 2019/04/12 15:08:54 render of "page" failed: "/Users/romanvg/dev/umccr/site/themes/hugo-academic/layouts/_default/single.html:22:34": execute of template failed: template: _default/single.html:6:5: executing "_default/single.html" at <partial "page_header...>: error calling partial: "/Users/romanvg/dev/umccr/site/themes/hugo-academic/layouts/partials/page_header.html:22:34": execute of template failed: template: partials/page_header.html:92:7: executing "partials/page_header.html" at <partial "page_metada...>: error calling partial: "/Users/romanvg/dev/umccr/site/themes/hugo-academic/layouts/partials/page_metadata.html:22:34": execute of template failed: template: partials/page_metadata.html:22:34: executing "partials/page_metadata.html" at <$.Site.Params.date_f...>: invalid value; expected string

Thanks again in advance!
/cc @ohofmann @MrsLaviniaG

@brainstorm the official installation guide for installing Academic with RStudio can be found here: https://sourcethemes.com/academic/docs/install/#install-with-rstudio

Yes, @gcushen, I did follow that guide closely, I'll give it a second/third look and get back to you.

The official installation guide assumes installation from scratch... in this case, the missing directory config made all the difference, only noticed it when starting from scratch all over again :-S
Thanks!


Code example for PreferenceFragment

I cannot find the code example for PreferenceFragment.
I searched the code base, but cannot find it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: efc8dc11-2c71-4103-a860-a7864b8244b5
Version Independent ID: 5f1d659a-b57a-d7fe-c640-33f958f28cb3
Content: Specialized Fragment Classes - Xamarin
Content Source: docs/android/platform/fragments/specialized-fragment-classes.md
Product: xamarin
Technology: xamarin-android
GitHub Login: @conceptdev
Microsoft Alias: crdun

This sample code includes a reference to PreferenceFragment.


[Refactor] Derive PartialEq for ConstValue

AFAICT our manual implementation does exactly what the derive one would, except for one case - ConstValue::Opaque. But with a bit of "wizardry" (?!) we can get provide that ourselves and get Rust to derive the rest.
The extra level of tupling inside ConstValue::Opaque was because without that the derive(serde::Serialize) on ConstValue produced a complex error about moving out of a Box. A struct OpaqueValue(CustomType, Box<dyn CustomConst>) is one way to do that but just the tuple works too.

Apparently the error that prevents us from deriving PartialEq with the barebones box is a known bug :/
rust-lang/rust#78808
Here's a minimal case. Hopefully it will be fixed some day... playground


Feature/exporter integration enrico

All tests passed but the tls one seemed to be a bit flakey, as I had to re-run it


[test]Fix Flaky-test: BrokerServiceTest.testLookupThrottlingForClientByClient

Master Issue: https://github.com/apache/pulsar/issues/16521
Motivation

Fixes https://github.com/apache/pulsar/issues/16521

If the broker responds quickly enough, there may never be concurrency in requests, so we use a Semaphore to block the response.

Modifications
Fix UT  BrokerServiceTest.testLookupThrottlingForClientByClient
Verifying this change

[x] Make sure that the change passes the CI checks.

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

Close this PR because this flaky test has already been  fixed

left a comment:
I feel the origin log "At least one should fail" is better

Updated

@codelipenghui @Technoboy-  PTAL also~

/pulsarbot run-failure-checks


bug fix: fragments are duplicated and overlaped

Hi, professor.
In FragmentProgrammaticLayout, the main Activity would add fragment whenever onCreate is invoked. As a result, when onCreate is invoked by configuration changing, duplicated fragments would be added and overlapped.
Here are the screenshots.Before configuration change:

After configuration change(since new fragment is overlapped with old fragment, multiple items in the list appear to be selected):

Hi, Good Afternoon.
My name is Joel.
I don't understanding your fix completely.
If this was your solution I appreciate some lite explanation  about it.   ;)
I think the Issue is about Fragment lifecycle.
maybe you can find a more appropriate solution on that way.
Thanks for your time,
Joel Espinal (JoelEspinal)
Developer
here a part snipped code:
if(savedInstanceState != null) {

 	return;

 }

Hi, Joel. Thank you for reply.
Sorry for leading to confusing. Let me explain if(savedInstanceState != null) {     return; } .
When configuration(such as orientation) changes, android system would restart the app. Before restart, it would save certain information into savedInstanceState, and pass it as parameter to onCreate method. If savedInstanceState is not null, it means this is not a newly created app, but restarted app. Since in this example FragmentProgrammaticLayout , fragments are only needed to be added once when the app is newly created, they don't need to be added again when restarted. So, it should just directly return.
Here is the link from google android site, where it uses exactly the same technique.


make clean doesn't remove html/.html…

…Ideally, we shouldn't even be creating this file, but since we are, clean it up.
BTW, the title is
  <title>Disambiguation for ''</title>

in
doc/Language/5to6-nutshell.pod6

the line
=head3 X<C<constant>|constant>

is the first thing that causes an empty .html to be generated.
And it's this block of code from htmlify that extracts the empty name:
when :("", Pod::FormattingCode $, "") {

Why does it happen that way? Maybe because of the C?


(Codeforces)112A petya and string

Solved Question 112A of Codeforces in c++

Please Assign this issue to me, under LGM-SOC

can I solve this issue under hacktoberfest 21?

Hey, Can i contribute to this issue under hacktoberfest 2021?


Add org Animal Advocacy Africa

https://forum.effectivealtruism.org/posts/acBujAbWrpCGaC75g/introducing-animal-advocacy-africa?fbclid=IwAR28TRkwj_XUksGdpQZNfXTc3akA7D924p66PDMs3G0PcDAuZCdxaJ8gygs

The org is so new it's not yet available on Wayback Machine. Ready for review.

I archived the page at https://web.archive.org/web/20201228011953/https://www.animaladvocacyafrica.org/about so can you use the snapshot URL instead?
In the future in cases like this, you can go to https://web.archive.org/save/ or https://archive.is/ and make a snapshot.

For Brett Thompson, Calvin Solomon Onyango, and Catherine Jerotich Chumo, the title is incorrect. They are all advisors, and you can use "Advisor" in title and "advisor" in employment_type. The titles that you used, like "East Africa Representative", are actually the titles at their day jobs rather than at Animal Advocacy Africa.

Everything else looks good.

Fixed. Ready for review.


Not compatible with fabric loader 0.15.0

Thanks for the issue.
Fixed with 97b06ea and 1.0.3 release.


Incorrect license for React files

Type: Incorrect
Summary:
Incorrect license for React files
Details:
License for /src/main/resources/webapp/vendor/react/* was set incorrectly.
It should be BSD-2-Clause plus the explicit PATENTS grant.
Resolution:
PLEASE NOTE: SPDX does not actually appear to have an actual license identifier for the Facebook "BSD + PATENTS" licensing scenario. SPDX "BSD-2-Clause-Patents" is not the Facebook Grant of Additional Patent Rights Version 2 seen in older versions of React. But there is no other option to choose from.
Affected definitions:

presto ba3080f3f7da6e7174f5d18d8b8ad5cb2e75c8a0

@bduranc 	If the license isn't in the SPDX list, then shouldn't it be "NOASSERTION" per the curation guidelines?

@capfei: Good point. I will make those changes.

@bduranc, are you going to edit this PR, submit a new one, or I can update it for you? Let me know.

@capfei : It appears that the license for the React components are already updated, citing BSD 3 Clause+ "NOASSERTION".
We can close this PR as it appears the discrepancy has already been resolved.

Thanks.

Thanks! 😃


Too many computers

Too many computers used within the last 24 hours for the same Cursor account. Please try again later, use a different Cursor account, or contact us at<EMAIL_ADDRESS>if you believe this is a mistake. We have this limit in place to prevent abuse, but if you have a legitimate use case for using many computers, please do email us and we will unblock you.

fixed,restart cursor-vip


Q: How to properly mount fileshare with local(docker-desktop) kubernetes

Hi,
My projects involves azure fileshare integration.
It is working fine in AKS cluster. I would like to replicate the same in local environment for developers using docker-desktop with kubernetes enabled but fileshare is currently is not working because it is provisioned from azure.
I tried https://github.com/andyzhangx/demo/tree/master/linux/azurefile with static provisioning because I already have fileshare.
1.Created azure-secret with storage name and key.
2.Then edited nginx-pod-azurefile-static.yaml file with my sharename. This is throwing ERROR ".......bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program."
How would I link my local setup with azure fileshare for my pod deployment? I am feeling that I have to change something in mount options.
FYI: I am running docker-desktop on windows machine with WSL2 enabled

I think first you should make sure mount -t cifs ... works on your local env, it requires cifs-utils package

Yes, I am trying to mount using cifs-utils. Found really interesting article on that.
But problem is I need to install cifs-utils inside docker-dekstop kubernetes node (shell).
I tried to node shell using Lens  and getting below error.

Looks like it's root/sudo previleges issue. I am not sure what command I should use to node exec with privileges. Lens didn't provide how it's _exec_ing into the node.
So how would I exec into the node and install cifs-utils in docker-desktop kubernetes?

I am not familiar with docker-desktop kubernetes


Fixed _isRelevant function in calculation module returns incorrect ca…

…ched result after initialization
It's a bit of an obscure issue due to the fact that in most of the Enketo world form values are never cleared when they become non-relevant. However, there are some remnants in the code (e.g. the emptyNonRelevant parameters at various places in calculate.js), that conflict with the preInitRelevance cache in calculate.js. See inline code comment below.
I had to write a unit test to demonstrate it and decided to do this at the lowest level since _isRelevant should probably always return a correct result no matter when it is called (so the test would survive any possible future removal of these emptyNonRelevant remnants). Though if you prefer I write a test for the update method, just let me know.

One thing we haven't completed after the monorepo migration is hooking up benchmarks. We should do a benchmark sanity check before release.

Thank you!

One thing we haven't completed after the monorepo migration is hooking up benchmarks. We should do a benchmark sanity check before release.

Ah, yes!


Invalid key in GuildModel

Hello, thank you for new update! It generally works, thank you for your work, but I found an exception when receiving IDiscordServerEvents OnServerJoined in OwnerId. The key is missing in dictionary.
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) [0x0001e] in :0
at DiscordUnity.State.DiscordServer..ctor (DiscordUnity.Models.GuildModel model) [0x0010f] in D:\Documents\Documents\DiscordUnity\src\DiscordUnity\State\DiscordServer.cs:69
at DiscordUnity.DiscordAPI+<>c__DisplayClass33_5.b__6 () [0x00000] in D:\Documents\Documents\DiscordUnity\src\DiscordUnity\DiscordAPI.cs:455
at DiscordUnity.DiscordAPI.Update () [0x00003] in D:\Documents\Documents\DiscordUnity\src\DiscordUnity\DiscordAPI.cs:168
UnityEngine.Debug:LogError(Object)
DiscordLogger:LogError(String, Exception) (at Assets/Plugins/DiscordUnity/DiscordManager.cs:233)
DiscordUnity.DiscordAPI:Update() (at D:/Documents/Documents/DiscordUnity/src/DiscordUnity/DiscordAPI.cs:173)
DiscordManager:Update() (at Assets/Plugins/DiscordUnity/DiscordManager.cs:66)
Payload was:
Received Message: {"t":"GUILD_CREATE","s":2,"op":0,"d":{"description":null,"preferred_locale":"en-US","joined_at":"2020-10-29T08:27:26.921569+00:00","unavailable":false,"roles":[{"position":0,"permissions_new":"104320577","permissions":104320577,"name":"@everyone","mentionable":false,"managed":false,"id":"768361295074033684","hoist":false,"color":0},{"tags":{"bot_id":"771280068290150411"},"position":1,"permissions_new":"68608","permissions":68608,"name":"UnityDevel","mentionable":false,"managed":true,"id":"771289081480871937","hoist":false,"color":0}],"channels":[{"type":4,"position":0,"permission_overwrites":[],"name":"Text Channels","id":"768361295715106816"},{"type":4,"position":0,"permission_overwrites":[],"name":"Voice Channels","id":"768361295715106817"},{"type":0,"topic":null,"rate_limit_per_user":0,"position":0,"permission_overwrites":[],"parent_id":"768361295715106816","name":"general","last_message_id":"771316716797952000","id":"768361295715106818"},{"user_limit":0,"type":2,"position":0,"permission_overwrites":[],"parent_id":"768361295715106817","name":"General","id":"768361295715106820","bitrate":64000}],"discovery_splash":null,"region":"europe","large":false,"rules_channel_id":null,"default_message_notifications":0,"system_channel_flags":0,"features":[],"emojis":[],"voice_states":[],"premium_tier":0,"public_updates_channel_id":null,"name":"UnityDevel","banner":null,"vanity_url_code":null,"explicit_content_filter":0,"premium_subscription_count":0,"presences":[],"verification_level":0,"max_members":100000,"icon":"7cc539173c3afd750c8464402238e292","lazy":true,"member_count":4,"members":[{"user":{"username":"UnityDevel","id":"771280068290150411","discriminator":"9857","bot":true,"avatar":null},"roles":["771289081480871937"],"premium_since":null,"nick":null,"mute":false,"joined_at":"2020-10-29T08:27:26.921569+00:00","is_pending":false,"hoisted_role":null,"deaf":false}],"application_id":null,"afk_channel_id":null,"splash":null,"afk_timeout":300,"max_video_channel_users":25,"system_channel_id":"768361295715106818","mfa_level":0,"owner_id":"573433006497202186","id":"768361295074033684","guild_hashes":{"version":1,"roles":{"omitted":false,"hash":"KcLAJ8AvHzw"},"metadata":{"omitted":false,"hash":"4/wOK7YPbUk"},"channels":{"omitted":false,"hash":"jMzOZis66jM"}}}}
UnityEngine.Debug:Log(Object)
DiscordLogger:Log(String) (at Assets/Plugins/DiscordUnity/DiscordManager.cs:217)
DiscordUnity.d__33:MoveNext() (at D:/Documents/Documents/DiscordUnity/src/DiscordUnity/DiscordAPI.cs:282)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start(d__33&)
DiscordUnity.DiscordAPI:OnSocketMessage(String)
DiscordUnity.d__30:MoveNext() (at D:/Documents/Documents/DiscordUnity/src/DiscordUnity/DiscordAPI.cs:216)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

Hmm, it is maybe in Members not in GuildModel.

And they are not. Server do not return meber list so id is not in dictionary.


All instances of a GLTF-inherited scene share state

Tested versions
4.2.1-stable
System information
Windows 11, Compatibility
Issue description
All instances of a GLTF-inherited scene share the same state.
Steps to reproduce
Export a 3d model, rigged and animated as a glb from Blender.
Create a new inherited scene based on the imported glb file.
Add an AnimationTree and an AnimationNodeAnimation as the root animation node.
Attach a script that plays different animation based on player input, using logic to ensure only one instance should be playing at a time (an exported bool to toggle, for example).
Add debug prints to verify.
(See and run attached minimal reproduced project.)
Minimal reproduction project (MRP)
sumo_minimal_gltf.zip

I think just check Local to Scene on the AnimationTree and it will work.

@narcissawright Thanks! It worked. Glad this was user error. I could have sworn I had tried before and it didn't work, but oh well, working now. Going to mark closed.


Commenting and messaging

These would be nice-to-haves:
a) commenting on assignments and submissions
b) a global feed of all the comments
c) direct messaging other users
d) chat

See:
https://medium.com/swlh/building-a-chat-app-on-react-and-aws-amplify-1-c96d456dffc1


Dev: Ignore redundant files to propagate to npm

This PR adds redundant files to .npmignore. Also, adds package to .gitignore.

This change is

Review status: 0 of 2 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Add throughput1 client library

This PR adds the client library for MSAK's throughput1 protocol. It's meant to be a (fully functional) MVP, with more nice-to-have features to be added in subsequent PRs. To test it:
# defaults: 2 BBR streams, 3s duration, no delay, plaintext, production Locate instance and servers
go run ./cmd/msak-client

Things not included:

A flag for multi-server testing (to run each stream against a different server)
Aggregate throughput in HumanReadable's output -- unclear how it should be computed, since each stream intentionally excludes the TLS and WS handshakes, while it would be more correct to include them in this case. I'm considering to add a second network byte counter that includes that overhead, but that will involve a design update and a separate PR.

This change is

@nkinkade I appreciate you taking the time to test and review the code. Thanks!
I'm going to add a second reviewer on this PR, but I'm still interested in any comments or ideas you might have. I've found that engaging in code reviews also improves one's own understanding of the language and code design, and any effort to increase code quality is always welcome.
Generally speaking, something that helped me a lot when approaching Go code reviews are these
resources:

Effective Go - "tips for writing clear, idiomatic Go code"
Go code review comments - a collection of good practices and things to look for in code reviews
Test comments - like the above, but focused on how to write good tests

If you have time, I'd love getting a second eye on how I'm following these conventions!
(For example, one thing I noticed while looking at my code again after a day is that I forgot to add docstrings on some exported identifiers - e.g. the Locator interface, or pretty much every field in Throughput1Client :sweat_smile: )

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

31 of 213 (14.55%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-12.6%) to 71.073%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/client/emitter.go
0
27
0.0%

pkg/client/client.go
31
186
16.67%

Totals

Change from base Build<PHONE_NUMBER>:
-12.6%

Covered Lines:
828

Relevant Lines:
1165

💛  - Coveralls


Fix duplicated products

Fix problem with products showing up in multiple categories, if they are of the same name.
Original PR: #1154
Pull Request Checklist

[ ] Privileged views and APIs are guarded by proper permission checks.
[ ] All visible strings are translated with proper context.
[ ] All data-formatting is locale-aware (dates, numbers, and so on).
[ ] Database queries are optimized and the number of queries is constant.
[ ] The changes are tested.
[ ] The code is documented (docstrings, project documentation).
[ ] Python code quality checks pass: pycodestyle, pydocstyle, pylint.
[ ] JavaScript code quality checks pass: eslint.

Codecov Report

Merging #1331 into master will decrease coverage by 1.17%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #1331      +/-   ##
=========================================
- Coverage   70.37%   69.2%   -1.18%
=========================================
  Files         117     114       -3
  Lines        6158    6212      +54
  Branches      786     754      -32
=========================================
- Hits         4334    4299      -35
- Misses       1661    1744      +83
- Partials      163     169       +6

Impacted Files
Coverage Δ

saleor/graphql/api.py
81.92% <100%> (ø)
:arrow_up:

saleor/dashboard/product/__init__.py
42.85% <0%> (-57.15%)
:arrow_down:

saleor/dashboard/product/widgets.py
75% <0%> (-19.12%)
:arrow_down:

saleor/core/sitemaps.py
83.33% <0%> (-16.67%)
:arrow_down:

saleor/registration/views.py
91.66% <0%> (-5.96%)
:arrow_down:

saleor/site/utils.py
95% <0%> (-5%)
:arrow_down:

saleor/dashboard/product/views.py
61.56% <0%> (-3.85%)
:arrow_down:

saleor/dashboard/order/views.py
51.55% <0%> (-3.16%)
:arrow_down:

saleor/core/middleware.py
82.85% <0%> (-2.51%)
:arrow_down:

saleor/order/models.py
87.45% <0%> (-2.4%)
:arrow_down:

... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c11988...3360e8c. Read the comment docs.


Problem: K8s deployment with Access Token Authorization expects Access Token for GET calls

Description
K8s deployment with secret-access-token based authorization instead of three-scale should expect Secret-Access-Token header to be set only for POST calls. But currently it expects the same header for GET calls as well.

@muawiakh fixed

Sigh, I don't understand NGINX, I mean could they be any more arcane? But if this works then good.

Maybe you could add some comments for future readers to understand what is happening, maybe with links to relevant documentation online.

@muawiakh for three scale there is a separate conf file nginx.conf.threescale.template this file will only we used if you have set AUTHORIZATION_MODE  to secrete-token

closing this in favor of https://github.com/bigchaindb/bigchaindb/pull/2134#pullrequestreview-103934212


regen

Does it need another update as I'm seeing this error unmarshalling a dialogflow request
"Couldn't Unmarshal request to jsonpb" error="unknown field \"version\" in dialogflow.OriginalDetectIntentRequest"

My bad, found this https://github.com/google/go-genproto/issues/74


Adding EasyDel Library

EasyDeL is a well-documented source library to make your training faster and more Optimized With cool Options for training and serving and supports a wide range of models that have never been supported in JAX/Flax before like

Falcon
MPT
PHI
PALM
MISTRAL
GPT NEOX

and many more with options to fine-tune, pre-train, RLHF, and other cool features💫

Thanks for the PR--seems like a lot has changed in this diff due to formatting, could you revert all the changes except your addition?


Missing IOTMs causing logic errors

[ I do not have camel which would otherwise be used here] Desert exploration will slot compass into the offhand and put kramco on lefty to burn delay. However this causes the creation of sausages to fail as it is expecting to find the grinder equipped but cannot access it to make a sausage.

EDIT: Defiled nook is not even attempted without camel, there is a ready: () => get("camelSpit") >= 100, that prevents it from running.

Thanks for the reports! These should be fixed now, although you may have to increase the limit on Defiled Nook without a camel to spit.

I had tried that fix for kramco, unfortunately mafia still cant find it half the time. Should i just try to disable sausage diet for desert locally?


Added next raw to community plugins.

Hey! We're deprecating this repository hence why this PR hasn't been merged. I just updated the readme. You can try adding it to the community awesome list: https://github.com/unicodeveloper/awesome-nextjs.


React Capstone Project, Capstone #3

This PR contains all the steps made to build the React Capstone Project, including:

Selection of Financial modeling prep API to get stock data.
When the page loads, the webapp shows the list of markets that could be filtered out by company name or symbol.
Display some numeric values that come from the API.
When the user clicks (or taps) on a market or stock, the application navigates to the details page.
In the details page, the webapp retrieves data from the API to show detailed data in the category.
When the user clicks on the "Back" button (<), the user navigates to the home page.
Creation of unit tests for pure functions in the Redux code.
Creation of integration tests using the React Testing Library.
The project is a single-page application (SPA) built with React and Redux.
The data retrieved from the API is stored in the Redux store.
You should filter the data that you retrieve from the API using a Filter stateless component.
Every page (the main page and the pages for each item) has a unique route within the SPA.

@jmonto55 would add the video presenation, to continue the review


add multi-proc in to_json

Closes #2663. I've tried adding multiprocessing in to_json. Here's some benchmarking I did to compare the timings of current version (say v1) and multi-proc version (say v2). I did this with cpu_count 4 (2015 Macbook Air)

Dataset name: ascent_kb - 8.9M samples (all samples were used, reporting this for a single run)
v1- ~225 seconds for converting whole dataset to json
v2- ~200 seconds for converting whole dataset to json

Dataset name: lama - 1.3M samples (all samples were used, reporting this for 2 runs)
v1- ~26 seconds for converting whole dataset to json
v2- ~23.6 seconds for converting whole dataset to json

I think it's safe to say that v2 is 10% faster as compared to v1. Timings may improve further with better configuration.
The only bottleneck I feel is writing to file from the output list. If we can improve that aspect then timings may improve further.
Let me know if any changes/improvements can be done in this @stas00, @lhoestq, @albertvillanova. @lhoestq even suggested to extend this work with other export methods as well like csv or parquet.

Thank you for working on this, @bhavitvyamalik
10% is not solving the issue, we want 5-10x faster.
So let's test it on an instance with many more cores, I can test with 12 on my deb box and 40 on JZ.
Could you please share the test I could run with both versions?
Should we also test the sharded version I shared in https://github.com/huggingface/datasets/issues/2663#issue-946552273 to 3 versions to test.

Since I was facing OSError: [Errno 12] Cannot allocate memory in CircleCI tests, I've added num_proc option instead of always using full cpu_count. You can test both v1 and v2 through this branch (some redundancy needs to be removed). I did benchmarking like this:
import timeit

# mysetup to be executed only once

mysetup = '''
import datasets
from datasets import load_dataset
dataset = load_dataset("<dataset_name>")
'''

# code snippet whose execution time is to be measured
mycode = '''
dataset['train'].to_json(SAVE_LOCATION, num_proc=4) # if num_proc 1 or None then it'll fall back to v1
'''

# timeit statement
print(timeit.timeit(setup = mysetup,
                     stmt = mycode,
                     number = 2))

Here are the benchmarks with the current branch for both v1 and v2 (dataset: ascent_kb, 8.9M samples):

batch_size
time (in sec)
time (in sec)

num_proc = 1
num_proc = 4

10k
185.56
170.11

50k
175.79
86.84

100k
191.09
78.35

125k
198.28
90.89

Increasing the batch size on my machine helped in making v2 around 50% faster as compared to v1. Timings may vary depending on the machine. I'm including the benchmarking script as well.
import time
from datasets import load_dataset
import pathlib
import os
from pathlib import Path
import shutil
import gc

batch_sizes = [10_000, 50_000, 100_000, 125_000]
num_procs = [1, 4]  # change this according to your machine

SAVE_LOC = "./new_dataset.json"

for batch in batch_sizes:
    for num in num_procs:
        dataset = load_dataset("ascent_kb")

        local_start = time.time()
        ans = dataset['train'].to_json(SAVE_LOC, batch_size=batch, num_proc=num)
        local_end = time.time() - local_start

        print(f"Time taken on {num} num_proc and {batch} batch_size: ", local_end)

        # remove that dataset and its contents from cache and newly generated json
        new_json = pathlib.Path(SAVE_LOC)
        new_json.unlink()

        try:
            shutil.rmtree(os.path.join(str(Path.home()), ".cache", "huggingface"))
        except OSError as e:
            print("Error: %s - %s." % (e.filename, e.strerror))

        gc.collect()

This will download the dataset in every iteration and run to_json. I didn't do multiple iterations here for to_json (for a specific batch_size and num_proc) and took average time as I found that v1 got faster after 1st iteration (maybe it's caching somewhere). Since you'll be doing this operation only once, I thought it'll be better to report how both v1 and v2 performed in single iteration only.
Important: Benchmarking script will delete the newly generated json and ~/.cache/huggingface/ after every iteration so that it doesn't end up using any cached data (to be a on safe side)

Thank you for sharing the benchmark, @bhavitvyamalik
That's a dangerous benchmark as it'd wipe out many other HF things. Why not wipe out:
~/.cache/huggingface/datasets/ascent_kb/

Running the benchmark now.

Weird, I tried to adapt your benchmark to using shards and the program no longer works. It instead quickly uses up all available RAM and hangs. Has something changed recently in datasets? You can try:
import time
from datasets import load_dataset
import pathlib
import os
from pathlib import Path
import shutil
import gc
from multiprocessing import cpu_count, Process, Queue

batch_sizes = [10_000, 50_000, 100_000, 125_000]
num_procs = [1, 8]  # change this according to your machine

DATASET_NAME = ("ascent_kb")
num_shards = [1, 8]
for batch in batch_sizes:
    for shards in num_shards:
        dataset = load_dataset(DATASET_NAME)["train"]
        #print(dataset)

        def process_shard(idx):
            print(f"Sharding {idx}")
            ds_shard = dataset.shard(shards, idx, contiguous=True)
            # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
            print(f"Saving {DATASET_NAME}-{idx}.jsonl")
            ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

        local_start = time.time()
        queue = Queue()
        processes = [Process(target=process_shard, args=(idx,)) for idx in range(shards)]
        for p in processes:
            p.start()

        for p in processes:
            p.join()
        local_end = time.time() - local_start

        print(f"Time taken on {shards} shards and {batch} batch_size: ", local_end)

Just careful, so that it won't crash your compute environment. As it almost crashed mine.

So this part seems to no longer work:
        dataset = load_dataset("ascent_kb")["train"]
        ds_shard = dataset.shard(1, 0, contiguous=True)
        ds_shard.to_json(f"{DATASET_NAME}-{0}.jsonl", orient="records", lines=True, force_ascii=False)

If you are using to_json without any num_procor num_proc=1 then essentially it'll fall back to v1 only and I've kept it as it is (the tests were passing as well)

That's a dangerous benchmark as it'd wipe out many other HF things. Why not wipe out:

That's because some dataset related files were still left inside ~/.cache/huggingface/datasets folder. You can wipe off datasets folder inside your cache maybe

dataset = load_dataset("ascent_kb")["train"]
ds_shard = dataset.shard(1, 0, contiguous=True)
ds_shard.to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)

I tried this lama dataset (1.3M) and it worked fine. Trying it with ascent_kb currently, will update it here.

I don't think the issue has anything to do with your work, @bhavitvyamalik. I forgot to mention I tested to see the same problem with the latest datasets release.
Interesting, I tried your suggestion. This:
python -c 'import datasets; dataset = datasets.load_dataset("lama")["train"]; dataset.shard(1, 0, contiguous=True).to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)'

works fine and takes just a few GBs to complete.
this on the other hand blows up memory-wise:
python -c 'import datasets; dataset = datasets.load_dataset("ascent_kb")["train"]; dataset.shard(1, 0, contiguous=True).to_json("ascent_kb-0.jsonl", orient="records", lines=True, force_ascii=False)'

That's because some dataset related files were still left inside ~/.cache/huggingface/datasets folder. You can wipe off datasets folder inside your cache maybe

I think recent datasets added a method that will print out the path for all the different components for a given dataset, I can't recall the name though. It was when we were discussing a janitor program to clear up space selectively.

and I have to kill it before it uses up all RAM. (I have 128GB of it, so it should be more than enough)

Same thing just happened on my machine too. Memory leak somewhere maybe? Even if you were to load this dataset in your memory it shouldn't take more than 4GB. You were earlier doing this for oscar dataset. Is it working fine for that?

Hmm, looks like datasets has changed and won't accept my currently cached oscar-en (crashes), so I'd rather not download 0.5TB again.
Were you able to reproduce the memory blow up with ascent_kb? It's should be a much quicker task to verify.
But yes, oscar worked just fine with .shard() which is what I used to process it fast.

What I tried is:
HF_DATASETS_OFFLINE=1 HF_DATASETS_CACHE=cache python -c 'import datasets; ds="oscar"; dataset = datasets.load_dataset(ds, "unshuffled_deduplicated_en")["train"]; \
dataset.shard(1000000, 0, contiguous=True).to_json(f"{ds}-0.jsonl", orient="records", lines=True, force_ascii=False)'

and got:
Using the latest cached version of the module from /gpfswork/rech/six/commun/modules/datasets_modules/datasets/oscar/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d (last modified on Fri Aug  6 01:52:35 2021) since it couldn't be found locally at oscar/oscar.py or remotely (OfflineModeIsEnabled).
Reusing dataset oscar (cache/oscar/unshuffled_deduplicated_en/1.0.0/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/load.py", line 755, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 737, in as_dataset
    datasets = utils.map_nested(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 203, in map_nested
    mapped = [
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 764, in _build_single_dataset
    ds = self._as_dataset(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/builder.py", line 834, in _as_dataset
    dataset_kwargs = ArrowReader(self._cache_dir, self.info).read(
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 217, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 238, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 173, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 308, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/arrow_reader.py", line 327, in read_table
    return table_cls.from_file(filename)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/table.py", line 450, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/gpfswork/rech/six/commun/conda/stas/lib/python3.8/site-packages/datasets/table.py", line 43, in _memory_mapped_arrow_table_from_file
    memory_mapped_stream = pa.memory_map(filename)
  File "pyarrow/io.pxi", line 782, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 743, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory

Were you able to reproduce the memory blow up with ascent_kb? It's should be a much quicker task to verify.

Yes, this blows up memory-wise on my machine too.
I found that a similar error was posted on the forum on 5th March. Since you already knew how much time #2663 comment took, can you try benchmarking v1 and v2 for now maybe until we have a fix for this memory blow up?

OK, so I benchmarked using "lama" though it's too small for this kind of test, since the sharding is much slower than one thread here.
Results: https://gist.github.com/stas00/dc1597a1e245c5915cfeefa0eee6902c
So sharding does really bad there, and your json over procs is doing great!
Any suggestions to a somewhat bigger dataset, but not too big? say 10 times of lama?

Looks great! I had a few questions/suggestions related to benchmark-datasets-to_json.py:

You have used only 10_000 and 100_000 batch size. Including more batch sizes may help you find the perfect batch size for your machine.
Why have you used num_procs 1 and 4 only?

You can use:

dataset = load_dataset("cc100", lang="af"). Even though it has only 2 fields but there are around 9.9 mil samples. (lama had around 1.3 mil samples)
dataset = load_dataset("cc100", lang="eu") -> 16 mil samples. (if you want something more than 9.9 mil)
dataset = load_dataset("neural_code_search", 'search_corpus') -> 4.7 mil samples

Thank you, @bhavitvyamalik
My apologies, at the moment I have not found time to do more benchmark with the proposed other datasets. I will try to do it later, but I don't want it to hold your PR, it's definitely a great improvement based on the benchmarks I did run! And the comparison to sharded is really just of interest to me to see if it's on par or slower.
So if other reviewers are happy, this definitely looks like a great improvement to me and addresses the request I made in the first place.

Why have you used num_procs 1 and 4 only?

Oh, no particular reason, I was just comparing to 4 shards on my desktop. Typically it's sufficient to go from 1 to 2-4 to see whether the distributed approach is faster or not. Once hit larger numbers you often run into bottlenecks like IO, and then numbers can be less representative. I hope it makes sense.

Tested it with a larger dataset (srwac) and memory utilisation remained constant with no swap memory used. @lhoestq should I also add test for the same? Last time I tried this, I got OSError: [Errno 12] Cannot allocate memory in CircleCI tests


Add SU Clear Times

Able to see SU Clear Times would be very useful

The data of the su needs to be pulled through an additional api, which may be more complex. Perhaps it will be added after the reconstruction is completed.


Use as well the default meteor loginWithPassword

Hi, thanks for the great works you have done making this package.
I'd like to know if is possible to use the default Meteor.loginWithPassword in conjunction while using this package whitin a project. Actually when I try to create two different login sections, one using two-factor and one using the default Meteor login, I get a login forbidden error using Meteor.loginWithPassword.

Hey thanks Emanuele, what you're looking for is the validateLoginAttempt method: https://github.com/dburles/meteor-two-factor#validateloginattempt-optional. See here for an example https://github.com/dburles/meteor-two-factor#usage-server

It works as expected. Thank you for your help!


VS Code keeps crashing

Type: Bug
I have been having an issue with VS Code where it keeps crashing, but it's not always doing the same operation and sometimes the same operation works well, and some times it crashes. I am not sure how to troubleshoot.
I was asked to create this issue the next time it crashed, and it did... https://github.com/microsoft/vscode/issues/169502#issuecomment-1418471747
Please see the Crash Dump in the next comment.
VS Code version: Code 1.75.0 (Universal) (e2816fe719a4026ffa1ee0189dc89bdfdbafb164, 2023-02-01T15:24:42.903Z)
OS version: Darwin x64 22.2.0
Modes:
Sandboxed: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz (16 x 2400)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
6, 7, 7

Memory (System)
32.00GB (0.37GB free)

Process Argv
--crash-reporter-id f7e547af-40ff-4634-94bc-4880ea4ad472

Screen Reader
yes

VM
0%

Extensions (19)

Extension
Author (truncated)
Version

apex-pmd
chu
0.5.9

vscode-eslint
dba
2.4.0

gitlens
eam
13.2.0

prettier-vscode
esb
9.10.4

lana
Fin
1.5.2

git-graph
mhu
1.30.0

vscode-xml
red
0.24.0

salesforce-vscode-slds
sal
1.4.8

salesforcedx-vscode
sal
56.16.0

salesforcedx-vscode-apex
sal
56.16.0

salesforcedx-vscode-apex-debugger
sal
56.16.0

salesforcedx-vscode-apex-replay-debugger
sal
56.16.0

salesforcedx-vscode-core
sal
56.16.0

salesforcedx-vscode-expanded
sal
56.16.0

salesforcedx-vscode-lightning
sal
56.16.0

salesforcedx-vscode-lwc
sal
56.16.0

salesforcedx-vscode-soql
sal
56.16.0

salesforcedx-vscode-visualforce
sal
56.16.0

sfdx-package-xml-generator
Vig
2.0.9

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscoreces:30445986
pythondataviewer:30285071
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
azure-dev_surveyone:30548225
vsccc:30610678
pyindex848cf:30577861
nodejswelcome1:30587005
2e4cg342:30602488
pyind779cf:30657577
89544117:30613380
pythonsymbol12:30657548

Here is the crash dump...
2023-02-06.zip

I keep getting crashes on my VS Code, I am not sure how to fix. Please help. There are a few more .dmp files. It's crashing a few times per hour...
20230316T1113AM 9b9fc4be-2dd2-435f-a7cc-44b59be21339.dmp.zip

20230316T1116AM b1eabdb0-0044-4f39-9581-51fb1d3bbd90.dmp.zip

20230316T1137AM 65dfd1d1-cb90-426b-b5b4-b662607e580c.dmp.zip

20230317T1031AM 23f52353-e356-4654-bc96-705b309e30bd.dmp.zip

@deepak1556
Today, I started getting the error from #169502 again.
I am on 1.77.3 (Universal) 704ed70d4fd1c6bd6342c436f1ede30d1cff4710

Is this issue present with latest insiders https://code.visualstudio.com/insiders ?

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!