Benchmarks

Benchmarks
This PR introduces functionality to measure benchmarking for memory, Word Error Rate (WER), and speed in Faster-whisper.
1. Memory
GPU
Use Python thread with the py3nvml module to monitor GPU memory. This approach will continuously track GPU memory consumption at set intervals, then return the maximum memory used for the inference function.
RAM
Use memory_profiler module to measure maximum increase of memory usage.
2. WER
Evaluate the Faster-whisper model on the LibriSpeech validation-clean dataset with streaming mode, meaning no audio data has to be downloaded to your local device.
Use jiwer and evaluate modules from HF to calculate WER.
3. Speed
Calculate the min of args.repeat x time-averaged over 10 inference function calls.
Feel free to edit/update more benchmarking methods !

I'd like to suggest using the psutil library directly instead of memory_profiler, which inherently uses psutil under the hood and hasn't been updated in years.  Also, I'd suggest using the official "pynvml" library, which is actually named nvidia-ml-py; it's more up-to-date.
https://pypi.org/project/nvidia-ml-py/
Here's an example of how I'm using it:
https://github.com/BBC-Esq/VectorDB-Plugin-for-LM-Studio/blob/main/src/metrics_bar.py

@BBC-Esq , tks for your suggestion. But I think your proposed pynvml module seems not official, it is built from this repo. I tried both (py3nvml + nvidia-ml-py) and they gave the same results.
I used py3nvml like in transformer repo from HF.

This screenshot says that it's maintained by nvidia so I'm not sure what the confusion is...

"py3nvml" on pypi says that some individual maintains it...

If it works, it works...but when I went to the repo for py3nvml the last time it was updated was awhile ago...
Any feedback on psutil versus memory_profiler.  All the same perhaps?

As an aside, it's my understanding that "nvidia-ml-py" (i.e. the one I'm using) imports as "pynvml" NOT nvidia-ml-py, which is what confused me initially.  Multiple forks exist, some that have created pypi packages named similarly so...I'm about 85%-95% sure I have the official one.  ;-)

Don't know if it's helpful, but here's a script of mine that collects more than just GPU VRAM usage...might be helpful for benchmarking.  I've found that gpu/cuda usage is a useful metric as well as power usage...
https://github.com/BBC-Esq/Nvidia_Gpu_Monitor/blob/main/metrics_pynvml.py

If Ctranslate2 ever supports RocM, that'd be great, and you want to benchmark on AMD gpus I've struggled unsuccessfully to do it.  I don't own an AMD gpu, almost bought one solely to program, but anyways here's where my research left off...
https://github.com/BBC-Esq/VectorDB-Plugin-for-LM-Studio/issues/130

This screenshot says that it's maintained by nvidia so I'm not sure what the confusion is...

NVIDIA Corporation can be setup in the author field of the setup.py file.

Don't know if it's helpful, but here's a script of mine that collects more than just GPU VRAM usage...might be helpful for benchmarking

Tks, I will look at this.


Using with SDWebImage

I have a list of images which was cached with SDWebImage. I turn off the internet, open photo, but AXPhotoViewer does not display that. Do I need some extra integration ? I installed pod 'AXPhotoViewer/SDWebImage'

sorry, it works


Hold Arrow_key to navigate through levels quicker

dk, let's try fixing it that everytime you want to go to a certain you don't need to hit and wait everytime to get there

This was implemented in dsn_levels v.0.3.2+

I was also thinking when we get to 50+ levels, the amount of levels passed for 1 animation jump would increment by 10 instead. During this quicker jump the animation could completely be removed to give it the illusion of going quickly.
An alternative solution is to group many levels into one set that would be selected. This would make the levels seem larger and make selecting levels easier.
What are your thoughts? Feel free to disagree.


fix: model should not inject if exists

Close #152

Coverage increased (+0.2%) to 94.382% when pulling caaaa9740d738d6032e2ba05d11800db86d91858 on fix-subscription-multiple into 58c27b6244155efa5bb07befb8380e0284e8aa96 on master.

Coverage increased (+0.2%) to 94.382% when pulling caaaa9740d738d6032e2ba05d11800db86d91858 on fix-subscription-multiple into 58c27b6244155efa5bb07befb8380e0284e8aa96 on master.

Coverage increased (+0.2%) to 94.382% when pulling caaaa9740d738d6032e2ba05d11800db86d91858 on fix-subscription-multiple into 58c27b6244155efa5bb07befb8380e0284e8aa96 on master.


fix:temporal fix for issue #209 in verida-js

This is a temporal fix for issue https://github.com/verida/verida-js/issues/209
Demo
https://www.loom.com/share/695c9e1f0e3245c1b2a2639a2bbb8ed2

@cmcWebCode40 there is a merge error on yarn.lock
Also it's a "temporary" fix, not temporal (please fix your comment in the code) :)


Use lua version of nmap, nnoremap, vnoremap, ...

Define the keymap functions (nnoremap, nmap, vnoremap, ...) and use that throughout the config. This will also allow me to pass in pure lua functions similar to that of command and autocmd.
Reference:

https://github.com/tjdevries/astronauta.nvim
https://github.com/akinsho/dotfiles/blob/main/.config/nvim/lua/as/globals.lua

Closed by 5bea927374ca7e085b92926db224d857aeb8db9a


testに失敗する

Jpmobile::Mail delivering delivers through SMTP
Failure/Error: expect {
expected no Exception, got #<Errno::ECONNREFUSED: Connection refused - connect(2)> with backtrace:
# ./vender/bundle/ruby/2.0.0/gems/mail-2.6.3/lib/mail/network/delivery_methods/smtp.rb:112:in deliver!'       # ./vender/bundle/ruby/2.0.0/gems/mail-2.6.3/lib/mail/message.rb:2141:in do_delivery'
# ./vender/bundle/ruby/2.0.0/gems/mail-2.6.3/lib/mail/message.rb:238:in deliver'       # ./spec/unit/mail_spec.rb:347:in block (4 levels) in <top (required)>'
# ./spec/unit/mail_spec.rb:346:in `block (3 levels) in <top (required)>'
./spec/unit/mail_spec.rb:346:in `block (3 levels) in <top (required)>'

Finished in 0.75642 seconds (files took 0.35785 seconds to load)
319 examples, 1 failure
Failed examples:
rspec ./spec/unit/mail_spec.rb:344 # Jpmobile::Mail delivering delivers through SMTP

テストを実行するときにメール転送エージェントが起動していないと落ちる。
1) Jpmobile::Mail delivering delivers through SMTP
Failure/Error: expect {
expected no Exception, got # with backtrace:
# ./vender/bundle/ruby/2.0.0/gems/mail-2.6.3/lib/mail/network/delivery_methods/smtp.rb:112:in deliver!' # ./vender/bundle/ruby/2.0.0/gems/mail-2.6.3/lib/mail/message.rb:2141:indo_delivery'
# ./vender/bundle/ruby/2.0.0/gems/mail-2.6.3/lib/mail/message.rb:238:in deliver' # ./spec/unit/mail_spec.rb:347:inblock (4 levels) in '
# ./spec/unit/mail_spec.rb:346:in block (3 levels) in <top (required)>' # ./spec/unit/mail_spec.rb:346:inblock (3 levels) in '

Finished in 0.75642 seconds (files took 0.35785 seconds to load)
319 examples, 1 failure

Failed examples:

rspec ./spec/unit/mail_spec.rb:344 # Jpmobile::Mail delivering delivers through SMTP


update to 1.6.12 breaks admin

On WSL (Windows 10-1903 / Ubuntu 18.04.3 LTS) after update to GRAV 1.6.12 & Admin 1.9.8 I get "Fetch Failed:" on all Admin pages and in the console:
PHP Warning:  require(/index.php): failed to open stream: No such file or directory in /path-to-grav/system/router.php on line 38
PHP Fatal error:  require(): Failed opening required '/index.php' (include_path='.:/usr/share/php') in /path-to-grav/system/router.php on line 38
All works fine when I revert the changes to system/router.php, so it should be related to the new GRAV_BASEDIR system variable.

Hmm.. what if you change line 23 in system/router.php to this:
$grav_basedir = getenv('GRAV_BASEDIR');

Are you setting  GRAV_BASEDIR ?

Actually think i found the bug, please try this for router.php:
<?php

/**
 * @package    Grav\Core
 *
 * @copyright  Copyright (C) 2015 - 2019 Trilby Media, LLC. All rights reserved.
 * @license    MIT License; see LICENSE file for details.
 */

if (PHP_SAPI !== 'cli-server') {
    die('This script cannot be run from browser. Run it from a CLI.');
}

$_SERVER['PHP_CLI_ROUTER'] = true;

if (is_file($_SERVER['DOCUMENT_ROOT'] . DIRECTORY_SEPARATOR . $_SERVER['SCRIPT_NAME'])) {
    return false;
}

$grav_index = 'index.php';

/* Check the GRAV_BASEDIR environment variable and use if set */
if ($grav_basedir = getenv('GRAV_BASEDIR') ?: '') {
    $grav_index = ltrim($grav_basedir, '/') . DIRECTORY_SEPARATOR . $grav_index;
    $grav_basedir = DIRECTORY_SEPARATOR . trim($grav_basedir, DIRECTORY_SEPARATOR);
    define('GRAV_ROOT', str_replace(DIRECTORY_SEPARATOR, '/', getcwd()) . $grav_basedir);
}

$_SERVER = array_merge($_SERVER, $_ENV);
$_SERVER['SCRIPT_FILENAME'] = $_SERVER['DOCUMENT_ROOT'] . $grav_basedir .DIRECTORY_SEPARATOR . 'index.php';
$_SERVER['SCRIPT_NAME'] = $grav_basedir . DIRECTORY_SEPARATOR . 'index.php';
$_SERVER['PHP_SELF'] = $grav_basedir . DIRECTORY_SEPARATOR . 'index.php';

error_log(sprintf('%s:%d [%d]: %s', $_SERVER['REMOTE_ADDR'], $_SERVER['REMOTE_PORT'], http_response_code(), $_SERVER['REQUEST_URI']), 4);

require $grav_index;

Yep, that does it, not even a warning in sight :) Thank you!

@rhukster I still seem to be getting this issue after trying your fix. Any advice?

@MichaelJDavies Please upgrade Grav to 1.6.13. :)


Corrent rate limiting when running ops with parent/child tables.

Also, throw correct error when multi-table writeMany is issued against server that doesn't support it.

Please fix typo in message (corrent -> correct?) when merging


estimated attendance

Ask from MAIN Event for their calendar listing, but good for us to have a number in mind as well. Last year's calendar listed NC Data Jam est. attendance at 150 with NC CityCamp est attendance at 300.

300
On Jun 22, 2017 3:38 PM, "suzanneccnc"<EMAIL_ADDRESS>wrote:

Assigned #430 https://github.com/codeforraleigh/NCOpenPass/issues/430
to @jhibbets https://github.com/jhibbets.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/codeforraleigh/NCOpenPass/issues/430#event-1135385372,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADgoVz7buqfIOwu1rQ5ys5ffzLjyX74Lks5sGsLIgaJpZM4OCwFc
.

All documented in the prospectus, so we have the exact numbers if interested.

The numbers that are in the prospectus are closer to 225, but 300 sounds good

Thanks Chris! Forgot they were there.

Chris and Jason, what numbers would you say for Saturday of City Camp
(DataPalooza kickoff)?
On Thu, Jun 22, 2017 at 6:17 PM, suzanneccnc<EMAIL_ADDRESS>wrote:

Thanks Chris! Forgot they were there.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/codeforraleigh/NCOpenPass/issues/430#issuecomment-310518115,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQ6Ixn4pGS_ojaI9V057-vuT-4K4Q0YAks5sGuf-gaJpZM4OCwFc
.

--
Zach Ambrose
<EMAIL_ADDRESS>919-438-2752 (office)
919-616-9969 (cell)
www.AmbroseStrategy.com
https://www.linkedin.com/in/zambrose

We did lunch for 80. So say 100-110.
On Fri, Jun 23, 2017 at 7:06 AM zachambrose<EMAIL_ADDRESS>wrote:

Chris and Jason, what numbers would you say for Saturday of City Camp
(DataPalooza kickoff)?
On Thu, Jun 22, 2017 at 6:17 PM, suzanneccnc<EMAIL_ADDRESS>wrote:

Thanks Chris! Forgot they were there.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<
https://github.com/codeforraleigh/NCOpenPass/issues/430#issuecomment-310518115
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AQ6Ixn4pGS_ojaI9V057-vuT-4K4Q0YAks5sGuf-gaJpZM4OCwFc
.

--
Zach Ambrose
<EMAIL_ADDRESS>919-438-2752 (office)
919-616-9969 (cell)
www.AmbroseStrategy.com
https://www.linkedin.com/in/zambrose
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/codeforraleigh/NCOpenPass/issues/430#issuecomment-310638153,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOGtQrzaYAI21byrJfv1vSEJbtM9f7Vks5sG5wpgaJpZM4OCwFc
.

--
Chris Mathews


Ternary conditionals not supported in typescript 2.8

Can we upgrade the typescript version so that there arent any errors during compilation for newer types?

I'm working on updating typescript now so that will be resolved separately. I'm not sure if ternary conditionals will work by default though.
PRs to add a ternary conditional test case would be helpful

Released in v0.15.6


Allow use of markdown for writing docs

[ ] Implement using myst-parser
[ ] Example
[ ] Tests
[ ] Docs

Turns out there is nothing that needs to be done here. :) Just use myst-parser.


Memory leak in np.linalg.inv

Numba implementation of np.linalg.inv is leaking memory. The following script demonstrates it (and check that np.dot does not).
I am willing to help fix this, and I would appreciate any pointers in where to look at and what to look for.
# -*- coding: utf-8 -*-
import gc
import tracemalloc
import numpy as np
from numba import njit

@njit
def dot():
    a = np.ones((100, 100))
    c = np.dot(a, a)
    return c

dot()

@njit
def inv():
    a = np.diag(np.linspace(1, 2, 100))
    c = np.linalg.inv(a)
    return c

inv()

tracemalloc.start()

print('Dot')
for i in range(100):
    d = dot()
    print(tracemalloc.get_traced_memory())

print('Inv')
for i in range(100):
    d = inv()
    print(tracemalloc.get_traced_memory())

gc.collect()
print(tracemalloc.get_traced_memory())

Leads to :
Dot
(80416, 160416)
(80448, 240624)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
(80448, 240656)
Inv
(132944, 293152)
(265368, 345472)
(397792, 477896)
(530216, 610320)
(662640, 742744)
(795064, 875168)
(927488, 1007592)
(1059912, 1140016)
(1192336, 1272440)
(1324760, 1404864)
(1457184, 1537288)
(1589608, 1669712)
(1722032, 1802136)
(1854456, 1934560)
(1986880, 2066984)
(2119304, 2199408)
(2251728, 2331832)
(2384152, 2464256)
(2516576, 2596680)
(2649000, 2729104)
(2781424, 2861528)
(2913848, 2993952)
(3046272, 3126376)
(3178696, 3258800)
(3311120, 3391224)
(3443544, 3523648)
(3575968, 3656072)
(3708392, 3788496)
(3840816, 3920920)
(3973240, 4053344)
(4105664, 4185768)
(4238088, 4318192)
(4370512, 4450616)
(4502936, 4583040)
(4635360, 4715464)
(4767784, 4847888)
(4900208, 4980312)
(5032632, 5112736)
(5165056, 5245160)
(5297480, 5377584)
(5429904, 5510008)
(5562328, 5642432)
(5694752, 5774856)
(5827176, 5907280)
(5959600, 6039704)
(6092024, 6172128)
(6224448, 6304552)
(6356872, 6436976)
(6489296, 6569400)
(6621720, 6701824)
(6754144, 6834248)
(6886568, 6966672)
(7018992, 7099096)
(7151416, 7231520)
(7283840, 7363944)
(7416264, 7496368)
(7548688, 7628792)
(7681112, 7761216)
(7813536, 7893640)
(7945960, 8026064)
(8078384, 8158488)
(8210808, 8290912)
(8343232, 8423336)
(8475656, 8555760)
(8608080, 8688184)
(8740504, 8820608)
(8872928, 8953032)
(9005352, 9085456)
(9137776, 9217880)
(9270200, 9350304)
(9402624, 9482728)
(9535048, 9615152)
(9667472, 9747576)
(9799896, 9880000)
(9932320, 10012424)
(10064744, 10144848)
(10197168, 10277272)
(10329592, 10409696)
(10462016, 10542120)
(10594440, 10674544)
(10726864, 10806968)
(10859288, 10939392)
(10991712, 11071816)
(11124136, 11204240)
(11256560, 11336664)
(11388984, 11469088)
(11521408, 11601512)
(11653832, 11733936)
(11786256, 11866360)
(11918680, 11998784)
(12051104, 12131208)
(12183528, 12263632)
(12315952, 12396056)
(12448376, 12528480)
(12580800, 12660904)
(12713224, 12793328)
(12845648, 12925752)
(12978072, 13058176)
(13110496, 13190600)
(13242920, 13323024)
(13242888, 13323024)

I think that one incorrect thing is the use of impl_ret_borrowed instead of impl_ret_new_ref at https://github.com/numba/numba/blob/master/numba/targets/linalg.py#L641 but this alone does not fix the issue.

It seems the i matrix (holding intermediate results) is never deallocated? Numba doesn't have a GC, we have to manage reference counts explicitly.

The same holds for work. How do I deallocate the memory ?

Calling context.nrt_decref should be sufficient (I'm assuming empty_nd_impl initialized the refcount to 1...).

Should I do the same for values allocated through cgutils.alloca_once ?

Ah no, alloca_once() just allocates stack slot (it's a compile-time instruction really). Those ones are totally ok.

Thanks that fixed it !


Egeria-IGC-connectivity issues - Part II - Connectivity to IGC can't be restored after failed asset mapping

Hi ODPI Egeria team,
I deployed Egeria on a dedicated Kubernetes cluster on the IBM Cloud (Yellow zone). Between the cluster and the IGC (Blue Zone) is a firewall, which was configured to allow communication. This time the communication breaks down after trying to publish an asset type, that can't be mapped. Again: From that point on, it was neither possible to exchange information (Glossary or assets), nor has the connection ever recovered. As in my earlier issue "Egeria-IGC-connectivity issues - Part I", purging and reinstalling the helm chart restores the working state, but naturally leads to losing all assets, that have been transferred up to this point. Here is a piece of the IGC pod log:
2019-06-11 18:48:54.469 ERROR 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : EntityDetail could not be retrieved for RID: b1c497ce.54bd3a08.9g868p8ap.1n94o64.biqmmt.d2vu7883hkqtnc5gknm6a
2019-06-11 18:48:54.469  WARN 1 --- [       Thread-4] o.o.o.a.r.i.r.stores.EntityMappingStore  : Unable to find mapping for IGC type: view
2019-06-11 18:48:54.469 ERROR 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.

Thanks again for submitting -- I've reproduced this one on my side, investigating a fix.

Hi Christopher,
I just realized, I could have a added a bit more information:
2019-06-11 11:37:06.253 ERROR 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : EntityDetail could not be retrieved for RID: b1c497ce.54bd3a08.9g868p8ap.1n94o64.biqmmt.d2vu7883hkqtnc5gknm6a
2019-06-11 11:37:06.253  WARN 1 --- [       Thread-4] o.o.o.a.r.i.r.stores.EntityMappingStore  : Unable to find mapping for IGC type: view
2019-06-11 11:37:06.254 ERROR 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.
java.lang.NullPointerException: null
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getMappers(IGCOMRSMetadataCollection.java:3748) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getIgcPropertiesToRelationshipMappings(IGCOMRSMetadataCollection.java:3381) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:597) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
2019-06-11 11:37:06.254  WARN 1 --- [       Thread-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Synchronous auto-commit of offsets {InfosphereEvents-0=OffsetAndMetadata{offset=4357, leaderEpoch=null, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
2019-06-11 11:37:06.254  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Revoking previously assigned partitions [InfosphereEvents-0]
2019-06-11 11:37:06.254  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] (Re-)joining group
2019-06-11 11:37:06.486  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Successfully joined group with generation 1
2019-06-11 11:37:06.487  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Setting newly assigned partitions [InfosphereEvents-0]
2019-06-11 11:38:41.272  INFO 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Skipping asset - no changes detected: 6662c0f2.e1b1ec6c.9g86m9t3m.t2imv1k.qiqbqo.6tfuum9v5ncp288rbemgc
2019-06-11 11:38:42.361  INFO 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Skipping asset - no changes detected: 6662c0f2.e1b1ec6c.9g86m9t3m.t2imv1k.qiqbqo.6tfuum9v5ncp288rbemgc
2019-06-11 11:42:06.876  WARN 1 --- [Mapper_consumer] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] This member will leave the group because consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
2019-06-11 11:42:06.876  INFO 1 --- [Mapper_consumer] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Sending LeaveGroup request to coordinator xxxxx.xxxx.xx.xx:59092 (id:<PHONE_NUMBER> rack: null)
2019-06-11 11:42:52.275  INFO 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Skipping asset - no changes detected: b1c497ce.60641b50.9g86all5v.kpa48s0.65v6qb.jtgk2uvn4kr74k56g8cdk
2019-06-11 11:43:57.901  INFO 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Skipping asset - no changes detected: b1c497ce.60641b50.9g86all5v.kpa48s0.65v6qb.jtgk2uvn4kr74k56g8cdk

@jan-frommann thanks for the additional details.  Per my comments on #21 it looks like you're using an older version of the connector...  Any chance you can re-test with the latest?

@cmgrote absolutely, just give me a couple of hours...

Hi @jan-frommann -- just a polite follow-up to see if by any chance you've been able to re-test with the latest connector, and had better luck or are still running into the same (or other) problems?

Hi @cmgrote, sorry, it took me so long to respond! I was able to make the majority run, but now job number 3 is having connectivity issues and ultimately fails. I rolled back to the old version and was happy to see, that it still works.
I used most parameters from my old values.yaml, but in some cases (credentials, for instance), I'm not entirely sure, if my settings are correct. For some reason, these errors look familiar....
Also, I spotted in the logs, that the host name can't be resolved and I haven't figured out why yet - the hosts files look good to me. I tested the connectivity using ping and nc -vz for all ports used, but found nothing out of the ordinary.
As far as I can tell, the job fails during instantiation.

I used most parameters from my old values.yaml, but in some cases (credentials, for instance), I'm not entirely sure, if my settings are correct. For some reason, these errors look familiar....

There have been some changes to the values.yaml over time, so it would be worth doing a diff between yours and the latest in master to see if there are missing settings that you need to configure, or if some of the settings have been slightly re-organised for consistency.
The overall IGC area should look like this in the latest values.yaml:
ibmigc:
  enabled: true
  user: isadmin
  password: isadmin
  proxyserver: ibmigc
  internal:
    enabled: false
  external:
    enabled: true
    hostname: "your.hostname.com"
    ip: "<IP_ADDRESS>"
    ports:
      https: "9446"
      broker: "59092"

(That's assuming a cluster-external environment running IGC: naturally change the details around hostname, IP, credentials, ports, etc as needed.)

Also, I spotted in the logs, that the host name can't be resolved and I haven't figured out why yet - the hosts files look good to me. I tested the connectivity using ping and nc -vz for all ports used, but found nothing out of the ordinary.

Hosts can be tricky, with k8s managing its own internal network and DNS...  Are you using a pre-existing IGC host (outside your k8s cluster) or a container that you're running within the cluster?  Make sure you're using the appropriate settings for internal vs external above -- if you're using a container, the hostname of the IGC container must basically be hard-coded to infosvr to be resolvable from everywhere.

I would recommend adding your overrides into a new file - ie ~/etc/cloud.yaml
You would set ONLY the values you want to modify from the default in values.yaml
This avoids changing the file. It won't help if we change the meaning of a parameter, but at least it will help if new values are added
Helm would then be run with
helm install vdc -f ~/etc/cloud.yaml
This doesn't address the failure, but is IMO a better practice for using helm charts

@jan-frommann If you're using an external IGC we basically end up create a host/ip mapping in the containers. But we might be getting deep into the specifics of your network setup. Feel free to ping me and/or @cmgrote on slack as I suspect we'll be into network addresses/ports that might not be best shared in github? . Also if you're using multiple IGCs, that may require other changes

Hi @cmgrote, I'm using an external IGC. In the last version I used, the connection worked mostly fine (the initial connection and the first few updates). The credentials used to be encrypted in the values.yaml - does it have to be clear text now?
Also, I left 'ibmigc' as proxyserver...is that right?
Hi @planetf1, I promise you, I will create such a file! I haven't ignored the advice, I swear - I just haven't gotten around to it yet.
Sounds like a great idea Nigel. I'll reach out to you!

Initial offline analysis is currently pointing to at least one problem (maybe not the root problem, still TBD): that container images being built by default cache the intermediate layers and therefore do not pickup changes to the master codebase on a re-build of the image...  Looks like there are some suggested workarounds to this which we most likely will want to adopt!
eg. https://stackoverflow.com/questions/36996046/how-to-prevent-dockerfile-caching-git-clone

@planetf1 ^^ fyi -- I added those lines into our images that build from Git to ensure the cache is invalidated when there's a code change (included as part of https://github.com/odpi/egeria/pull/1127)

We're building a bit of a backlog on old atlas. I'm getting quite keen to just get things moving again now especially with the super work on IGC.
So... I'm going to start by merging some of the PRs we have outstanding .. and will test on IBM cloud.


Feature: add cloud image kubernetes-reflector

What is the problem this feature will solve?
https://github.com/emberstack/kubernetes-reflector
https://github.com/labring/sealos/blob/main/docs/4.0/i18n/zh-Hans/cloud/local-deploy/index.md using kubernetes-reflector
If you have solution，please describe it
No response
What alternatives have you considered?
No response

https://github.com/labring-actions/cluster-image/issues/343 done


Added Searching Algorithms (in C++)

Issue no. #16
Uploaded the codes of some of the famous Searching Algorithms in C++, these include:

Linear Search Algorithm
Iterative approach of Binary Search Algorithm
Recursive approach of Binary Search Algorithm
Jump Search Algorithm
Interpolation Search Algorithm

Please goto issue section create a issue there or find a related issue and come back here and then tag that issue so that I can assign it to you and merge it.

Issue #16
As requested, I have opened an Issue and tagged the same. Please do the required
With Regards,
Swoyam Siddharth Nayak

Now you have made a valid Contribution but I suggest you to please add a single code next time.

Now you have made a valid Contribution but I suggest you to please add a single code next time.

Sorry, Actually coudn't understand, did you mean that I should upload the complete code in a single file ??


[FEAT] Add initial implementation of Redshift linker

Type of PR

[ ] BUG
[x] FEAT
[ ] MAINT
[ ] DOC

Is your Pull Request linked to an existing Issue or Pull Request?
https://github.com/moj-analytical-services/splink/issues/1220
Give a brief description for the solution you have provided
My coworker @BartBaddeley has implemented a bare-bones linker for Redshift.
Redshift is quite limited really e.g. it doesn't have any built-in string distance functions. Python UDFs could be used to implement any missing functionality but I don't know how fast that would be.
Anyway we haven't needed any such features so far. We think this could be useful to other people.
Maybe someone else will eventually pick this up and add more features. "Scratch your own itch" and all that 🙂
PR Checklist

[ ] Added documentation for changes
[ ] Added feature to example notebooks at tutorial in splink_demos (if appropriate)
[ ] Added tests (if appropriate)
[x] Made changes based off the latest version of Splink
[x] Run the linter

This looks great @mausch - thank you so much for putting a PR together! 🎉👏

Hey @mausch and @BartBaddeley - thanks again for putting this together! We are keen to get this implemented at some point, but we have a number of higher-priority pieces of work in progress so it will probably be a few months before we can look at it properly. I will leave the PR open for now as a reminder to revisit 😊

Thank you for this.  I'm going to closing this for now because we're working on a new and hopefully (eventually) simpler approach to contributing additional backends.
Hopefully we can bring in some of these ideas once we're finished.


Tests

Added all the tests. Separated into test_views.py, test_urls.py, and test_models.py.

@ldko, @damonkelley, this PR is ready. It's a little weird, since I branched after I committed most of the changes to master. So basically this PR covers all the tests, not just what is shown as changed.

@somexpert, any clue on how close this is to working with Python 3?

yep, basically I would just need to replace the __unicode__ methods in the models with __str__ methods, then decorating the models with python_2_unicode_compatible()

at least, that's according to the test results. haven't actually tried it yet.

Couple of things I noticed:

I see there are a few tests that are commented out. What it the plan for those?
Can we get a requirements file added?
The installation requirements can be consolidated. Step 1-3 can be turned into a single setup git clone https://github.com/unt-libraries/django-citeit'

Steps consolidated.

Continuing with the README....

Now that we have the requirements files, do we still need the Additional Testing Requirments section?
What do you think about Django CiteIt as the title instead of CiteIt Django app
Can we up the Django requirement to >= 1.8?

I can get on board with all three of those. Although just FYI, the app won't work with Django 1.10 since the way it does it's URLs is currently deprecated (in 1.8) and then completely removed in that version.

Good riddance!
I was thinking that we were also testing against djangomaster, but it looks like that has since been removed. NM.

Yeah, we can always add that, python 3.4, and flake8 back in if we ever get to cleaning up the code.

Can we get the Travis badge on the README as well?

will do. so adding [ci skip] in the commit message will skip the Travis stuff? Didn't know that.

Yup.
http://docs.travis-ci.com/user/customizing-the-build/#Skipping-a-build

There is a lot of repetition of creating similar/same instances of model objects in test_models.py. Would it make sense to use  a [setUpClass] (https://docs.python.org/2/library/unittest.html#unittest.TestCase.setUpClass) to reduce this?

I completely agree with both comments. Done and done.

On the views that have "_" replaced with " " , you might add tests with underscore values. I'm not sure if that functionality is working as is expected.

Hmmmm, I'll start working on that now.

We might want to have expected failure tests for these issues https://github.com/unt-libraries/django-citeit/issues/3 and https://github.com/unt-libraries/django-citeit/issues/4.

:+1:


Add HOWTO: Batch Normalization

Batch Normalization is more complicated than most layers because of the mutation of moving averages during training.
There is a Discussion on this already (#921), but it may be nice to add a HOWTO for this since we could add the code below as well. The code below is copied from a Colab by @levskaya, which highlights the general state management API you use for any state-computation in a NN.
import numpy as np

import jax
from jax import random, lax, numpy as jnp

import flax
from flax import linen as nn

We define a trivial conv + BN layer
class Foo(nn.Module):
  train: bool
  filters: int

  @nn.compact
  def __call__(self, x):
    x = nn.Conv(self.filters, (1, 1), use_bias=False, dtype=jnp.float32)(x)
    x = nn.BatchNorm(use_running_average=not self.train,
                     momentum=0.9,
                     epsilon=1e-5,
                     dtype=jnp.float32)(x)
    return x

key = random.PRNGKey(0)
x = jnp.ones((5,4,4,3))

# We instantiate the layer then call its init function to get initial variable collections.
foo_vars = Foo(filters=7, train=True).init(key, x)
foo_vars

This returns the following:
FrozenDict({
    params: {
        Conv_0: {
            kernel: DeviceArray([[[[ 0.50138927,  0.7354811 ,  0.7896391 , -0.63713336,
                             1.081016  , -0.29067358, -0.3780927 ],
                           [ 0.7357814 ,  0.24682549, -0.55378306,  0.16909008,
                             0.85014457,  1.0167135 ,  0.19896305],
                           [ 1.1461202 ,  0.8548834 , -1.0578486 , -0.6013309 ,
                             0.2501557 ,  0.3332178 , -0.36248836]]]], dtype=float32),
        },
        BatchNorm_0: {
            scale: DeviceArray([1., 1., 1., 1., 1., 1., 1.], dtype=float32),
            bias: DeviceArray([0., 0., 0., 0., 0., 0., 0.], dtype=float32),
        },
    },
    batch_stats: {
        BatchNorm_0: {
            mean: DeviceArray([0., 0., 0., 0., 0., 0., 0.], dtype=float32),
            var: DeviceArray([1., 1., 1., 1., 1., 1., 1.], dtype=float32),
        },
    },
})

We explicitly say which variable collections are to be mutated by the apply function, those are then returned as auxilliary variables.
y1, new_batch_stats = Foo(filters=7, train=True).apply(foo_vars, x, mutable=['batch_stats'])
new_batch_state

This returns the following:
FrozenDict({
    batch_stats: {
        BatchNorm_0: {
            mean: DeviceArray([ 0.23832898,  0.18371896, -0.08219925, -0.10693741,
                          0.21813159,  0.10592576, -0.05416182], dtype=float32),
            var: DeviceArray([0.9000007 , 0.90000004, 0.9       , 0.90000004, 0.9000001 ,
                         0.9       , 0.9       ], dtype=float32),
        },
    },
})

We stitch together params and batch stats collections to evaluate again.
Normally the params would have been updated by a training step using an optimizer.
new_foo_vars = {'params': foo_vars['params'], 'batch_stats': new_batch_stats}
y2, even_newer_batch_stats = Foo(filters=7, train=True).apply(new_foo_vars, x, mutable=['batch_stats'])

Hey, thanks for the guide. This is useful.
I found something unintuitive:
class SimpleModel(nn.Module):
    @nn.compact
    def __call__(self, x):
        x = nn.BatchNorm()(x)
        x = nn.Dense(1)(x)
        return x
model = SimpleModel()

rng = jax.random.PRNGKey(0)
dummy_input = jnp.ones((1, 10))
params = model.init(rng, dummy_input)

params['params'] gives me:
FrozenDict({
    BatchNorm_0: {
        scale: DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32),
        bias: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),
    },
    Dense_0: {
        kernel: DeviceArray([[ 0.15454544],
                     [ 0.5659187 ],
                     [ 0.04327311],
                     [ 0.08302793],
                     [-0.00323147],
                     [-0.12254263],
                     [-0.22088017],
                     [-0.2184048 ],
                     [-0.05149708],
                     [-0.42985576]], dtype=float32),
        bias: DeviceArray([0.], dtype=float32),
    },
})

and params.pop('params') gives me a tuple:
(FrozenDict({
     batch_stats: {
         BatchNorm_0: {
             mean: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),
             var: DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32),
         },
     },
 }),
 FrozenDict({
     BatchNorm_0: {
         scale: DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32),
         bias: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),
     },
     Dense_0: {
         kernel: DeviceArray([[ 0.15454544],
                      [ 0.5659187 ],
                      [ 0.04327311],
                      [ 0.08302793],
                      [-0.00323147],
                      [-0.12254263],
                      [-0.22088017],
                      [-0.2184048 ],
                      [-0.05149708],
                      [-0.42985576]], dtype=float32),
         bias: DeviceArray([0.], dtype=float32),
     },
 }))

Why are they different?
BTW, I found this usage here:
https://github.com/google/flax/blob/d068512a932da3e05b822790a591bac391aeab36/examples/imagenet/train.py#L69

Hey, thanks for the guide. This is useful.
I found something unintuitive:
class SimpleModel(nn.Module):
    @nn.compact
    def __call__(self, x):
        x = nn.BatchNorm()(x)
        x = nn.Dense(1)(x)
        return x
model = SimpleModel()

rng = jax.random.PRNGKey(0)
dummy_input = jnp.ones((1, 10))
params = model.init(rng, dummy_input)

params['params'] gives me:
FrozenDict({
    BatchNorm_0: {
        scale: DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32),
        bias: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),
    },
    Dense_0: {
        kernel: DeviceArray([[ 0.15454544],
                     [ 0.5659187 ],
                     [ 0.04327311],
                     [ 0.08302793],
                     [-0.00323147],
                     [-0.12254263],
                     [-0.22088017],
                     [-0.2184048 ],
                     [-0.05149708],
                     [-0.42985576]], dtype=float32),
        bias: DeviceArray([0.], dtype=float32),
    },
})

and params.pop('params') gives me a tuple:
(FrozenDict({
     batch_stats: {
         BatchNorm_0: {
             mean: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),
             var: DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32),
         },
     },
 }),
 FrozenDict({
     BatchNorm_0: {
         scale: DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32),
         bias: DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32),
     },
     Dense_0: {
         kernel: DeviceArray([[ 0.15454544],
                      [ 0.5659187 ],
                      [ 0.04327311],
                      [ 0.08302793],
                      [-0.00323147],
                      [-0.12254263],
                      [-0.22088017],
                      [-0.2184048 ],
                      [-0.05149708],
                      [-0.42985576]], dtype=float32),
         bias: DeviceArray([0.], dtype=float32),
     },
 }))

Why are they different?
BTW, I found this usage here:
https://github.com/google/flax/blob/d068512a932da3e05b822790a591bac391aeab36/examples/imagenet/train.py#L69

Hi @cccntu -- what do you mean by "different"? pop returns two parts: the variable collection that you popped, and the remaining collections still all grouped together, e.g.:
variables = model.init(...)
# assume variables['params'] and variables['batch_stats'] are present here
other_variables, params = variables.pop('params')

# here params == variables['params'], and other_variables['batch_stats'] == variables['batch_stats']

Maybe the docstring for FrozenDict.pop isn't clear enough? But this is working as intended.
(BTW in your example you wrote params = model.init(rng, dummy_input) -- it should probably be variables = model.init(rng, dummy_input) as parameters are one of possible multiple variable collections)
Does this help? What could we improve in our documentation so that this would be less confusing?

@avital  Thanks for the reply.
I said different because I expect it to behave like python dict.
But I get it now, since flax/jax is functional, it make sense to return the remaining dictionary. Thanks for the help and the naming suggestion!
I didn't find the doc for FrozenDict from google search and readthedocs search is extremely slow. So I didn't read the doc.
I just found it by searching Github and I think it's clear enough.

I didn't find the doc for FrozenDict from google search and readthedocs search is extremely slow. So I didn't read the doc.
@marcvanzee -- another data point for "slow RTD" ;)

I don't think we even have reference dosc for FrozenDict on RTD, we probably should. I filed an issue: https://github.com/google/flax/issues/969

@cccntu could you tell me a bit more about the slowness of readthedocs? Do you think it is slow in general, or only when you search? What search string did you try? I'd like to see if I can reproduce this.
@avital yes I also noticed we don't have FrozenDict in RTD, thanks for filing the issue!

@marcvanzee I think it's slow in general from my experiences with different packages on RTD. But right now search seems to take forever. (the 'search...' has been running for several minute now.)
Here are some string I just tried: 'BatchNorm', 'FrozenDict',  'dense', 'compact'
(My internet speed is ~100Mbps rn.)
Here is the link produced by using the search box for dense
https://flax.readthedocs.io/en/latest/search.html?q=dense&check_keywords=yes&area=default

Hi @cccntu, that is surprising. I've heard similar complaints from @apaszke from the JAX team as well, but I haven't experienced it myself (your queries are fast when I try it) and also heard from other Flax users they have no problems with it.
In any case, thanks for reporting it, we will see if this is something that occurs more often, and if so, how we can resolve it.

Hey!
I am very interested in the best practices for BatchNorm (or batch_stats in general I guess) when used inside a pmap. Is the recommendation to use lax.pmean to synchronize them? Are there example available?
Thanks!

I am very interested in the best practices for BatchNorm (or batch_stats in general I guess) when used inside a pmap

See the ImageNet example for the canonical example of combining BatchNorm and pmap. There indeed we sync the statistics before evaluation with pmean.


Missing setuptools dependency

Prework

[x] Read and agree to the code of conduct and contributing guidelines.
[x] If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.

Description
great tables imports pkg_resources but doesn't list setuptools as dependency.
Reproducible example

[x] Post a minimal reproducible example (MRE) so the maintainer can troubleshoot the problems you identify. A reproducible example is:

[x] Runnable: post enough code and data so any onlooker can create the error on their own computer.
[x] Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
[x] Readable: format your code according to the Style Guide for Python Code.

Using uv to create the virtual environment because it doesn't include setuptools by default.
uv venv && source .venv/bin/activate && uv pip install great_tables && python3 -c "import great_tables"

Using Python 3.10.13 interpreter at: /home/glefalher/.pyenv/versions/3.10.13/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Resolved 17 packages in 16ms
Installed 17 packages in 27ms
 + babel==2.14.0
 + commonmark==0.9.1
 + great-tables==0.4.0
 + htmltools==0.5.1
 + importlib-metadata==7.1.0
 + mizani==0.11.1
 + numpy==1.26.4
 + packaging==24.0
 + pandas==2.2.1
 + python-dateutil==2.9.0.post0
 + pytz==2024.1
 + scipy==1.13.0
 + six==1.16.0
 + typing-extensions==4.11.0
 + tzdata==2024.1
 + webcolors==1.13
 + zipp==3.18.1

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/glefalher/nwork/gtpkg2/.venv/lib/python3.10/site-packages/great_tables/__init__.py", line 10, in <module>
    from .gt import GT
  File "/home/glefalher/nwork/gtpkg2/.venv/lib/python3.10/site-packages/great_tables/gt.py", line 15, in <module>
    from great_tables._formats import (
  File "/home/glefalher/nwork/gtpkg2/.venv/lib/python3.10/site-packages/great_tables/_formats.py", line 20, in <module>
    from ._locale import _get_locales_data, _get_default_locales_data, _get_currencies_data
  File "/home/glefalher/nwork/gtpkg2/.venv/lib/python3.10/site-packages/great_tables/_locale.py", line 4, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

Expected result
great_tables should be importable
Development environment

Operating System: [e.g., Linux, macOS, Windows]
great_tables Version: 0.4.0
uv 0.1.26

Additional context
The same example works when using python to create the virtual enviroment
python3 -m venv venv && source venv/bin/activate && python3 -m pip install -U great_tables && python3 -c "import great_tables"

According to the doc:

Use of pkg_resources is deprecated in favor of :mod:importlib.resources, :mod:importlib.metadata and their backports (:pypi:importlib_resources, :pypi:importlib_metadata).

I am getting the same ModuleNotFoundError: No module named 'pkg_resources' using Poetry.

I noticed hit this while working on #261, and replaced setuptools code with importlib_resources. Should be resolved their!

Fixed by https://github.com/posit-dev/great-tables/pull/261.


Add new commands, cmake.buildType and cmake.buildDirectory

This change addresses items #564 and #334

${command:cmake.buildType} can be used to get the selected build type.
${command:cmake.buildDirectory} can be used to get the current build directory.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: notskm sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thank you!


Hello, could you please send me the processed or original data? I've been applying here for over a week and haven't come down yet

Hello, could you please send me the processed or original data? I've been applying here for over a week and haven't come down yet

It is against the agreement to disclose the original data, and it is not difficult to apply for datasets such as ADNI and AIBL.  If the data application is not approved for a long time, please supplement your application form.


Improved Error Handling and Testability

Added some more logic to role and app selection, so the script exits with a useful message if the user has no apps or roles available to them.
Refactored the gimme-aws-creds file so that the project is more testable, by moving the GimmeAWSCreds class into gimme_aws_creds/main.py.  Implemented a few tests for role and app input.  We'll want to add more tests to improve code coverage.
Added another, more direct installation method to the readme file, which leverages pip.

LGTM


Feature: preview equation on hover for referenced equations.

Inspired by LaTeX Workshop, the goal is to dynamically display the referenced equation when hovered over a reference link in a notie page. The result should look like the following:

However, instead of hovering over an equation when editing, user should hover over a rendered reference link. For example, in the following notie render below,

When users hover over (1.11), notie should dynamically display the rendered equation D = E + F.

Result here:


Use RBAC for authentication and authorization

This modifies @luksa's initial PR #454 to use the k8s rest api directly as https://github.com/fabric8io/kubernetes-model/pull/240 has not yet been merged to master.

@luksa An idea I just had:
How about instead of using nonResourceUrls for the role, using the 'configmap' resource? we don't use configmaps for anything other than storing address space and address resources, so having a Role for CRUD operations on a configmap would give the same effect as using CRDs + a Role for those.
That would remove the need for creating the 'enmasse-admin' cluster role + any 'enmasse-address-space-X' cluster roles for each address space and instead create a namespaced Role.
Wdyt?

While I don't like the idea of abusing features by not using them as they were intended to be used, abusing ConfigMaps is not any worse than abusing nonResourceUrls. So, I like this change. I guess you're ok with allowing users to actually change the ConfigMap manually?
You can now use this same mechanism for per-addressspace rights by granting access to a specific addressspace ConfigMap. You can even make it more fine-grained: users which can read a specific ConfigMap, can perform read operations inside the addressspace; those who can put/patch the ConfigMap, can perform update operations inside the addressspace.

@luksa Exactly my thoughts as well, thanks for confirming! I'm doing some more changes that I will push (along with review fixes) where the basic-auth for OSB is removed and only supports RBAC. I will also change AuthInterceptor to only do authentication, and set a SecurityContext (which performs the subject access review) that can be used in the jax-rs handlers for performing more fine-grained authorization on address spaces and addresses. Do you think that is a viable approach?

Sounds good!


Offline approvement is not working

Basically, I can't see contours when I'm trying to understand should I approve or rejected the image.

Nevertheless, when I'm downloading results rejected by me they actually have something inside. In the archive below you can find example .tsv results - all and rejected only.
ExampleTSV.zip

@nmingazov, delayed acceptance feature was implemented in version 1.4, and is now available in develop-toloka branch.
See this video for usage reference.
NOTE: You can also download the source files that are needed to run the system on Yandex Toloka crowdsourcing platform (main.html, app.js, and concat.min.css) from the "Releases - Version 1.4." repository page.

I tried your new release and find out the following bugs in delayed acceptance mode:

Navigation using hand instrument isn't working at all
If I click in empty space, always creates new point. I think it is not needed for acceptance mod.
Zoom works only to the top left corner. Usually it zooms to the centre.
During the zoom, points are not moving with the picture and stand still.
It would be great if near marked blobs will be shown their's classes.

@mihbort, thank you for your feedback.

Navigation using hand instrument isn't working at all

Please, elaborate more on this. What exactly is not working? If possible, record a screencast showing inappropriate system behavior, provided with your comments.

It would be great if near marked blobs will be shown their's classes.

We'll take your feedback into account.
As for other points, we are considering whether delayed acceptance users will need to perform minor fixes of (possibly extremely complex) markup submitted by Mechanical Turks, and, consequently, will need to use the full specter of markup functionality.
For the release 1.4 we concentrated on the straightforward delayed acceptance functionality, meaning checking out objects marked up by Mechanical Turks, their classes and parameters (which can be viewed in the "Label Parameters" block, after the marked up object was selected with the "Hand" tool).

@AlNedorezov @mihbort we could close this issue and split the improvements into different tasks, I believe this is a better way to track it. Is it ok for everyone?

@nmingazov It's Ok. In each task I can explain better what is wrong.

@nmingazov it's ok. I agree that it will be a better way to track issues and improvements.

If I click in empty space, always creates new point. I think it is not needed for acceptance mod.
During the zoom, points are not moving with the picture and stand still.

I wasn't able to repeat this bug again, so decided not to open such issues.
What about this issue, I think it can be closed.


TuyaOpenAPI response: {"code":1106,"msg":"permission deny"

Describe the bug
New devices are not added.  They show up in Smart Life and on the Tuya IoT Platform.  When I turned on Debug mode I seen the following error.
After uninstalling the Tuya plugin and reinstalling it no longer adds any devices.
Thank you for any assistance.
Initializing TuyaPlatform...
TuyaOpenAPI request: method = post, endpoint = https://openapi.tuyaus.com, path = /v1.0/iot-01/associated-users/actions/authorized-login, params = null, body = {"country_code":1,"username":"wrmacj@outlook.com","password":"91d9724498318b07aa4c48eeb0d4bee0","schema":"smartlife"}, headers = {"t":"1666220994913","client_id":"7facv9y9wnhgtcfu8qrf","nonce":"24a4af91-5003-11ed-8c9b-25b59857685a","Signature-Headers":"client_id","sign":"D70EFF696B52404977362F72334810301ABEAC439A2A04ADFAFC56FAE519E39B","sign_method":"HMAC-SHA256","access_token":"","lang":"en","dev_lang":"javascript","dev_channel":"homebridge","devVersion":"1.5.0"}
[10/19/2022, 7:09:54 PM] Homebridge v1.5.0 (HAP v0.10.2) (TuyaPlatform) is running on port 35360.
TuyaOpenAPI response: {"code":1106,"msg":"permission deny","success":false,"t":1666220995329,"tid":"24e982cb500311eda5021297425bcc23"} path = /v1.0/iot-01/associated-users/actions/authorized-login
TypeError: Cannot destructure property 'access_token' of 'res.result' as it is undefined.
at TuyaSHOpenAPI._refreshAccessTokenIfNeed (/homebridge/node_modules/homebridge-tuya-platform/lib/tuyashopenapi.js:49:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at TuyaSHOpenAPI.request (/homebridge/node_modules/homebridge-tuya-platform/lib/tuyashopenapi.js:145:7)
at TuyaSHOpenAPI.getDevices (/homebridge/node_modules/homebridge-tuya-platform/lib/tuyashopenapi.js:63:15)
at TuyaPlatform.initTuyaSDK (/homebridge/node_modules/homebridge-tuya-platform/index.js:90:19)
Attention⚠ ⚠ ⚠ ! You get an error!
Failed to get device information. Please check if the config.json is correct.
        "options": {
            "username"<EMAIL_ADDRESS>            "password": "xxxxxxxxxxxxxxxxxxxx",
            "accessId": "xxxxx",
            "accessKey": "xxxxx",
            "lang": "en",
            "projectType": "2",
            "appSchema": "smartlife",
            "countryCode": 1,
            "debug": true
        },
        "name": "TuyaPlatform",
        "_bridge": {
            "username": "xxxxx",
            "port": 35360
        },
        "platform": "TuyaPlatform"

If I restore a config from last week this is the debug.  The think is that I do not know if I had a problem at this time.  I have reset my

I had updated my Smart Life password.  I had mistakenly been using my Tuya password....Who knew :)  Arrrg....
Sorry.


--version-suffix description is incorrect

The description is Defines the version suffix for an asterisk (*) in the version field of the project file. The format follows NuGet's version guidelines..
Testing with the .NET CLI v2.1 I get errors like the following when including an (*) anywhere in my <Version></Version> element in my .csproj
'*' is not a valid version string.
I have tried the following in my .csproj
<Version>1.0.0-*</Version>
<Version>1.0.0.*</Version>
<Version>*</Version><VersionPrefix>1.0.0</VersionPrefix>
using the command dotnet build .\PathTo.csproj --version-suffix abc which works correctly if I only have <VersionPrefix></VersionPrefix> or if I do not have (*) in my <Version></Version>.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 910cd365-d63a-00dc-417c-db017f83bf6f
Version Independent ID: 5e495ddd-140e-de5c-a793-d4a9f88be51e
Content: dotnet build command - .NET Core CLI
Content Source: docs/core/tools/dotnet-build.md
Product: dotnet-core
GitHub Login: @mairaw
Microsoft Alias: mairaw

@livarcocc @KathleenDollard @dsplaisted can one of you help with this question? Should I simply change the description for all SDK versions to what we have in --help ("Defines the value for the $(VersionSuffix) property in the project")? Anything else we can say about that option?

The simplest fix to the docs would be just to change it to what is in the command line help.
However, if we want to do better, here's the actual logic for how the Version is set:
  <PropertyGroup Condition=" '$(Version)' == '' ">
    <VersionPrefix Condition=" '$(VersionPrefix)' == '' ">1.0.0</VersionPrefix>
    <Version Condition=" '$(VersionSuffix)' != '' ">$(VersionPrefix)-$(VersionSuffix)</Version>
    <Version Condition=" '$(Version)' == '' ">$(VersionPrefix)</Version>
  </PropertyGroup>

So if the Version is not otherwise set, then Version will be set to the VersionPrefix combined with the VersionSuffix, separated by a dash.

Thanks @dsplaisted. Also copying here another feedback I got for this option that was sitting on an internal backlog:
"--version-suffix <VERSION_SUFFIX> Defines the version suffix for an asterisk (*) in the version field of the project file. The format follows NuGet's version guidelines. That's not true, the asterisk isn't replaced. See https://github.com/dotnet/cli/issues/5681 The correct documentation should be: 1.0.0$(VersionSuffix)"


Filter pending state manager ops while getLocalState by sequence number.

Vlad called out the fact that we should filter saved ops in the pending state manager based on their sequence number.
Interesting enough, I needed to bring back Tyler hack on one of his tests that removed savedOps, otherwise our assert ""saved op should already have a sequence number"" is triggered. If I remove the assert, we end up trying to load from a newer snapshot.
More test coverage is needed and the feature will remain disabled until we have improve our refresh signal and test coverage.

I do not fully understand that hack that Tyler added (or the flow that caused it) - I'd need to debug that UT to learn.
But other changes look correct to me.
The fact that tests pass is great from one POV, but really bad from another POV. It basically says - we have zero coverage for these kinds of interesting cases. I'd definitely invest into some kind of either stress test (that randomly tries to hit all these cases) or a test skeleton function that has a bunch of knobs and can expose different orderings, and thus we can call it many times with all combinations of knobs to ensure system performs correctly with all such cases.
This will give us more confidence. I.e. right now, if we treat actual behavior as black box, we have two solutions that pass tests, but it's impossible for both of them to be correct (but possible for both solutions to be wrong). It would be great to have signal that more reliably tells "wrong" (it will never tell us "It's correct", but if it filters out all the cases we can come up with that are wrong, then it's enough in terms of building confidence).

I will work in follow ups provided here in subsequent PRs and track it here https://dev.azure.com/fluidframework/internal/_workitems/edit/7891


Change canvasInRendering to a WeakSet instead of a WeakMap

Note how nowhere in the code canvasInRendering.get() is ever called, and that this structure is really only used to store references to <canvas> DOM elements.
The reason for this being a WeakMap is probably because at the time we weren't using core-js polyfills yet, and since there already existed a manually implemented WeakMap polyfill it was probably simpler to use that.

/botio test

Looks like a better choice to me too. Thanks!


chore(differenceWith): updated code example for difference with to be…

…tter illuminate the behaviour of the function through the documentation

This looks good to me once @CrossEye's changes are made. A few things I considered were whether this behavior is tested for (it is) and whether we need to consider this for other set operations. I don't think we do because all other operations are symmetrical.
:bowtie:

Not to bikeshed, but I wonder whether a different set of examples would be more helpful. The problem with these examples is that they use R.equals, which may not make it clear to the reader that the predicate is being used to dedupe the result array.
Here's another suggestion, worked into the existing docs:
 *      const cmp = (x, y) => x.a === y.a;
 *      const l1 = [{a: 1}, {a: 2}, {a: 3}];
 *      const l2 = [{a: 3}, {a: 4}];
 *      R.differenceWith(cmp, l1, l2); //=> [{a: 1}, {a: 2}]
 *
 *      const l3 = [{a: 2}, {a: 2}, {a: 3}];
 *      R.differenceWith(cmp, l3, l2); //=> [{a: 2}]
 *      R.differenceWith(cpm, l3, []); //=> [{a: 2}, {a: 3}]

@kurtmilam: Not to bikeshed your bikeshed :smile:, but I have a difficult time distinguishing between
 *      R.differenceWith(cmp, l3, l2); //=> [{a: 2}]

and
 *      R.differenceWith(cmp, 13, 12); //=> [{a: 2}]

("ell-3" versus "thirteen")
I like this example, but perhaps with different variable names.

I completely agree about the variable names. I just kept the l1 and l2 currently in the docs and extended it to l3, and I can report that it was a little confusing to keep them straight.
How about swapping 'l' for 'a (for 'array'):
 *      const cmp = (x, y) => x.a === y.a;
 *      const a1 = [{a: 1}, {a: 2}, {a: 3}];
 *      const a2 = [{a: 3}, {a: 4}];
 *      R.differenceWith(cmp, a1, a2); //=> [{a: 1}, {a: 2}]
 *
 *      const a3 = [{a: 2}, {a: 2}, {a: 3}];
 *      R.differenceWith(cmp, a3, a2); //=> [{a: 2}]
 *      R.differenceWith(cpm, a3, []); //=> [{a: 2}, {a: 3}]

Or maybe list123, list34 and list223:
 *      const cmp = (x, y) => x.a === y.a;
 *      const list123 = [{a: 1}, {a: 2}, {a: 3}];
 *      const list34 = [{a: 3}, {a: 4}];
 *      R.differenceWith(cmp, list123, list34); //=> [{a: 1}, {a: 2}]
 *
 *      const list223 = [{a: 2}, {a: 2}, {a: 3}];
 *      R.differenceWith(cmp, list223, list34); //=> [{a: 2}]
 *      R.differenceWith(cpm, list223, []); //=> [{a: 2}, {a: 3}]

@kurtmilam: Sorry, I hadn't remembered that we had l1/l2 anywhere in the docs.
I really like your last examples.  I think those show it off better than anything else we've tried.
You are probably right that this behavior was intended.  When I sit down to think of what I'd want, I can't come up with anything better.  I really don't remember the history of this function, but I think there's no reason to change it unless someone can do better.


Document repository

Add needed documentation

Some changes in https://github.com/alastria/alastria-identity/wiki.
@VictorNS69, is Trello/Miro still in use?

I don't think so. We migrated all our boards to GitHub, and Trello is kinda outdated.
Im not sure about Miro.

The information has been updated on the alastria-identity wiki, and the rest of the project's repositories now point to it. We leave the SU open to continue improving the onBoarding process for new developers and users

Needed points to be documented:

[ ] NPM scripts (inside package.json)
[ ] Truffle versions
[ ] Configurations used
[ ] Smart Contracts structure
[ ] How to test this in a test network

For the first point I don't know the detail that you want. As well, for the second one, just mention the 5.1.53 version which it is installed once you run the npm i ?

For the first point, we want to explain what the command does. For example, "the command compile compiles the Smart contracts". I think that with this little explanation is enough.
To be honest, I'm not sure what we should explain in the second point. Any hint @nefera606 ?

Everything is written in #31 .


Filter fix publishing

Docs Build status updates of commit e9b8ecb:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msbuild-api-docs/xml/_filter.xml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


cmd/compile: byte array fields escape to the heap if they are sliced and passed to copy

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version devel +4909ecc Sun Mar 26 21:29:31 2017 +0000 linux/amd64
What operating system and processor architecture are you using (go env)?
linux/amd64
What did you do?
package main

type Buf struct {
	buf [100]byte
	off int
}

// Insert inserts b at offset i.
func (a *Buf) Insert(i int, b byte) {
	a.off++
	copy(a.buf[i+1:a.off], a.buf[i:a.off-1])
	a.buf[i] = b
}

func main() {
	var a Buf
	a.Insert(0, 1)
}

What did you expect to see?
a does not escape to the heap
What did you see instead?
./escape.go:9:31: leaking param content: a
./escape.go:9:31: 	from a.buf (dot of pointer) at ./escape.go:11:26
./escape.go:9:31: 	from a.buf (address-of) at ./escape.go:11:30
./escape.go:9:31: 	from a.buf[i:a.off - 1] (slice) at ./escape.go:11:30
./escape.go:9:31: 	from *a.buf[i:a.off - 1] (indirection) at ./escape.go:11:30
./escape.go:9:31: 	from copy(a.buf[i + 1:a.off], a.buf[i:a.off - 1]) (copied slice) at ./escape.go:11:6

@brtzsnr @randall77 @dr2chase

I think you're misinterpreting the escape analysis output.  a is not escaping, its content is.  Its contents contains no pointers, so that isn't really relevant to a itself escaping.
The assembly output is clear that a is allocated on the stack in main.
Reopen if I'm missing something.  Certainly the escape analysis output could be clarified.

@randall77 thanks Keith. I've confirmed that this doesn't allocate. Thanks
lucky(~/src/esc) % go test -bench=.
goos: linux
goarch: amd64
pkg: esc
BenchmarkBufInsert-4    100000000               13.7 ns/op             0 B/op          0 allocs/op


docs: clarify behavior of hedge_on_per_try_timeout

@snowp I had a hard time understanding the original phrasing so I took a stab at rewriting it, but I'm not really sure I'm describing how it actually works.

Thanks for improving the docs!
@mpuncel You wanna give this a look before I do?
@ikonst Can you fix DCO? There are steps to do so in CONTRIBUTING.md

bump @mpuncel

@ikonst I think this change looks good now. Could you a) merge master b) fix the formatting issue and c) apply the same change to the V3 docs (then run tools/proto_format.sh fix to propagate to v4alpha)?
Also friendly reminder that force pushing breaks the reviewing flow for many, so avoid it if you can.
Thanks!

I'm not a fan of rebases either since they break the already-reviewed / new-commits separation. (Perhaps I did this one because I forgot some sign-offs mid-way?)

retest

/retest

/retest

@snowp ^

/lgtm v2-freeze
/lgtm api


Fix channel --remove command

Issue #204
Order of the arguments was incorrect in the URL generation.

LGTM: this api endpoint is located here: https://github.com/Anaconda-Server/anaconda-server/blob/develop/binstar/sites/api/views/channels.py#L101


Sol-util: API changes

#1673

+1

It seems ok, but I believe that at this point you'll need to rebase and fix a few new occurrences of the old API

#2011


Sync eng/common directory with azure-sdk-tools for PR 9300

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/9300  See eng/common workflow

/check-enforcer override
These are existing failures in main.


Integration Test - Release - 2019/05/10 10:06:44

Auto create for integration test. AnotherAuthorUpdatedArticleFlow

@adunndevster : Thanks for your contribution! The author, @v-caxian, has been notified to review your proposed change.


Cannot install

Hi,
I'm pretty new to node.js but I want to use my xbox controller with my app. However, I hit the following issue when trying to install:
Registry url: https://registry.npmjs.org/
Current Time: 26/02/2015 11:29:05 AM
Last Refreshed: 26/02/2015 8:27:25 AM
Number of Results: 135668
====Executing command 'npm install xbox-controller --save '====
npm WARN package.json<EMAIL_ADDRESS>No repository field.

<EMAIL_ADDRESS>preinstall C:\Users\cslaviero\Documents\Visual Studio 2013\Projects\NodejsConsoleApp2\NodejsConsoleApp2\node_modules\xbox-controller\node_modules\node-hid
sh get-hidapi.sh
'sh' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\Program Files\nodejs\\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "xbox-controller" "--save"
npm ERR! node v0.12.0
npm ERR! npm  v2.5.1
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>preinstall: sh get-hidapi.sh
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>preinstall script 'sh get-hidapi.sh'.
npm ERR! This is most likely a problem with the node-hid package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     sh get-hidapi.sh
npm ERR! You can get their info via:
npm ERR!     npm owner ls node-hid
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\cslaviero\Documents\Visual Studio 2013\Projects\NodejsConsoleApp2\NodejsConsoleApp2\npm-debug.log

====npm command completed with exit code 1====
Hopefully, it's something obvious I am doing wrong. I tried installing Cygwin to overcome the problem but that seemed to open up more issues.
Any help is greatly appreciated.
Thanks

Hi starbug,
you can wrap your console output in 3 backticks (```) to prevent it from being interpreted as markdown.
The error states, that a linux shell is needed. So cygwin is the right path. You could also use the GitShell from GitHub for Windows.
The package doesn't seem to work on node >= 0.11.
Try with node 0.10, that works for me.

Since you wrote you're new to node, it might be nice to know there is a tools to switch node versions. https://github.com/coreybutler/nvm-windows

Wow, thanks for the quick reply. I will give it go.

Hi @starbug8866
If you trying to install by npm install xbox-controller on windows like @bwin said you can use git shell to do this. But before this you need install Visual Studio 2013 (for windows 8 and 8.1) because xbox-controller uses node-hid that uses node-gyp to build the C++ addon.
These links can be usefl:

node-hid prerequisites
node-gyp installation

Thanks mapaiva! That worked


add malawi homepage fb share image

@projectmushroom the {{request.site.root_url}} or root URL image is already defied on the else statement.  Maybe you can feed this Zathu static image there?

@Mitso sure NP, was trying to avoid the extended length inline statement, updated


SSL handshake failed (-9829)

Hello !
Thanks for NWPusher, it really nice for my work !
But today, i get this error "SSL handshake failed (-9829)", and my app crash.
Problem is, not alway crash when app run. Sometime it error, sometime it OK
This my code (swift)

This error

Sorry my bad English !
Thanks for support !
Regard !

Hi. Error -9829 is described in SecureTransport.h with "The certificate is unknown" (errSSLPeerCertUnknown). A similar issue has been reported two years ago: https://github.com/noodlewerk/NWPusher/issues/8. I believe the problem there was that the file contained multiple certificates. Can you take a look there and see if your issue is related?


ntext, text, image SQL Data Types Not Supported

When I tested these data types that will be removed in the future, I got the following error:
TEXT, NTEXT, SQL_VARIANT and IMAGE types cannot be used as column types in OPENJSON function with explicit schema.

We should document this in relnotes at a minimum.

@dzsquared any thoughts on how to best accomplish this one?

by design, no deprecated types 🔥


[SPARK-15180][SQL] Support subexpression elimination in Fliter

What changes were proposed in this pull request?
This patch tries to add the support of subexpression elimination in wholestage codegen Fliter.
Because the predicate expressions are evaluated in Filter with an optimized ordering that reduces unnecessary evaluation as much as possible, we follow this ordering when doing subexpression elimination too.
Due to that, we can't just extract all common subexpressions and evaluate them first. Instead, we extract common subexpressions but don't evaluate them. We evaluate common subexpressions only when the predicate expressions containing them are evaluated.
How was this patch tested?
Existing tests.

Codes:
val ds = Seq(("a", 10), ("a", 20), ("b", 1), ("b", 2), ("c", 1)).toDS().filter("_2 + 1 > 5").filter("_2 + 1 > 20")
ds.collect()

Generated codes:
/* 030 */   protected void processNext() throws java.io.IOException {
/* 031 */     /*** PRODUCE: Filter (((_2#3 + 1) > 20) && ((_2#3 + 1) > 5)) */
/* 032 */
/* 033 */     /*** PRODUCE: INPUT */
/* 034 */
/* 035 */     while (inputadapter_input.hasNext()) {
/* 036 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 037 */       /*** CONSUME: Filter (((_2#3 + 1) > 20) && ((_2#3 + 1) > 5)) */
/* 038 */
/* 039 */       /* ((input[1, int] + 1) > 20) */
/* 040 */       // Common expression
/* 041 */       /* (input[1, int] + 1) */
/* 042 */       /* input[1, int] */
/* 043 */       /* input[1, int] */
/* 044 */       int inputadapter_value1 = inputadapter_row.getInt(1);
/* 045 */
/* 046 */       int filter_value = -1;
/* 047 */       filter_value = inputadapter_value1 + 1;
/* 048 */
/* 049 */       /* (input[1, int] + 1) */
/* 050 */
/* 051 */       boolean filter_value3 = false;
/* 052 */       filter_value3 = filter_value > 20;
/* 053 */       if (!filter_value3) continue;
/* 054 */       /* ((input[1, int] + 1) > 5) */
/* 055 */       /* (input[1, int] + 1) */
/* 056 */
/* 057 */       boolean filter_value5 = false;
/* 058 */       filter_value5 = filter_value > 5;
/* 059 */       if (!filter_value5) continue;
/* 060 */
/* 061 */       filter_numOutputRows.add(1);
/* 062 */
/* 063 */       /*** CONSUME: WholeStageCodegen */
/* 064 */
/* 065 */       /* input[0, string] */
/* 066 */       boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 067 */       UTF8String inputadapter_value = inputadapter_isNull ? null : (inputadapter_row.getUTF8String(0));
/* 068 */       filter_holder.reset();
/* 069 */
/* 070 */       filter_rowWriter.zeroOutNullBytes();
/* 071 */
/* 072 */       if (inputadapter_isNull) {
/* 073 */         filter_rowWriter.setNullAt(0);
/* 074 */       } else {
/* 075 */         filter_rowWriter.write(0, inputadapter_value);
/* 076 */       }
/* 077 */
/* 078 */       filter_rowWriter.write(1, inputadapter_value1);
/* 079 */       filter_result.setTotalSize(filter_holder.totalSize());
/* 080 */       append(filter_result);
/* 081 */       if (shouldStop()) return;
/* 082 */     }
/* 083 */   }
/* 084 */ }

Test build #57997 has started for PR 12956 at commit 55e43ef.

Test build #57997 has finished for PR 12956 at commit 55e43ef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SubExprEliminationState(isNull: String, value: String, exprCode: Option[ExprCode])

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57997/
Test FAILed.

Merged build finished. Test FAILed.

Test build #58083 has started for PR 12956 at commit ff1b7d0.

Test build #58083 has finished for PR 12956 at commit ff1b7d0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58083/
Test FAILed.

Test build #58091 has started for PR 12956 at commit b40fd4b.

Test build #58091 has finished for PR 12956 at commit b40fd4b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58091/
Test PASSed.

Merged build finished. Test PASSed.

cc @davies This is ready for review. Please take a look of this. Thanks.

@viirya It's too complicated, we may not do this. I think we will not support  subexpression elimination in most of the operators.

@davies yea, I think so. The evaluation ordering of predicates makes subexpression elimination more complicated. I will try to simplify it. If it can't be, I will close it then. Thanks.


BertConfig object has no attribute pruned_layers_dim

I ran
bash factorize.sh
factorize.sh is:
python ./flop/factorize.py \
  --bert_config_file=./uncased_L-12_H-768_A-12/bert_config.json \
  --checkpoint=/workspace/BERT-Pruning/output/fine_tune_outputs/SST-2/lr_3e-5/model.ckpt-6313 \
  --output_dir=./uncased_L-12_H-768_A-12_SST-2_f/bert_model_f.ckpt \
  --finetuned

I got the following error
Traceback (most recent call last):
  File "./flop/factorize.py", line 135, in <module>
    finetuned=args.finetuned)
  File "./flop/factorize.py", line 69, in save_factorized_model
    finetuned=finetuned)
  File "./flop/factorize.py", line 46, in create_model
    factorize=True)
  File "/workspace/BERT-Pruning/flop/modeling_flop.py", line 178, in __init__
    pruned_layers_dim=config.pruned_layers_dim)
AttributeError: 'BertConfig' object has no attribute 'pruned_layers_dim

Any comments / help would be really appreciated :)

{
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 30522,
  "pruned_layers_dim": []
}

Does this fix it?

Sorry to see your message now. The code does have such a bug. Adding the pruned_layers_dim attribute in the config file can be a solution. However, I have made a modification to the code and commit. This new code does not need to modify the config file.

Thanks for finding this bug. If there are other problems, I am also willing to help you : )

Hello,
Thank you for helping me with that fix, unfortunately I am getting this error still:
`
INFO:tensorflow:Tensor: bert/encoder/layer_11/output/dense_p/kernel:0 INIT_FROM_CKPT
INFO:tensorflow:Tensor: bert/encoder/layer_11/output/dense_q/bias:0 INIT_FROM_CKPT
INFO:tensorflow:Tensor: bert/encoder/layer_11/output/dense_q/kernel:0 INIT_FROM_CKPT
WARNING:tensorflow:From ./flop/remove_mask.py:151: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
Traceback (most recent call last):
File "./flop/remove_mask.py", line 182, in
threshold=args.threshold)
File "./flop/remove_mask.py", line 157, in remove_mask
1 - dense_pruned_params / dense_total_params),
ZeroDivisionError: division by zero
`
Do you know what I can do about it?
I printed this
print("\n*****\n", dense_total_params, "\n", dense_pruned_params, "\n", dense_origin_params, "\n", non_kernel_params, "\n", total_params, "\n", pruned_total_params, "\n", total_params, "\n*****\n")
and got the following output
`

0
0
0
23959298
23959298
23959298
23959298
*****`

It seems that the code remove_mask.py may have some problems, maybe I forgot to upload the latest code last year. I will figure out how to solve these problems these days.

I ran through all five steps of the project (please see the Structured Pruning section of the README if you are unclear) and found that the code can run through. Probably because my README is not clear enough, it has brought you a lot of problems. Before running the code, please make sure your GPU memory is greater than 24GB.

In the first step, you need to finetune Google's BERT pretrained model on a task (SST-2 for example), using the following example command:

export ROOT_DIR=`pwd`
export BERT_BASE_DIR=$ROOT_DIR/uncased_L-12_H-768_A-12
export GLUE_DIR=$ROOT_DIR/datasets
export DATA_DIR=$ROOT_DIR

task_name="SST-2"

batch_size="32"
max_seq_length="128"
fine_tune_epoch="3.0"
learning_rate="3e-5"

cd bert

python run_classifier.py \
    --task_name=$task_name \
    --do_train=true \
    --do_eval=true \
    --data_dir=$GLUE_DIR/$task_name/ \
    --vocab_file=$BERT_BASE_DIR/vocab.txt \
    --bert_config_file=$BERT_BASE_DIR/bert_config.json \
    --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
    --max_seq_length=$max_seq_length \
    --train_batch_size=$batch_size \
    --learning_rate=$learning_rate \
    --num_train_epochs=$fine_tune_epoch \
    --output_dir=$DATA_DIR/fine_tune_outputs/$task_name/lr_$learning_rate

Then you have an original finetuned BERT model in $DATA_DIR/fine_tune_outputs/$task_name/lr_$learning_rate.

Second step, run factorize.sh to factorize the BERT model into BERT-f:

The path ./path/to/finetuned/model/bert_model.ckpt should be same as $DATA_DIR/fine_tune_outputs/$task_name/lr_$learning_rate which you get in the first step.
At this time, there is no bert_config.json and vocab.txt in the output directory ./uncased_L-12_H-768_A-12_SST-2_f, you need to copy them from ./uncased_L-12_H-768_A-12.

Then run run.sh (I have committed a new version) to train the mask layer, note that --init_checkpoint= should point to the path ./uncased_L-12_H-768_A-12_SST-2_f in the second step.

Since this part of the training time is long, you can set fine_tune_epoch to 1 for testing.

Run remove_mask.sh, the --checkpoint flag should point to the output_dir in the third step.

When finish, --output_folder_dir will generate bert_model_f, a new bert_config.json with pruned_layers_dim property, and info.txt.

I will find time to update the README and code~

Okay, thanks @Holldean , I will try this soon and get back to you :)

Hello @Holldean
Another quick question. I downloaded the model from your repository instead (the download model here) link,
The .ckpt.meta file was missing from the download, so I copied the meta file from the _f stage (just in case that works). But I get the following error if i run the following code:
CODE
import tensorflow as tf
sess = tf.InteractiveSession()
saver = tf.train.import_meta_graph('bert_model_f.ckpt.meta')
saver.restore(sess, 'bert_model.cpkt')
g = tf.get_default_graph()

ERROR
>>> saver.restore(sess, 'bert_model.cpkt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xyz/.local/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1282, in restore
    checkpoint_prefix)
ValueError: The passed save_path is not a valid checkpoint: bert_model.cpkt

Any clue how I can inspect the weights of the checkpoint you have given?

bert_model_f is just an intermediate model, not the pruned one. Also, the structure of final model from different training results are not the same.
A short advice, you can try to load the download model using my code first, and then export the .ckpt.meta file. That might work

Hi @akhauriyash , here I use bert_config.json and vocab.txt files to define the model, just run /flop/run_classifier.py to load this model and evaluate on some tasks. When loading model successfully, you can modify my code to get the .ckpt.meta file.

I dont think I need to modify the code to get the .ckpt.meta file. It is being automatically saved as this is printed at the end of running run.sh (I keep factorized=true)
I0316 03:47:32.583339<PHONE_NUMBER>20896 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /data1/bert_pruning_attempt/BERT-Pruning/SST-2_Pruning/uncased_L-12_H-768_A-12_SST-2_f/2021-03-16-03:46:41/model.ckpt.

I take the .ckpt.meta file from here, put it in the SST-2_85%_t=0.8 folder (after downloading from your repository)
Then I run the following code:
import tensorflow as tf
sess=tf.Session()
saver = tf.train.import_meta_graph('bert_model.ckpt.meta')
saver.restore(sess, 'bert_model.ckpt')

>>> saver = tf.train.import_meta_graph('bert_model.ckpt.meta')
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Can you please upload the whole model including the .ckpt.meta file in the Download model here section of your readme?
I am not too familiar with tensorflow but why did you not include the meta file in your model download?
Thank you for the help!

I dont think I need to modify the code to get the .ckpt.meta file. It is being automatically saved as this is printed at the end of running run.sh (I keep factorized=true)
I0316 03:47:32.583339<PHONE_NUMBER>20896 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /data1/bert_pruning_attempt/BERT-Pruning/SST-2_Pruning/uncased_L-12_H-768_A-12_SST-2_f/2021-03-16-03:46:41/model.ckpt.

I take the .ckpt.meta file from here, put it in the SST-2_85%_t=0.8 folder (after downloading from your repository)
Then I run the following code:
import tensorflow as tf
sess=tf.Session()
saver = tf.train.import_meta_graph('bert_model.ckpt.meta')
saver.restore(sess, 'bert_model.ckpt')

>>> saver = tf.train.import_meta_graph('bert_model.ckpt.meta')
INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Can you please upload the whole model including the .ckpt.meta file in the Download model here section of your readme?
I am not too familiar with tensorflow but why did you not include the meta file in your model download?
Thank you for the help!

Sorry @akhauriyash , since I'm a bit busy lately, I have no time to update my README.md. Actually I am also a novice to Tensorflow. Before you mention the .ckpt.meta file, I don’t even know its usefulness 🤣, so I really appreciate your comment.
Here is a simple bash to run my model.
You can find .ckpt.meta under the directory .../BERT-Pruning/SST-2_Pruning/SST-2_85%_t=0.8. In my case, there are several files in my path BERT-Pruning/SST-2_Pruning/SST-2_85%_t=0.8/2021-03-18-12:20:48:
checkpoint  eval.tf_record  events.out.tfevents.1616041273.amax  graph.pbtxt  model.ckpt-0.data-00000-of-00001  model.ckpt-0.index  model.ckpt-0.meta  train.tf_record

Then I run:
import tensorflow as tf
sess = tf.Session()
saver = tf.train.import_meta_graph('model.ckpt-0.meta')
saver.restore(sess, 'model.ckpt-0')

No error 👍! But I have not verified its performance on SST-2 dataset.
And my question is, if the model is loaded in this way, how do we feed the data to the model?
BTW, you can evaluate that downloaded model on SST-2 without training, by simply setting the flag --do_train=false and --do_eval=true.

Thank you for your help!
Really appreciate it. :)


Subtraction using number 257 and above on the first number input results to incorrect output if second number input is lower

Describe the bug
Subtraction using numbers starting from 257 and above has an error if the first number input (257 - 999) is larger than the second number input (0 - 256).
To Reproduce
Steps to reproduce the behavior:

Start using numbers from 257 to 999 ( try 257)
Subtract it to a lower value (try 1)
Output is 0

Expected behavior
257 - 1 = 256

Fix at 6e44e1553414f22a5069e37c8ba9b344e41bb3af


UNION could not convert type "varchar" to tinyint

Originally posted by @psrinivasa2 in https://github.com/wiltondb/wiltondb/issues/53#issuecomment-2434162313

@psrinivasa2
This should be already fixed in the latest dev version,  please try running the following on Postgres connection to apply the fix to the 11.16.0.rc6 version:
CREATE OR REPLACE FUNCTION sys.varchar2tinyint(sys.VARCHAR)
RETURNS sys.TINYINT
AS 'babelfishpg_common', 'varchar2tinyint'
LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE;

CREATE CAST (sys.VARCHAR AS sys.TINYINT)
WITH FUNCTION sys.varchar2tinyint(sys.VARCHAR) AS IMPLICIT;

CREATE OR REPLACE FUNCTION sys.tinyint2varchar(sys.TINYINT)
RETURNS sys.VARCHAR
AS $$
  SELECT $1::SMALLINT;
$$
LANGUAGE SQL IMMUTABLE STRICT PARALLEL SAFE;

CREATE CAST (sys.TINYINT AS sys.VARCHAR)
WITH FUNCTION sys.tinyint2varchar(sys.TINYINT) AS IMPLICIT;

For now I should manually add, In further release you will include this right?

@psrinivasa2
Yes, it this is correct, this fix will be automatically included with newly created DB clusters, and on the existing cluster you will need to run the following after the MSI upgrade:
ALTER EXTENSION "babelfishpg_common" UPDATE TO '3.3.4'

@psrinivasa2

please try running the following on Postgres connection to apply the fix to the 11.16.0.rc6 version:

Sorry, this won't actually run directly in PgAdmin:
ERROR:  cast from type sys."varchar" to type sys.tinyint already exists

This must go through the ALTER EXTENSION instead, I will prepare the update script and will post it here in a few minutes.

@psrinivasa2
Please save this attached file babelfishpg_common--3.3.3--3.3.4.sql.txt on path:
C:\Program Files\WiltonDB Software\wiltondb3.3\share\extension\babelfishpg_common--3.3.3--3.3.4.sql

(note, remove the .txt suffix) and then run the following in PgAdmin in wilton DB:
ALTER EXTENSION "babelfishpg_common" UPDATE TO '3.3.4'

You can check the resulting version with:
select * from pg_extension

The fix to this is included in 12.16.1 update.

Thanks @staticlibs

This can be closed


from_json/to_json issue#40

From_json/to_json: read and write nonce and code fields as hexstrings
Default initializer added for Account fields
Test added checking that from_json/to_json are mutually inverse

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: unboxedtype sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Just to ensure: everything is considered done regarding this issue, and PR will be merged into master some time later?

Just to ensure: everything is considered done regarding this issue, and PR will be merged into master some time later?

Yes, sorry for the delay - this is merged now 👍


Strange problem with scaling

Hi @ospfranco , as you can see I am testing in all ways sol.
I tried to resize it as seen from the video and I get this result.
The problem is that I can't get back to its original shape or try to shrink everything after enlarging the app.
https://user-images.githubusercontent.com/20476002/159161108-ede818b0-c0f0-4864-9dd1-97c07d563311.mov

the window is not meant to be resizable, feel free to open a PR though, this is minor

@ospfranco : Would it be possible to make sure that it is not resized?

ok, should be fixed


data package installed

Because of this line we are installing the data module as a package called data. Problems with this:

this is a bit of a generic name for a package!
constants like this one no longer work: they refer to .json files that aren't installed

When I did https://github.com/CUNY-CL/wikipron/pull/479, I was on the fence about whether to move the wikipron package under src/, which is something I've already done for all other projects I maintain (in which case we'd have where = [ "src" ] for the line you've singled out). I didn't do it in order not to accidentally break things that I might not be aware of, but I guess it's time to reconsider this...

@jacksonllee I like that idea, will assign this to myself and do now.


http://www.springframework.org/schema/security/spring-security-oauth2.xsd is outdated

Attributes check-token-endpoint-url and check-token-enabled are missing in authorization-server element.

Thanks for the report. This is now fixed


Add PluginDependenciesSpec.embeddedKotlin function

Context
If you for example want to apply the plugin.serialization plugin of the embedded Kotlin version,
you currently have to do kotlin("plugin.serialization") version embeddedKotlinVersion.
Imho it would be nice being able to do like in dependencies { ... } just embeddedKotlin("plugin.serialization").
This PR adds this function.
As I did not find any tests for DependencyHandler.embeddedKotlin either for which I searched to find the
proper place to add tests for this, I assume and hope that also for this function no dedicated tests are necessary.
If you consider merging this PR but want to have tests, I'd welcome some hints on where and how to properly test this. :-)
Contributor Checklist

[x] Review Contribution Guidelines
[x] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[x] Make sure all contributed code can be distributed under the terms of the Apache License 2.0, e.g. the code was written by yourself or the original code is licensed under a license compatible to Apache License 2.0.
[x] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
n/a Provide integration tests (under <subproject>/src/integTest) to verify changes from a user perspective
n/a Provide unit tests (under <subproject>/src/test) to verify logic
n/a Update User Guide, DSL Reference, and Javadoc for public-facing changes
[ ] Ensure that tests pass sanity check: ./gradlew sanityCheck
[x] Ensure that tests pass locally: ./gradlew <changed-subproject>:quickTest

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation

The santiy check is failing as the API compare task does not find the generated class file.
I guess the root cause for this is the same as #21114 and is unrelated to my changes.

As the API compare task was reworked to also find generated files, the PR is all green now. :-)

Hi Björn,
This looks like a sensible addition we want to merge.
Could you please add a unit test in PluginDependenciesSpecScopeTest?

Thanks for telling me where.
Would probably have added a test earlier if I had known the proper location. :-)
It's available now.

@bot-gradle test this

I've triggered the following builds for you:

PullRequestFeedback build

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

I've triggered a build for you. Click here to see all failures if there's any.

Pre-tested commit build failed. Click here to see all build failures.

Your PR is queued. See the queue page for details.

I've triggered a build for you. Click here to see all failures if there's any.

Thank you @Vampire, this will be in Gradle 8.3


Topic/fix kr to buf

These changes are based on the suggestions of @LFSaw from this issue: https://github.com/flucoma/flucoma-sc/issues/51 and PR: https://github.com/flucoma/flucoma-sc/pull/52.
They modify FluidBufToKr and FluidKrToBuf to allow for reading/writing subvectors to/from buffers and control rate streams. This is an extension of some stuff that is already in the dev branch--namely the ability to pass FluidBufToKr a buffer index and specify a numFrames, rather than the need to pass it a buffer.
I think I've done a good amount of testing and feel comfortable putting it in the nightlies and if no bugs arise before beta5 it can also go there.
I'm looking for feedback on any interface/naming things I might have missed or could be improved (and anything else you see!).

2. There was no example of the offset code to test so I had to bake my own... and I thought I found a bug, then I remembered I was wrong and that you gave a good error and explanation in why numBuf is needed... then there is no test code to test -1 with startframe only, so I made my own again and it works.

Cool thanks. I usually put my tests in the "test" folder. https://github.com/flucoma/flucoma-sc/blob/topic/fix-KrToBuf/test/FluidBufToKr test.scd Is this the one you were using? I try to not have the examples in the help file be "test-code-y".

I was requested to put the test code in the PR before (in this thread) but I'll take it wherever.