Frictionless Data Lib: a standard pattern for our data libraries

I wrote up the Frictionless Data lib pattern here:
https://okfnlabs.org/blog/2018/02/15/design-pattern-for-a-core-data-library.html
The intention is that we move away from "datapackage-xx" style libraries to more of a Frictionless data lib pattern in the future as it is more oriented to what people actually need / want / think about.
We already have a lot of this in python with tabulator and datapackage-py.
Tasks

[ ] Review existing libraries and do a gap analysis
[ ] Short intro / guide to writing a Frictionless Data lib
[ ] Trial for one language (e.g. python or javascript)
[ ] Decide whether worth doing generally as recommended default pattern for libs

Analysis
Appendix: Existing Library APIs
Table Schema JS
https://github.com/frictionlessdata/tableschema-js
Table.load

Table.iter

Table.read

Infer a schema
const tableschema = require("tableschema")

(async () => {
  const source = 'path/to/data/file'
  const schema = await tableschema.infer(source)
  console.log(schema)
})()

More about "infer" - https://github.com/frictionlessdata/tableschema-js#async-infersource-headers1-options
Read a data file
const {Table} = require("tableschema")

(async () => {
  const source = 'path/to/data/file'
  const table = await Table.load(source)
  const data = await table.read()
  console.log(data)
})()

Python Data Package Library
Pandas
>>> import datapackage

>>> data_url = 'http://data.okfn.org/data/core/country-list/datapackage.json'
>>> storage = datapackage.push_datapackage(data_url, 'pandas')

>>> storage.buckets
['data___data']

>>> type(storage['data___data'])
<class 'pandas.core.frame.DataFrame'>

>>> storage['data___data'].head()

Why is it called storage.buckets
Why different from http://okfnlabs.org/blog/2016/08/01/using-data-packages-with-pandas.html

Why not:
import datapackage
import datapackage2pandas

dataset = datapackage.Dataset(url)

# pandas dataframe ...
dataframe = datapacakge2pandas.dump(dataset)

Original Motivation for the article https://okfnlabs.org/blog/2018/02/15/design-pattern-for-a-core-data-library.html
Rufus original issue: https://github.com/frictionlessdata/tableschema-js/issues/78
With a few tweaks we can get something really nice as an interface and we can centralize it in one lib (at the moment, whilst the libs are nicely factored you do have to look into too many places -- e.g. this issue ends up being here but relating to 3 other libs).
First I must emphasize a key point:
People don't care about Data Packages / Resources they care about opening a data file and doing something with it

Data Packages / Resources come up because they are a nicely agreed metadata structure for all the stuff that comes up in the background when you do that.

Put crudely: Most people are doing stuff with a file (or dataset), and they want to grab it and read it preferably in a structured way e.g. as a row iterator -- sometimes inferring or specifying stuff a long the way e.g. encoding, formatting, field types.
=> Our job is to help users to open that file (or dataset) and stream it as quickly as possible.
Comments
The above was done for JS but could be common to python and any other language too (tabulator-py Stream would need to have a stream() method rather than open() or similar but that's minor).
At the moment we have all the ingredients for this but we don't provide this in one place often e.g.

https://github.com/frictionlessdata/datapackage-js#resource - Resource is missing the stream methods and we have to call Table and then iter to get objects (and why don't we support Node Stream API for the object streams?)
Python: you have to work your way across tabulator-py, tableschema-py and datapackage-py to put the underlying concepts and approach together.
General point: I'm wondering if we almost want to leave the Data Package stuff to one side and present this front and center - this data interface is what people really want and the Data Package and Data Resource are just the descriptor structures that you happen to need. (This is more a presentation aspect but its important e.g. you'd start examples not loading a datapackage.json but loading a file path)

Asides

More and more think the profile stuff should be a separate library outside of core. 90% of the time I don't want and I don't want to know about it and I don't want to validate before I start -- I just want to load the data! If I want it i can import/require it separately and use it separately to test whatever i have once i've loaded it.

As an example of how this gets in the way: two of the first four items in the first example for datapackage-py deal with validation and the fairly obscure issue of schemas not in the local cache https://github.com/frictionlessdata/datapackage-py

I always for making things simpler to use. Just like, in Python, when using pandas, you can do something like this:
import pandas as pd
df = pd.read_csv('https://example.com/table.csv') # get a Pandas DataFrame

and in the background Pandas automagically deals with things like requests, we could have something like
import frictionless as fd
tables = fd.read_dataset('https://example.com/datapackage.json')
df = tables['mytable'] # get a Pandas DataFrame

This could be made with just a simple wrapper around tableschema-py and datapackage-py. It would be even easier to use than it already is (see this tutorial), which in turn would encourage more people to use it.

@lwinfree
That's a good example of why we need to improve the docs. This code is deprecated like 3 years ago and still popping out in different places:
>>> import datapackage

>>> data_url = 'http://data.okfn.org/data/core/country-list/datapackage.json'
>>> storage = datapackage.push_datapackage(data_url, 'pandas')

>>> storage.buckets
['data___data']

>>> type(storage['data___data'])
<class 'pandas.core.frame.DataFrame'>

>>> storage['data___data'].head()

@augusto-herrmann
The modern API is:
from datapackage import Package

# Pandas
package = Package(''https://example.com/datapackage.json'')
storage = package.save(storage='pandas')
# now pandas dataframe is storage['country_codes']

# SQL
package = Package(''https://example.com/datapackage.json'')
storage = package.save(storage=sqlalchemy.create_engine('DATABASE_URL'))
# now it's in your database

# The same for:
# BigQuery
# Elasticsearch
# SPSS
# ckan-datastore
# etc

TBH I don't see why it's more complex than your example :smiley:
The problem that project is by definition is really general. Consider this snippet,
import frictionless as fd
tables = fd.read_dataset('https://example.com/datapackage.json')
df = tables['mytable'] # get a Pandas DataFrame

But why e.g. it exports the dataset to Pandas instead of SQL or not reading into memory. For Pandas user it's obvious but it's not for e.g. BigQuery user. And it's not needed if you just want to play with it in Python. So we need to add flags to this function etc. Then think about the terminology read/dataset/etc. The devil in the details as usual
I agree we can have something like:
dfs = fd.package_to_pandas('datapackage.json')

But my honest opinion that proper documentation is more important

@roll
I completely agree with you, Евгени.
Having up to date documentation is essential. And even though Pandas is so popular now, it's a completely valid argument to leave the lib non opinionated and let people use and specify explicitly the tool they want to use.
Do you think cutting a step like this make sense, then?
import frictionless as fd

tables = fd.read_dataset('https://example.com/datapackage.json', storage='pandas')
df = tables['mytable'] # get a Pandas DataFrame

classes = fd.read_dataset('https://example.com/datapackage.json', storage=sqlalchemy.create_engine('DATABASE_URL'))
q = session.query(classes['mytable']) # query with SQL Alchemy

BTW, I think it's a bit inconsistent that for Pandas we use a string storage='pandas' and for SQL Alchemy we pass an object as parameter. Perhaps it should be storage=pd.DataFrame instead?

@augusto-herrmann

BTW, I think it's a bit inconsistent that for Pandas we use a string storage='pandas' and for SQL Alchemy we pass an object as parameter. Perhaps it should be storage=pd.DataFrame instead?

You're absolutely right :smiley: There is a typo:
# SQL
package = Package('https://example.com/datapackage.json')
storage = package.save(storage='sq', engine=sqlalchemy.create_engine('DATABASE_URL'), **opther_sql_options)

Regarding some umbrella library, I think it makes sense to think about what can we do using tabulator/tableschema/datapackage/goodtables as building blocks. We just need to think it thoroughly because another layer of functionality means another level of documentation, propagating new features  etc

DUPLICATE. Closing this in favour of the RFC https://github.com/frictionlessdata/project/blob/master/rfcs/0003-library-implementation-reference.md - let's do PRs and discussion there.

Hi @rufuspollock do you have some guidelines already created for how you want people to interact with the RFC documents? I'm worried that people might not comment/do a PR/contribute to them since it isn't as easy as writing a comment in an issue. Maybe you could write a sentence or two at the top of the RFCs telling people how to contribute? (like, please open a PR...etc etc..)

@lwinfree first to emphasize that people can still open issues - in fact that is the best way to start. We would only move stuff to RFCs only when needed (and it is clear it is substantive, longer-term type of thing).
I also agree on having some info on RFCs and have created #510

Awesome thanks @rufuspollock !

FIXED. we have agreed this as part of https://github.com/frictionlessdata/project/blob/master/rfcs/0006-software-structure.md and marked https://github.com/frictionlessdata/project/blob/master/rfcs/0005-migrate-libraries-to-frictionless-data-lib-pattern.md as approved.


IP from  curl ipv4.icanhazip.com not working

Entered the IP (provider AT&T) and got the Error popup. Any suggestions?

Plenty of answers in 598...


Update get-status-badge.md

The way to find the status badge was changed so step 2 needed to be updated

@srednakun : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

Fixed in https://github.com/MicrosoftDocs/azure-devops-docs/pull/11962
Working on another commit with a credit!


Google-Auth(Google Identity) Blank popup

Google Identity
Django

My Google Auth is Stuck in the popup auth flow. The one-tap authentication works just fine but not the button. I click on it, the popup opens but that's it,  it remains there blank with no progress, never closes or anything.

I have all the domains, localhost added in Authorized redirect URIs and Redirects.  Any help is appreciated.

At this point the error I am getting in FIREFOX:

Uncaught TypeError: window.opener is null li https://ssl.gstatic.com//gsi//js/k=gsi.gsi.en.i....O/am=wg/d=1/rs=AF0...w/m=credential_page_library:291 mi https://ssl.gstatic.com//gsi//js/k=gsi.gsi.en.i.....O/am=wg/d=1/rs=AF0...w/m=credential_page_library:29

And in CHROME:

Uncaught TypeError: Cannot read properties of null (reading 'postMessage') at li (m=credential_page_library:291:151)

The problem seems to be this picker_popup in this line of code:
a,c);return b.g};li=function(a,b){"picker_popup"===a.i?window.opener.postMessage(b,a.j):window.parent.postMessage(b,a.j)};_.ni=function(a,b){var c=new _.Nd;a.g?(a.g.postMessage(b),c.resolve()):c.reject("Illegal state: try to send message before message channel set up.");return c.g};

From the error message it looked like the popup window did not have the access to the opener page. Did you set the Cross-Origin-Opener-Policy to same-origin? If that is the case, you need to set it to same-origin-allow-popups to allow the popup window to talk to its opener.

Just kinda happened upon this, and I think I know how to fix it, so I'll leave the answer here. The Cross-Origin-Opener-Policy is likely same-origin, since that is the default for Django sites. To set it to same-origin-allow-popups on Django v4, set SECURE_CROSS_ORIGIN_OPENER_POLICY to same-origin-allow-popups in django settings. Hope it helps you or whoever also finds this!

Glad to hear that you found a solution. Please post any more authentication/authorization questions on Stack Overflow with the google-oauth tag. We're unable to provide support for it here.

does anyone know how to solve this in next js? i've been experiencing the same issue.
the dialog box shows to signin into google if I have no accounts already signed in but does not show anything if I have already signed into any gmail account.

same-origin-allow-popups

try this, it helped me https://stackoverflow.com/a/68469319

Had a similar issue when using VS2022 17.3.5, where the pop-up for Google sign-in would be blank if already signed in to a Google account. Fixed it by disabling JavaScript debugging from Tools > Options.

Just kinda happened upon this, and I think I know how to fix it, so I'll leave the answer here. The Cross-Origin-Opener-Policy is likely same-origin, since that is the default for Django sites. To set it to same-origin-allow-popups on Django v4, set SECURE_CROSS_ORIGIN_OPENER_POLICY to same-origin-allow-popups in django settings. Hope it helps you or whoever also finds this!

It is solid

I also got the same problem in my wordpress login can anyone help

Hi @varad615 did you found the solution of this?

 Please show us mere mortals mercy by allowing a glimpse of your infinite knowledge used for solving this problem.

I recently faced this problem as well when upgrading our Django from 3.2 to 4.2 and setting SECURE_CROSS_ORIGIN_OPENER_POLICY to same-origin-allow-popups didn't help, but setting it to unsafe-none solved the issue.

Is this solution good?
unsafe-none

Settings something as unsafe-none doesn't sound, well, safe?

@Megatronicus I thought the same, but again, it's the only documented value for that setting that did the job in my case. I don't know, but here's how I understand it: it's an additional layer of security introduced to django 4 and Django docs explicitly mention that it can be set to None, which is of course less secure than unsafe-none (c.f. https://docs.djangoproject.com/en/4.2/ref/middleware/#cross-origin-opener-policy). Plus, in the Mozilla HTTP docs it is stated that it's the default browser behavior.
What are you thoughts on this?

You need to able the origin in the user account where you generate the clientId

Thank you, it works!

Hi @varad615 did you found the solution of this?

Ya
while creating token ig i added
Authorized JavaScript origins
and it started working

Good good

does anyone know how to solve this in next js? i've been experiencing the same issue. the dialog box shows to signin into google if I have no accounts already signed in but does not show anything if I have already signed into any gmail account.

try this, it helped me https://stackoverflow.com/a/68469319

It really helped thanks mate


Improvments on custom ops

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 84.75%. Comparing base (0991415) to head (2f1576f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #176   +/-   ##
=======================================
  Coverage   84.75%   84.75%
=======================================
  Files          58       58
  Lines        7051     7051
=======================================
  Hits         5976     5976
  Misses       1075     1075

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Peek] Plugin pattern to enable any file type previewing

Create an internal plugin system that enables developers to easily interface plugins that render specific file types.

Support bitmap rendering via the Image control
Support web content rendering via WebView2

Does a plugin management like in PT Run makes sense?

A plugin for drives showing its name, icon, type/format, total space, free space, used space and a space graphic would be nice.


bug in jax-metal or in equinox?

I've used equinox on Linux boxes and standard Mac installs with no problems. However, I just installed jax-metal to use the AMD GPU on my desktop and I get the following unhashable type list error right at the import equinox as eqx line. Not quite sure how or why a list would not give this error  on a Mac CPU/Linux GPU but only on jax-metal.  Any suggestions appreciated. Thank you for all the work you've done on this framework. Big fan of simplicity!
TypeError                                 Traceback (most recent call last)
Cell In[1], line 10
7 import numpy as np
8 import optax  # https://github.com/deepmind/optax
---> 10 import equinox as eqx
File ~/jax-metal/lib/python3.9/site-packages/equinox/init.py:3
1 import importlib.metadata
----> 3 from . import internal as internal, nn as nn
4 from ._ad import (
5     filter_closure_convert as filter_closure_convert,
6     filter_custom_jvp as filter_custom_jvp,
(...)
11     filter_vjp as filter_vjp,
12 )
13 from ._better_abstract import (
14     AbstractClassVar as AbstractClassVar,
15     AbstractVar as AbstractVar,
16 )
File ~/jax-metal/lib/python3.9/site-packages/equinox/internal/init.py:5
1 from .._better_abstract import (
2     AbstractClassVar as AbstractClassVar,
3     AbstractVar as AbstractVar,
4 )
----> 5 from .._compile_utils import (
6     hashable_combine as hashable_combine,
7     hashable_partition as hashable_partition,
8 )
9 from .._doc_utils import doc_remove_args as doc_remove_args, doc_repr as doc_repr
10 from .._enum import Enumeration as Enumeration
File ~/jax-metal/lib/python3.9/site-packages/equinox/_compile_utils.py:12
9 from jaxtyping import PyTree
11 from ._caches import internal_caches
---> 12 from ._module import Module
15 def hashable_filter(pytree: PyTree, filter_fn: Callable):
16     leaves = jtu.tree_leaves(pytree)
File ~/jax-metal/lib/python3.9/site-packages/equinox/_module.py:17
15 from ._doc_utils import doc_repr
16 from ._pretty_print import tree_pformat
---> 17 from ._tree import tree_equal
20 _P = ParamSpec("_P")
21 _T = TypeVar("_T")
File ~/jax-metal/lib/python3.9/site-packages/equinox/_tree.py:10
8 from ._custom_types import sentinel
9 from ._doc_utils import doc_repr
---> 10 from ._filters import is_array
13 if TYPE_CHECKING:
14     _Node = Any
File ~/jax-metal/lib/python3.9/site-packages/equinox/_filters.py:11
7 import numpy as np
8 from jaxtyping import PyTree
---> 11 AxisSpec = Union[bool, Callable[[Any], bool]]
14 #
15 # Filter functions
16 #
19 def is_array(element: Any) -> bool:
File /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/typing.py:243, in _tp_cache..inner(*args, **kwds)
241 except TypeError:
242     pass  # All real errors (not unhashable args) are raised below.
--> 243 return func(*args, **kwds)
File /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/typing.py:316, in _SpecialForm.getitem(self, parameters)
314 @_tp_cache
315 def getitem(self, parameters):
--> 316     return self._getitem(self, parameters)
File /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/typing.py:421, in Union(self, parameters)
419 msg = "Union[arg, ...]: each arg must be a type."
420 parameters = tuple(_type_check(p, msg) for p in parameters)
--> 421 parameters = _remove_dups_flatten(parameters)
422 if len(parameters) == 1:
423     return parameters[0]
File /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/typing.py:215, in _remove_dups_flatten(parameters)
213         params.append(p)
214 # Weed out strict duplicates, preserving the first of each occurrence.
--> 215 all_params = set(params)
216 if len(all_params) < len(params):
217     new_params = []
TypeError: unhashable type: 'list'

I think I recognise this as a bug with Python 3.9.. Try updating your Python version to 3.9. and I think should be resolved.

Thank you for that very quick response. That was it.


sliding by using finger on mobile is not working properly

Below is the config
public carouselOne: NguCarousel;
this.carouselOne = {
grid: {xs: 1, sm: 1, md: 1, lg: 1, all: 0},
slide: 1,
point: {
visible: false
},
touch: true,
loop: true,
easing: 'ease',
}

Try this:
In app.module.ts add these lines:
import * as Hammer from 'hammerjs';
  import { HammerGestureConfig, HAMMER_GESTURE_CONFIG } from '@angular/platform-browser';

  export class HammerConfig extends HammerGestureConfig {
    overrides = <any>{
      'swipe': { direction: Hammer.DIRECTION_ALL }
    };
  }

@NgModule({
    declarations: [
      AppComponent
    ],
    imports: [
      BrowserModule
    ],
    providers: [
      {
        provide: HAMMER_GESTURE_CONFIG,
        useClass: HammerConfig
      }
    ],
    bootstrap: [AppComponent]
  })
  export class AppModule { }

ERROR in Error during template compile of 'HammerCo
nfig'
Class HammerConfig in D:/project/b2go/c2go-fronte
nd/src/app/app.module.ts extends from a Injectable
in another compilation unit without duplicating the
decorator
Please add a Injectable or Pipe or Directive or
Component or NgModule decorator to the class.

Can you please post your code here?

I get the same error when trying to add this code. Also carousels are not working on IOS, both on the demo site, and on our build.

Same issue here!!! Swipe is working but is not working properly. Sometimes goes ok, sometimes goes back to the previous item. Buggy swipe.

Guys, please refer to this thread for my comments. I hope it should work now :)
https://github.com/sheikalthaf/ngu-carousel/issues/65

@mhosman @stephenmacnaughtan @shansubra Please check the latest build. Thanks


Quantization dequanitze flag

Hi, while looking at the quantization function I found that there is a dequantize flag.
But as I think, the network should be forwarded to the dequantized value.
What's the purpose of the dequantize flag?
In other words, what's the difference between dequantized and non-dequantized cases?
Thank you.

I'm not sure what you mean by "the network should be forwarded to the dequantized value", but in any case, this is a utility function called from many places:
In some cases we want to just get the effect of quantization but still let the model run with float tensors (aka "fake-quantization"). This is common in quantization-aware training scenarios. In these cases we'll set dequantize=True.
In other cases we want to quantize a tensor and keep working on the quantized values. This happens in post-training quantization.

That's what I exactly wanted to know.
Thank you so much


nativescript-dev-webpack

Hi, in the package.json file, the devDependencies "nativescript-dev-webpack" is assigned to "github:nativescript/nativescript-dev-webpack#sis0k0/platform-host" and that cause a problem when you run npm i
so what version of "nativescript-dev-webpack" we should use?
Thanks
Marwan

Hey @iMarwan-k,
the sis0k0/platform-host branch referenced in the package.json file was merged and released in<EMAIL_ADDRESS>which version you could use.
It will be updated in here shortly.


[DOCFIX] Update Chinese index.md.

For consistency with the English docs.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/12449/
Test PASSed.


delay eureka registration problem

delay eureka registration problem
Hi,
i want to delay eureka registration,make sure my application are ready to provide services;
and i set set the initial status to out of service like this:
eureka.instance.initialStatus=OUT_OF_SERVICE

and enable health check like this:(application.yml)
eureka:
  client:
    healthcheck:
      enabled: true

and i change it to UP when ContextStartedEvent come, it works
but when i refresh my config center(spring config), post /bus/refresh ,my instance status changed to  OUT_OF_SERVICE, it just same with eureka.instance.initialStatus as i set, i know /refresh will cause all the beans to be recreating, it seem use eureka.instance.initialStatus to rebuild java beans ;but i expect my application status is UP not OUT_OF_SERVICE
i dont know how to do ,can somebody give some tips, thx
version
<parent>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-parent</artifactId>
   <version>1.5.7.RELEASE</version>
   <relativePath />
</parent>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-dependencies</artifactId>
    <version>Edgware.SR3</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

Dont you get the ContextStartedEvent when refreshing?

hi ryan，thanks for your reply; no, i can not get the ContextStartedEvent when refreshing ,but EnvironmentChangeEvent, and i change instance status in this event like this:,but i found this event before discovery beans reload;
@Service
public class EurekaRegisterAfterRefreshListener implements ApplicationListener<EnvironmentChangeEvent> {

    private static final Logger LOGGER = LoggerFactory.getLogger(EurekaRegisterAfterRefreshListener.class);

    @Override
    public void onApplicationEvent(EnvironmentChangeEvent event) {

        LOGGER.info(" ------------------->>> hell refresh");

    }

trace:
2018-10-31 15:08:34.776  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] c.s.e.EurekaRegisterAfterRefreshListener : ------------------->>> hell refresh
2018-10-31 15:08:39.181  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] com.netflix.discovery.DiscoveryClient    : Shutting down DiscoveryClient ...
2018-10-31 15:08:39.182  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] o.s.c.n.eureka.InstanceInfoFactory       : Setting initial instance status as: STARTING
2018-10-31 15:08:39.185  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] com.netflix.discovery.DiscoveryClient    : Unregistering ...
2018-10-31 15:08:39.198  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_MG-ZUUL/<IP_ADDRESS>:18001 - deregister  status: 200
2018-10-31 15:08:39.230  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] com.netflix.discovery.DiscoveryClient    : Completed shut down of DiscoveryClient
2018-10-31 15:08:39.235  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] com.netflix.discovery.DiscoveryClient    : Initializing Eureka in region us-east-1
2018-10-31 15:08:39.239  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] c.n.d.provider.DiscoveryJerseyProvider   : Using JSON encoding codec LegacyJacksonJson
2018-10-31 15:08:39.239  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] c.n.d.provider.DiscoveryJerseyProvider   : Using JSON decoding codec LegacyJacksonJson
2018-10-31 15:08:39.239  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] c.n.d.provider.DiscoveryJerseyProvider   : Using XML encoding codec XStreamXml
2018-10-31 15:08:39.240  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] c.n.d.provider.DiscoveryJerseyProvider   : Using XML decoding codec XStreamXml
2018-10-31 15:08:39.499  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2018-10-31 15:08:39.503  INFO [bootstrap,58cb01bc0b192339,58cb01bc0b192339,true] 71874 --- [           -L-1] com.netflix.discovery.DiscoveryClient    : Disable delta property : false
im thinking Is there a way a way not recreating eureka instance beans; because i just want refreh some specific properties？or shoud i found some event that come just after the DiscoveryClient recreated ? thx

RefreshScopeRefreshedEvent works，but i worry about that the period of refreshing ,the status change to OUT_OF_SERVICE, in this moment, other eureka client think this instance is out of service, and no service provide services,can This happen yet？

We recently added a post registration event, maybe this might be useful to listen on and change the status
http://cloud.spring.io/spring-cloud-static/Finchley.SR2/single/spring-cloud.html#_serviceregistry_auto_registration_events

Although I am not clear why the RefreshScopeRefreshedEvent doesnt work


Week 19 Review Ticket

Kaylee Hou
Period 4 Yeung

Problems + Resolved
CB Grading
Database Video
API video
- endpoints frontend

1 API
1 DB
.95 CPT


deps can be installed via pip or apt, for example

Ubuntu has dependencies in its apt repo, so give an example how to install them.

@joe-bowman, can I just merge?

Yeah, always happy for trivial stuff (esp docs) to be merged without review.


Cythonize GIL error

Error compiling Cython file:
...
cdef inline DTYPE_t dist(self, DTYPE_t* x1, DTYPE_t* x2,
                         ITYPE_t size) except -1 with gil:
    cdef np.ndarray x1arr
    cdef np.ndarray x2arr
    with gil:
        ^

sklearn/neighbors/dist_metrics.pyx:1111:13: Trying to acquire the GIL while it is already held.
Traceback (most recent call last):
File "/home/rbrault/OCRF/build_tools/cythonize.py", line 198, in
main(root_dir_arg)
File "/home/rbrault/OCRF/build_tools/cythonize.py", line 190, in main
check_and_cythonize(root_dir)
File "/home/rbrault/OCRF/build_tools/cythonize.py", line 182, in check_and_cythonize
cythonize_if_unchanged(cur_dir, cython_file, gen_file, hashes)
File "/home/rbrault/OCRF/build_tools/cythonize.py", line 156, in cythonize_if_unchanged
cythonize(full_cython_path, full_gen_file_path)
File "/home/rbrault/OCRF/build_tools/cythonize.py", line 75, in cythonize
raise Exception('Cythonizing %s failed' % cython_file)
Exception: Cythonizing sklearn/neighbors/dist_metrics.pyx failed
Traceback (most recent call last):
File "setup.py", line 315, in
setup_package()
File "setup.py", line 299, in setup_package
generate_cython()
File "setup.py", line 185, in generate_cython
raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!
Need investigation.
Linux-4.10.0-40-generic-x86_64-with-debian-stretch-sid
Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 16 2017, 15:28:36)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
NumPy 1.13.3
SciPy 0.19.1
GCC 7.0.1
Related issue:
https://stackoverflow.com/questions/46920490/sklearn-installation-trying-to-acquire-the-gil-while-it-is-already-held

I got the same issue. Thanks for investigating the error.

Related issues:
https://github.com/scikit-learn/scikit-learn/issues/9272
Possible fix:
https://github.com/scikit-learn/scikit-learn/pull/9311

Could fix the issue with the possible fix you mentioned(scikit-learn/scikit-learn#9311). Thanks a lot.

@minoriinoue, thanks for the feedback.

Could you elaborate more on how to fix the bug? It is not clear for me :-(


merlin-locate on case insensitive mac os

I have merlin 2.5.0 on mac os x yosemite 10.10.5. I have the standard installation of mac os, which is case insensitive. I am using merlin from emacs, which is case sensitive.
When I use C-c C-l to locate the definition of a function Module.function located in the file module.ml, merlin looks for the upper case file Module.ml and opens up file module.ml as Module.ml. If I modify and save the file, the name of module.ml changes to Module.ml, leading to problems for the compilation and git.

I pushed some code in master trying to deal with case insentivity.
Can you check if it improves your situation?

To directly try the git version: opam pin merlin --dev-repo

It works, thank you very much.

@let-def ah, that's neat! Thanks for the tip!


orthosnap: error: unrecognized arguments: -ip median_seq_len

Hello,
I am trying to invoke the option for in-paralog selection but it seems it is not recognized
$ orthosnap -f OG0000452.fa -t OG0000452_tree.txt -ip median_seq_len
orthosnap: error: unrecognized arguments: -ip median_seq_len
Also in the $orthosnap -h command there is no such option but it exists in the documentation
$orthosnap -h
       _   _
       | | | |

___  _ | |_| |   ___  ___ _ __   __ _ _ __
/ _ | '| | ' \ / _ / | ' \ / ` | ' \
| () | |  | || | | | () _ \ | | | (| | |) |
_/||   _|| ||_/|/| ||_,| .__/
| |
||
Version: 0.0.1
Citation: Steenwyk et al.
Usage: orthosnap  [optional arguments]
required arguments:
-t, --tree
input tree file in newick format
-f, --fasta
input sequence file in fasta format
optional arguments:
-s, --support
support threshold for bipartition collapsing
default: 80
-o, --occupancy
occupancy threshold for minimum number of tips in orthologous subgroup
default: 50 percent of total number of taxa

Detailed explanation of arguments

-t, --tree

  - input tree file in newick format
  - taxa name and gene name should be separate by a "|" symbol
    For example, "gene_a" from "species_a" should appear in the
    tree as "species_a|gene_a", and "gene_b" from "species_a"
    should appear in the tree as "species_a|gene_b", and so
    on and so forth.

-f, --fasta
- input sequence file in fasta format
- taxa name and gene name should be formatted the same as in
the tree file. Thus, "gene_a" from "species_a" should appear
in the tree as "species_a|gene_a" and so on and so forth.
-s, --support
- support threshold for bipartition collapsing
- all bipartitions will values less than the specified value
will be collapsed. For example, if the support threshold
value is 80, all bipartitions with 79 or less support
will be collapsed.
- default value is 80 and is set for ultrafast bootstrap
approximations. If bipartitions support was evaluated
using standard bootstrap, a common threshold to use is 70.
-o, --occupancy
- occupancy threshold for minimum number of tips in orthologous
subgroup.
- default value is 50 percent of the total number of taxa
- values are rounded to the nearest integer. For example,
if there are 15 taxa, the occupancy threshold will be 8.
What might be the issue?
Thanks
Alex

Hi Alex,
First, thank you very much for checking out OrthoSNAP and for providing a comment. OrthoSNAP and other software would be nothing without a helpful user base—i.e., researchers like you!
Second, the issue stems from what version of OrthoSNAP you are using. The -ip argument was not added until a later version of OrthoSNAP. A changelog of OrthoSNAP is available here: changelog, docs. To update your version of OrthoSNAP, you can use the following command: pip install orthosnap -U.
Thanks again for using OrthoSNAP. You may find some of the other software I have developed useful for your research; here is the link in case you are interested :D https://jlsteenwyk.com/software.html.
best,
Jacob


CLI/Generator issue on SAP Sample service

This is to track and issue in the CLI repo: https://github.com/SAP/cloud-sdk-cli/issues/279
Related to #707

Duplicate of #707


Placeholders: stop upward traversal at ObjectProperty

I'd like input on this one in terms of what's expected. I'm proposing that when placeholders are used in the value of an ObjectProperty, upward traversal will stop at that object property, causing the expression to be transformed essentially in place rather than wrapping the CallExpression further up.
input:
doSomething({
  key: _ + _
})

current:
;(_arg, _arg2) => {
  return doSomething({
    key: _arg + _arg2
  })
}

proposed:
doSomething({
  key: (_arg, _arg2) => {
    return _arg + _arg2
  }
})

The current transformation seems like it allows for too much complexity in my opinion, like in this example:
getNameByIdAsync(id)
  .then(doSomething({
    name: _,
    /*...*/
  }))

OK, some thoughts:

My first flash of intuition, without thinking about it too deeply, is that the current compilation is what I would have expected it to be.

Digging a little more deeply, a JS object is often used as a substitute for named parameters, so these two things should probably do roughly the same thing:

f(_)

f({firstNamedArg: _})

The current compilation technique appears to satisfy this rule of thumb.

OTOH, it's certainly easy to come up with examples where the other behavior is desired as well, like:

dataSet.mapReduce({
  map: _ + 1,
  reduce: _ * _
})

So not an easy call one way or another, it seems whichever choice you make would hamper some relevant use cases.
This is the sort of stuff that made me want to get explicit about where the function boundaries are when I did placeholderArgs for LSC.

Good points, thanks!

So not an easy call one way or another, it seems whichever choice you make would hamper some relevant use cases.

Agreed, I'm pretty split. I'm going to think on this one for a while but will probably end up sticking with the current output. Both can allow for arguably "too complex" constructs, so maybe it's best to stick with the current strategy.

Yep, let's keep the current logic of wrapping the first CallExpression to which the placeholder's an argument. This keeps it consistent in all cases and object methods like this could still be defined outside the object, ie. in these contrived examples:
const map = _ + 1
const reduce = _ * _

dataSet.mapReduce({ map, reduce })

Thanks again for the input @wcjohnson

Stopping in to note this is actually possible now by explicitly using the lift modifier added in v2.1.0:
import { _, lift } from 'param.macro'

dataSet.mapReduce({
  map: lift(_ + 1),
  reduce: lift(_ * _)
})

... which will become:
dataSet.mapReduce({
  map: _arg => {
    return _arg + 1
  },
  reduce: (_arg2, _arg3) => {
    return _arg2 * _arg3
  }
})

lift is simply a dummy function that's removed entirely but essentially acts as signal to the other macros to stop upward traversal.


Add docs for kv:bucket

include usage documentation for kv:bucket upload, sync, and delete

Closed by #514


Add insertCard link to actor tests

Add insertCard link to actor tests
Change-type: patch
Signed-off-by: YingwenChen<EMAIL_ADDRESS>

duplicate PR


Using dictionary format under [meta.concat] fails

Thanks for sharing and maintaining wellmap! I use it extensively but now face a new issue:
Documentation specifies:
"The paths can be specified either as a string, a list, or a dictionary. "
I have this in my TOML file:
[meta.concat]
TRX-0032 = ['../human/TRX-0032_dmem_20240327.toml', '../human/TRX-0032_20231004.toml']
TRX-0033 = ['../human/TRX-0033_dmem_20240327.toml', '../human/TRX-0033_20231004.toml']

Call to wellmap.load on this aggregator TOML file fails at
  File "/Users/me/.pyenv/versions/3.11.6/lib/python3.11/pathlib.py", line 493, in _parse_args
    a = os.fspath(a)
        ^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not dict

Did I miss here something or the documentation is not in line with the actual implementation?
Thanks for any hints on how to get this feature working!

Diving into the code I guess the documentation can be misunderstood, I had a wishful thinking that a dict of lists is also valid. It is not right now, here my mod to file.py from line 440:
        elif isinstance(paths, dict):
            paths1 = []
            for pk, pv in paths.items():
                if isinstance(pv, list):
                    paths1.extend([(pk, x) for x in pv])
                elif isinstance(pv, str):
                    paths1.append(pk, pv)
            paths = paths1

This seems to work though wellmap.show produces somewhat weird results but the data itself is parsed correctly and works well in my downstream analysis.
Happy to produce a pull request - though I do not think my workaround is the best way to extend the functionality here, a recursive solution would be more elegant.

Thanks very much for pointing me towards the test system. I learn here (using pytest extensively myself but not at the magician level as you do). However I got stuck at actually running the tests. I could add a new case in test_config_from_toml.nt but pyproject.toml is not using poetry. While I am pretty happy with poetry (and pycharm), I can imagine reasons you prefer to keep things as they are. If so, please point me (or future contributors to wellmap) to some hints on how to run/add tests. Thank you.


Ingest uncatalogued accession 2535 (~180GB)

For #4183

Failed at the "Assign checksums and file sizes to objects" step:
updatesizeandchecksum_v0.0: INFO      2020-03-03 23:03:45,140  archivematica.mcp.client.updateSizeAndChecksum.call:171  Invoked as updatesizeandchecksum_v0.0 /var/archivematica/sharedDirectory/ --filePath /var/archivematica/sharedDirectory/currentlyProcessing/WT_2535-25cd7f1c-7246-4b84-a84c-64a0f8b30ef6/objects/Documentation Photos, Ben Gilbert for Wellcome_unedited selection/BTG170319144740.jpg --fileUUID 6a6275a3-4fc4-4d8d-9dfd-89ce33a3affa --eventIdentifierUUID b58e2314-64cc-4940-957f-d2f9d05f3cd7 --date 2020-03-03T23:02:54.996344+00:00.
An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.Traceback (most recent call last):
  File "/src/MCPClient/lib/job.py", line 111, in JobContext
    yield
  File "/src/MCPClient/lib/clientScripts/update_size_and_checksum.py", line 182, in call
    args.event_uuid,
  File "/src/MCPClient/lib/clientScripts/update_size_and_checksum.py", line 116, in main
    file_ = File.objects.get(uuid=file_uuid)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 59, in execute
    self.db.validate_no_broken_transaction()
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 327, in validate_no_broken_transaction
    "An error occurred in the current transaction. You can't "
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

It’s got past this step before, so hoping this is just a transient thing?

Ehmagerd it’s done! https://wellcome-ingest-inspector.glitch.me/ingests/d3e9474f-9680-4d83-b214-811afe07db25


Make the component wrapper more robust

What
Improve the component wrapper helper in the following ways.

let it recognise if blank or nil values are passed for certain things. Previously it was possible to pass nil as an ID attribute, at which point the wrapper would error because nil is not a valid HTML ID, rather than simply ignoring it
improve the error messages to include the thing that is causing the error, in order to make troubleshooting issues easier

Why
Frontend had a bit of code where nil was being passed to a component as an ID, which was causing the build to fail. This change should prevent similar errors in the future.
Visual Changes
None.

The only comment I have, which would be a separate issue, is whether we should add another method to check for valid or accepted data attributes (for things like "track-" or "module") but an invalid data attribute is going to to have less impact than an invalid class or ARIA attribute so maybe not necessary.

Thanks @jon-kirwan. Very interesting question! I'd omitted checking this so far because data attributes can be anything, so long as they start with data-, but thinking about it now I realise you could check for some invalid input at least (e.g. with spaces in) so will have a think.

@jon-kirwan I've had a go at validating data attributes - have added an extra commit if you'd like to take a look when you have time, thanks.

Nice one! All looks good to me. I've tested from the component guide and an application (feedback) entering various invalid data attribute combinations (e.g. moDule, mo dule) which are correctly raise an ArgumentError with a warning message: Data attributes (moDule) cannot contain capitals, spaces or colons, or start with 'xml'"


Ship youth court fee

Description of change
Deploy the changes for YCF

This is pending extra QA, do not merge yet until Mary is happy.


Copying text from vim.exe in Windows Terminal removes newlines

Using this program:
https://github.com/vim/vim-win32-installer/releases/download/v8.2.2800/gvim_8.2.2800_x64.zip
If I extract and start vim.exe inside Windows Terminal, I see this:
    version 8.2.2800
by Bram Moolenaar et al.

I have copyOnSelect enabled, so I select this text, then paste into Notepad. Result:
version 8.2.2800                                                    ~                                               by Bram Moolenaar et al.

Notice everything is on a single line. I tried disabling copyOnSelect and using Ctrl+C, but same result. I saw someone say that you have to hold Shift, I tried that and it doesnt work either. I also saw someone mention singleLine, but my setting seems to be correct:
{ "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" },

https://github.com/noborus/ov/issues/81

https://github.com/gdamore/tcell/issues/451

Wait how specifically are you copying? With right click, with ctrl+c, with ctrl+shift+c, with copy on select? Could you share your settings.json?

(this has flavors of #9696 in it)

Wait how specifically are you copying? With right click, with ctrl+c, with ctrl+shift+c, with copy on select?

I tried copy on select, and Ctrl+C

Could you share your settings.json?

{
   "$schema": "https://aka.ms/terminal-profiles-schema",
   "copyOnSelect": true,
   "multiLinePasteWarning": false,
   "profiles": {
      "defaults": {},
      "list": [
         {
            "commandline": "C:/PowerShell/pwsh.exe -NoLogo",
            "fontSize": 11,
            "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}"
         }, {
            "guid": "{b453ae62-4e3d-5e58-b989-0a998ec441b8}",
            "hidden": false,
            "name": "Azure Cloud Shell",
            "source": "Windows.Terminal.Azure"
         }
      ]
   },
   "keybindings": [
      { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" }
   ]
}

https://github.com/walles/moar/issues/51

Looks like this has been a problem for a while, I tried these versions, and they all have the same problem:
v1.8.1444.0
v1.4.3243.0
v1.3.2382.0
v1.2.2022.0
v1.1.1671.0
v1.0.1401.0

https://github.com/microsoft/terminal/issues/174

Closing for lack of interest

note to self:

still repros in 1.16
repros in conhost with quick edit manually enabled
I vaguely remember an issue in the past where vim.exe does not use newlines to actually break rows. IIRC, they know that the line is about to wrap, so they let it just happen.
repros with vim in WSL too

I don't believe this can be considered a bug in Windows Terminal as it happens in all other terminals as well without further integration into or configuration of vim. Searching for this issue on the internet brings up a lot of helpful advice how to fix this issue. One in particular that I liked is: https://stackoverflow.com/a/4608206
After running :vmap <C-C> "+y, I can simply select some text with my cursor (this enters the visual mode) and press Ctrl+C to copy to my clipboard. This works for me even if copyOnSelect is enabled, but one thing is a bit confusing: vim doesn't color the cursor position, so if you use the default I-beam (the thin vertical "|" line) it seems like one less character is selected, but as far as I can see that's exactly what vim wants us to draw.
Would you say that we should close this issue @89z? We can leave it open, but I don't believe we can do anything about this on our side at this time.

You seem to be getting hung up on the the Vim part of this issue, which is a classic mistake in troubleshooting.
Further, you seem to come to the conclusion that "well if all terminals are broken, then no one needs to fix anything!" which hopefully can be recognized as a silly conclusion. The issue isn't specific to Vim, thats just a good example because its a popular tool that demonstrates the problem. The issue is that any tool which activates "interactive mode", will suffer from this problem.

You seem to be getting hung up on the the Vim part of this issue, which is a classic mistake in troubleshooting.
Further, you seem to come to the conclusion that "well if all terminals are broken, then no one needs to fix anything!" which hopefully can be recognized as a silly conclusion. The issue isn't specific to Vim, thats just a good example because its a popular tool that demonstrates the problem. The issue is that any tool which activates "interactive mode", will suffer from this problem.
I tried to make it clear that this wasn't a specific issue with Vim, but then @zadjii-msft went and changed the title, which has only helped with the confusion.

At no point did you mention that this occurs with other applications? Your repro steps specifically called out vim, without mentioning anything else that repros this.
Vim does play silly games with line wrapping, hence my assumption of the root cause here. Other applications that play the same silly games would also experience the same issues, but without knowing where else you're seeing this, all we an do is presume that this is an issue specific to vim.
Do you have another commandline tool that reproduces the same behavior?

MITM Proxy demonstrates the same problem. Thats actually the tool that spurned this issue in the first place. I only changed it to Vim, because thats more "known". @zadjii-msft perhaps you could have just asked me before changing the title?
Like I said, its a classic mistake in troubleshooting. people have a tendency to want to blame something, in this case Vim, rather than consider a broader issue, or the simple fact that Vim might be acting "normally" and that its the Terminal doing something wrong. Because if Vim is the culprit, you can just close the issue and be done with it. But if its actually the Terminal, then work needs to be done. I'm not blaming anyone, but its something to keep in mind in the future.

Could you expand on how we could solve this issue in technical terms given only the VT output / console API calls of vim without modifying vim itself (or "MITM proxy")?

@lhecker I don't really care with how to entered the conversation, so I not inclined to speak with you at all. However in the interest of at least saying "I tried", I will try. However I want to make sure we're on the same page before I put forth any effort. From your comments, you seem to want me to solve the issue for you, which I am also not inclined to do.  As far as I'm concerned, my duty stops and ends with posting an issue. However I also understand that you might be inclined to close any issues where the poster doesn't solve the issue for you.
Is that your situation, or are you willing to let others work on this without me giving a solution. Put another way, do you require all issue posters to solve their own issue? Again I am will to put work in service of a goal here, but I don't like the way you entered the conversation, so I want to make damn sure I'm not wasting my time before making the decision to do that work.

Hey @89z! I'm sorry. We didn't mean to offend you.
There's a constellation of issues here, all of which present in the same way. All of them involve applications that write to the entire screen (using Windows console APIs or by switching to the alternate screen buffer mode) and cause a set or reset of the "hard wrap" flag when they touch the end of the line. By and large, the decision we make here in the absence of line ending information is informed by what would be best for most applications.
We need to approach this from a few angles (simultaneously):

Ensure that applications are using the best available mechanism today for reporting line wrapping, through community outreach and targeted work with those projects.
Better heuristics ("when xyz.exe touches the end of the buffer to draw spaces using WriteConsoleXyz API, know to ignore the wrap flag")
Better diagnostics (like: how did we not know that mitmproxy was in the same boat? Could we have caught it earlier?)

To Leonard's question: the burden is of course not on you, but if you do have any ideas as to how we can "know better" we would love to hear them. Not because it's your job, but because Terminal is a community and we want to make the best use of all our collective knowledge. :)

OK, here is one way to repro the issue. First, as I originally said, get Vim [1], and Alacritty [2]. Start Alacritty, then start Vim. Then enter copy mode with Alacritty (Ctrl+Shift+Space) and select the text mentioned in the original post. Copy with y. Then paste somewhere else. Result should be:
~                               version 8.2.2800
~                           by Bram Moolenaar et al.

Then start Windows Terminal, and start Vim. Select the text mentioned in the original post. I have copyOnSelect, so selection is copied. Then paste somewhere else. Result should be:
~                                version 8.2.2800                                  ~                            by Bram Moolenaar et al.

as mentioned in previous comments, this is just one example. Please do not use this comment as an excuse to target Alacritty, as I think other examples could probably be constructed with other terminals as well.

https://github.com/vim/vim-win32-installer/releases/download/v8.2.2800/gvim_8.2.2800_x64.zip
https://github.com/alacritty/alacritty/releases/download/v0.11.0/Alacritty-v0.11.0-portable.exe


Xcode build error on React Native 0.65.1

Describe the bug
Build not successful on React Native 0.65.1
To Reproduce
Steps to reproduce the behavior:

Initialize new react native project
Install @stripe/stripe-react-native
run pod install in project's ios folder
Open project workspace in Xcode and build

Expected behavior
For brand new project the build process should be successful
Screenshots

Error's in text format
Undefined symbols for architecture arm64:
  "protocol conformance descriptor for __C.NSOperationQueue : Combine.Scheduler in Foundation", referenced from:
      lazy protocol witness table accessor for type __C.NSOperationQueue and conformance __C.NSOperationQueue : Combine.Scheduler in Foundation in libStripe.a(URLSession+Retry.o)
  "type metadata accessor for (extension in Foundation):__C.NSOperationQueue.SchedulerTimeType", referenced from:
      closure #1 (Foundation.Data?, __C.NSURLResponse?, Swift.Error?) -> () in (extension in Stripe):__C.NSURLSession.stp_performDataTask(with: Foundation.URLRequest, completionHandler: (Foundation.Data?, __C.NSURLResponse?, Swift.Error?) -> (), retryCount: Swift.Int) -> () in libStripe.a(URLSession+Retry.o)
  "(extension in Foundation):__C.NSScanner.scanUpToString(Swift.String) -> Swift.String?", referenced from:
      static Stripe.STPPhoneNumberValidator.formattedRedactedPhoneNumber(for: Swift.String, forCountryCode: Swift.String?) -> Swift.String in libStripe.a(STPPhoneNumberValidator.o)
  "(extension in Foundation):__C.NSOperationQueue.SchedulerTimeType.init(Foundation.Date) -> (extension in Foundation):__C.NSOperationQueue.SchedulerTimeType", referenced from:
      closure #1 (Foundation.Data?, __C.NSURLResponse?, Swift.Error?) -> () in (extension in Stripe):__C.NSURLSession.stp_performDataTask(with: Foundation.URLRequest, completionHandler: (Foundation.Data?, __C.NSURLResponse?, Swift.Error?) -> (), retryCount: Swift.Int) -> () in libStripe.a(URLSession+Retry.o)
  "_swift_getOpaqueTypeConformance", referenced from:
      associated type witness table accessor for Body : SwiftUI.View in Stripe.PaymentSheet.PaymentButton<A> : SwiftUI.View in Stripe in libStripe.a(PaymentSheet+SwiftUI.o)
      associated type witness table accessor for Body : SwiftUI.View in Stripe.PaymentSheet.FlowController.PaymentOptionsButton<A> : SwiftUI.View in Stripe in libStripe.a(PaymentSheet+SwiftUI.o)
      associated type witness table accessor for Body : SwiftUI.View in Stripe.PaymentSheet.FlowController.ConfirmButton<A> : SwiftUI.View in Stripe in libStripe.a(PaymentSheet+SwiftUI.o)
      associated type witness table accessor for SwiftUI.ViewModifier.Body : SwiftUI.View in Stripe.PaymentSheet.PaymentSheetPresentationModifier : SwiftUI.ViewModifier in Stripe in libStripe.a(PaymentSheet+SwiftUI.o)
      associated type witness table accessor for SwiftUI.ViewModifier.Body : SwiftUI.View in Stripe.PaymentSheet.PaymentSheetFlowControllerPresentationModifier : SwiftUI.ViewModifier in Stripe in libStripe.a(PaymentSheet+SwiftUI.o)
      associated type witness table accessor for Body : SwiftUI.View in Stripe.ConfirmPaymentPresentationModifier<A, B> : SwiftUI.ViewModifier in Stripe in libStripe.a(STPPaymentConfirmation+SwiftUI.o)
  "__swift_FORCE_LOAD_$_swiftCoreMIDI", referenced from:
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(STPPaymentMethodGiropayParams.o)
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(STPPaymentMethodAUBECSDebit.o)
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(STPContactField.o)
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(ApplePayHeaderView.o)
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(BottomSheet3DS2ViewController.o)
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(STPThreeDSLabelCustomization.o)
      __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe in libStripe.a(ConfirmButton.o)
      ...
     (maybe you meant: __swift_FORCE_LOAD_$_swiftCoreMIDI_$_stripe_react_native, __swift_FORCE_LOAD_$_swiftCoreMIDI_$_Stripe )
  "__swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers", referenced from:
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(STPPaymentMethodGiropayParams.o)
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(STPPaymentMethodAUBECSDebit.o)
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(STPContactField.o)
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(ApplePayHeaderView.o)
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(BottomSheet3DS2ViewController.o)
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(STPThreeDSLabelCustomization.o)
      __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe in libStripe.a(ConfirmButton.o)
      ...
     (maybe you meant: __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_stripe_react_native, __swift_FORCE_LOAD_$_swiftUniformTypeIdentifiers_$_Stripe )
  "__swift_FORCE_LOAD_$_swiftCoreML", referenced from:
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(STPPaymentMethodGiropayParams.o)
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(STPPaymentMethodAUBECSDebit.o)
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(STPContactField.o)
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(ApplePayHeaderView.o)
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(BottomSheet3DS2ViewController.o)
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(STPThreeDSLabelCustomization.o)
      __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe in libStripe.a(ConfirmButton.o)
      ...
     (maybe you meant: __swift_FORCE_LOAD_$_swiftCoreML_$_stripe_react_native, __swift_FORCE_LOAD_$_swiftCoreML_$_Stripe )
  "__swift_FORCE_LOAD_$_swiftWebKit", referenced from:
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(STPPaymentMethodGiropayParams.o)
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(STPPaymentMethodAUBECSDebit.o)
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(STPContactField.o)
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(ApplePayHeaderView.o)
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(BottomSheet3DS2ViewController.o)
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(STPThreeDSLabelCustomization.o)
      __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe in libStripe.a(ConfirmButton.o)
      ...
     (maybe you meant: __swift_FORCE_LOAD_$_swiftWebKit_$_stripe_react_native, __swift_FORCE_LOAD_$_swiftWebKit_$_Stripe )
ld: symbol(s) not found for architecture arm64

My working environment
MacOs: v11.2 (Big Sur)
Node: v14.16.1
React Native: v0.65.1
@stripe/stripe-react-native: v0.2.2

You can either let me know which version of react native does this plugin build successfully so we can downgrade because it does not work even if i create bridging header according to troubleshooting docs https://github.com/stripe/stripe-react-native#troubleshooting, Thanks.

I fixed mine by adding what is suggested in this link https://github.com/stripe/stripe-react-native#troubleshooting

@nywooz  what react native version is your build process successful..

i did exactly the same process in brand new react native project(0.65.1) and followed suggestion in here https://github.com/stripe/stripe-react-native#troubleshooting and build process failed, furthermore in brand new react native project i didn't see any entries relating to ../usr/swift.. in library search paths..
Thanks.

@takeerdevs you need to change project Settings rather than target ones

I found the solution, suggested solution didn't work, you don't even have to remove swift related entries, all you have to do is change ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES to Yes in Project->Build Settings

This one is also not working


Pressing enter on the document search exits the plugin

Bug reported by @elisedufief.
This is because the form is being submitted, which shouldn’t be possible.

Fixed in 92b060895ce535c2012174c89b89748ec5fbae9a.


AsyncIterator support?

I am searching for a faster alternative to Node.js EventEmitter. I am using events through its AsyncIterator interface (https://nodejs.org/api/events.html#eventsonemitter-eventname-options).
EventEmitter3 claims compatibility with Node.js EventEmitter, but I did not find mentions on AsyncIterator support. Is such in plans, or should I bake my own solution on top EventEmitter3? If so, I would appreciate any pointers towards a working sample solution.

eventemitter3 is compatible with the EventEmitter class. Everything else is outside the scope of the project.

Roger that. In my reading, hooking EventEmitter3 to events.on(, ), as in https://nodejs.org/api/events.html#eventsonemitter-eventname-options might work if the original EventEmitter API is followed (and if Node.js also uses them and not some internal API, instead).
I'll have a quick try soonish and let you know how it went.


Build some PDF Generation tool (like bread) into the BMLT Root Server

Older versions of root server had this

seems like unnecessary bloat

I think the same could be said for implementing this kind of software as a WordPress plugin. Generally WordPress plugins are related to displaying information on a website, not for performing tasks like PDF Generation.
When I was originally introduced to the ecosystem, I found it very confusing that I'd install a plugin into a framework for building websites for the purpose of PDF generation. It just seems normal to me now because I'm used to it, but it's not intuitive. It'd be more intuitive to put something like this in the root server.
Of course, there are many higher priority issues that need to be done first.

The other option is to go back to my old plan of creating a hosted service for this, like scroll. It might make sense to do that instead.

Not saying that WordPress is the best choice either.  I look at WordPress as the simplest way to get someone to build an IT infrastructure.  Minimal thought/experience required.  Some suggested that the Yap admin interface be a WordPress plugin for a similar reason.
My comment is generally about bloating the root server codebase.  It seems like the main hurdle here is not having someone technical that can maintain these things.  A better approach would be to possibly direct them to someone who hosts websites for service bodies for a living and have them respond to these requests, rather than having the software solve this problem.

Maintain and/or provision.

Seems like someone could start a SaaS company for these things and make a decent living.


SexyMotd console ?

Found another Console spam.. it think it's not that bad but it spam's my console whith this..
http://pastebin.com/WgipsacF

Please use pastebin or something.

Sorry :S never used that

http://pastebin.com

This is nothing to do with this plugin.

ohh srry :s


The log is stuck on the "running build_py" line for 2 hours， is this normal?

hello, I would like to consult you on a question. Thanks a lot.
I run T-Mac on ubuntu. In order to speed the install speed, I use the command "pip install -e . -v -i https://pypi.tuna.tsinghua.edu.cn/simple", But, the log (running build_py) has been ongoing for 2 hours; no changes, is this normal?
Building wheels for collected packages: t-mac
Running command Building editable for t-mac (pyproject.toml)
running editable_wheel
creating /tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info
writing /tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/dependency_links.txt
writing requirements to /tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/requires.txt
writing top-level names to /tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/top_level.txt
writing manifest file '/tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac.egg-info/SOURCES.txt'
creating '/tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac-1.0.0a3.dist-info'
creating /tmp/pip-wheel-0oi5xt_5/.tmp-ozbv9npk/t_mac-1.0.0a3.dist-info/WHEEL
running build_py

After i tried many times, I found too much time spent on downloading clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz, so I download clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz file manually, and put it on T-MAC/build/, then use the command tar -xJf clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz to unzip the file.  After that, when I use the command  "pip install -e . -v -i https://pypi.tuna.tsinghua.edu.cn/simple", the error occurred at the line “cmake ...”
(tmac) root@ubuntu:/home/ictuser/T-MAC/3rdparty/tvm/build# cmake ..
-- Forbidding undefined symbols in shared library, using -Wl,--no-undefined on platform Linux
-- Build with RPC support...
-- Build with Graph Executor support...
-- Build with profiler...
-- Build with AOT Executor support...
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- Build Alloc alignment set to 64
-- Didn't find the path to CCACHE, disabling ccache
-- VTA build with VTA_HW_PATH=/home/ictuser/T-MAC/3rdparty/tvm/3rdparty/vta-hw
-- Build VTA runtime with target: sim
-- Use llvm-config=/home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config
/home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config) /home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config)
/home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config) /home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config)
CMake Error at cmake/utils/FindLLVM.cmake:80 (message):
Fatal error executing:
/home/ictuser/T-MAC/build/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-config
--libfiles
Call Stack (most recent call first):
cmake/modules/LLVM.cmake:31 (find_llvm)
CMakeLists.txt:559 (include)
-- Configuring incomplete, errors occurred!

Thanks very much, I fixed the promblem. But，when I use the command "pip install -e . -v -i https://pypi.tuna.tsinghua.edu.cn/simple", a new problem happen:
Consolidate compiler generated dependencies of target tvm_libinfo_objs
[  0%] Performing configure step for 'project_libbacktrace'
[  0%] Built target tvm_libinfo_objs
/bin/sh: 1: /T-MAC/3rdparty/tvm/cmake/libs/../../3rdparty/libbacktrace/configure: not found
make[2]: *** [CMakeFiles/project_libbacktrace.dir/build.make:131: libbacktrace/src/project_libbacktrace-stamp/project_libbacktrace-configure] Error 127
make[1]: *** [CMakeFiles/Makefile2:229: CMakeFiles/project_libbacktrace.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Consolidate compiler generated dependencies of target tvm_objs
[ 91%] Built target tvm_objs
make: *** [Makefile:136: all] Error 2
Download and extraction completed successfully.
running egg_info
creating /tmp/tmpre3bcgei.build-temp/t_mac.egg-info
writing /tmp/tmpre3bcgei.build-temp/t_mac.egg-info/PKG-INFO
writing dependency_links to /tmp/tmpre3bcgei.build-temp/t_mac.egg-info/dependency_links.txt
writing requirements to /tmp/tmpre3bcgei.build-temp/t_mac.egg-info/requires.txt
writing top-level names to /tmp/tmpre3bcgei.build-temp/t_mac.egg-info/top_level.txt
writing manifest file '/tmp/tmpre3bcgei.build-temp/t_mac.egg-info/SOURCES.txt'
reading manifest file '/tmp/tmpre3bcgei.build-temp/t_mac.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file '/tmp/tmpre3bcgei.build-temp/t_mac.egg-info/SOURCES.txt'
/tmp/pip-build-env-mhp_c6f4/overlay/lib/python3.8/site-packages/setuptools/command/editable_wheel.py:289: SetuptoolsDeprecationWarning: Customization incompatible with editable install
!!
      ********************************************************************************
                      Traceback (most recent call last):
        File "<string>", line 119, in build_tvm
        File "/root/anaconda3/envs/t-mac/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['make', '-j4']' returned non-zero exit status 2.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/tmp/pip-build-env-mhp_c6f4/overlay/lib/python3.8/site-packages/setuptools/command/editable_wheel.py", line 295, in _safely_run
          return self.run_command(cmd_name)
        File "/tmp/pip-build-env-mhp_c6f4/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-mhp_c6f4/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 950, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-mhp_c6f4/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "<string>", line 152, in run
        File "<string>", line 121, in build_tvm
      RuntimeError: Failed to build TVM

                      If you are seeing this warning it is very likely that a setuptools
                      plugin or customization overrides the `build_py` command, without
                      taking into consideration how editable installs run build steps
                      starting from setuptools v64.0.0.

                      Plugin authors and developers relying on custom build steps are
                      encouraged to update their `build_py` implementation considering the
                      information about editable installs in
                      https://setuptools.pypa.io/en/latest/userguide/extension.html.

                      For the time being `setuptools` will silence this error and ignore
                      the faulty command, but this behaviour will change in future versions.

      ********************************************************************************

!!
self._safely_run(name)
      Editable install will be performed using .pth file to extend `sys.path` with:
      ['python']

Options like package-data, include/exclude-package-data or
packages.find.exclude/include may have no effect.

deeply, when i use the command "make -j4", the error tell me /T-MAC/3rdparty/tvm/cmake/libs/../../3rdparty/libbacktrace/configure: not found, but it is exist really.
make -j4
[  0%] Performing configure step for 'project_libbacktrace'
[  0%] Building CXX object CMakeFiles/tvm_libinfo_objs.dir/src/support/libinfo.cc.o
/bin/sh: 1: /T-MAC/3rdparty/tvm/cmake/libs/../../3rdparty/libbacktrace/configure: not found
make[2]: *** [CMakeFiles/project_libbacktrace.dir/build.make:131: libbacktrace/src/project_libbacktrace-stamp/project_libbacktrace-configure] Error 127
make[1]: *** [CMakeFiles/Makefile2:229: CMakeFiles/project_libbacktrace.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

I met the problem  that got stuck here. Have you fixed this problem? I am downloading clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz file manually now as you mentioned.

thanks for the author of issue 42, who told me use TVM's GitHub repository to compile and ran T-MAC. It works.

I met the problem that got stuck here. Have you fixed this problem? I am downloading clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04.tar.xz file manually now as you mentioned.

thanks for the author of issue 42, who told me use TVM's GitHub repository to compile and ran T-MAC, it works.

@sunj0104 Thanks for your info. We have been using fixed TVM commit 36b9535 (June 20) for some time, but we didn't meet your problem. Did you meet problem with this commit? And which TVM commit solved your problem?

After I pull the TVM’s repository (default commit and branch at that time )manually , I can run it successfully without any other issues.

@sunj0104 Thanks for your info. We have been using fixed TVM commit 36b9535 (June 20) for some time, but we didn't meet your problem. Did you meet problem with this commit? And which TVM commit solved your problem?

After I pull the TVM’s repository (default commit and branch at that time )manually , I can run it successfully without any other issues.

Do you remember the TVM commit ID you pulled?

@sunj0104 Thanks for your info. We have been using fixed TVM commit 36b9535 (June 20) for some time, but we didn't meet your problem. Did you meet problem with this commit? And which TVM commit solved your problem?

After I pull the TVM’s repository (default commit and branch at that time )manually , I can run it successfully without any other issues.

Do you remember the TVM commit ID you pulled?

Sorry, I don't remember the commit ID, but I remain the two tvm folders, do you know have any method I can get the commit ID?

hello,
I'm sorry to bother you, but I've run into a problem and need your help.
as the issue I write on the github said, when I running "python tools/run_pipeline.py -o /root/QwenQwen2-7B-Instruct-GPTQ-Int4 -m gptq-auto -q int_n" report error like this:
......
INFO:hf-to-gguf:blk.27.attn_v.bias, torch.float16 --> F32, shape = {512}
INFO:hf-to-gguf:blk.27.attn_v.weight, torch.float16 --> I4, shape = {3584, 512}
INFO:hf-to-gguf:output_norm.weight, torch.float16 --> F32, shape = {3584}
Traceback (most recent call last):
File "convert_hf_to_gguf.py", line 4543, in
main()
File "convert_hf_to_gguf.py", line 4537, in main
model_instance.write()
File "convert_hf_to_gguf.py", line 527, in write
self.prepare_tensors()
File "convert_hf_to_gguf.py", line 352, in prepare_tensors
for name, data_torch in chain(self.generate_extra_tensors(), self.get_tensors()):
File "convert_hf_to_gguf.py", line 190, in get_tensors
raise ValueError("Mismatch between weight map and model parts for tensor names:\n"
ValueError: Mismatch between weight map and model parts for tensor names:
Missing tensors: []
Extra tensors: ['model.layers.0.mlp.down_proj.bias', 'model.layers.0.mlp.gate_proj.bias',......
Do you know the reason, thanks for your help~


react-native-elements - vector icons

It seems like create-react-native app has some issues with "react-native-elements". I've tried 2 scenarios which both produces the same error:
Test 1

created a new project with create-react-native-app
npm i react-native-elements --save
Added a react-native-elements button in App.js
npm run ios

Test 2

created a new project with create-react-native-app
npm i react-native-vector-icons --save && react-native link react-native-vector-icons
npm i react-native-elements --save
Added a react-native-elements button in App.js
npm run ios

In both test the simulator throws this warning. I've tried to run the commands described in the error message, but with no luck :(

Hi! Thanks for the report. We're working on making it easier to use these transparently in CRNA.
If you first install https://www.npmjs.com/package/@expo/vector-icons and then try using react-native-elements, this should work for you. Note that if/when you eject to a regular RN project, you'll need to switch from Expo's vector icons to the regular react-native-vector-icons.
Let me know if that resolves your immediate issue. In the near future we hope to resolve this more transparently for users.

@dikaiosune thx got me a little further. Now this issues arrises

Hello! Thanks for catching this, we'll add ttf as a default support asset extension in our next release this afternoon.
See this repo for a working example of react-native-elements: https://github.com/exponent/elements-example-crna
The most important part is this: https://github.com/exponent/elements-example-crna/blob/master/app.json#L4-L6 -- if you add it to your app.json and restart the packager (necessary to pick up the changes to the packager options) it should work

@brentvatne In regards to https://github.com/react-native-community/react-native-elements/issues/221 I am trying to create documentation on how to use react-native-elements with create-react-native-app. Is the change in the app.json required only till the new release or will that change be required?

@binoy14 I just released a temporary fix. Going to leave this open until I can fix upstream.
TODO: release new xdl version and update react-native-scripts to it.


Properly animate Spectral Thief

In case of boosts, the attacking animation would play 1st, stealing the
boosts last, instead of the other way around.
This approach was chosen since Spectral Thief is the only move that
steals boosts this way, for now

This is probably the wrong way to animate it :|


Error about missing symbols in the symbols table

Hi, maybe I don't completely understand the structure for the required files, but I tried to guess them looking at the code.
Command I use is
./laia/scripts/htr/train_ctc.py syms.txt [./train_rus2/images/] tr.txt val.txt
Where
./train_rus2/images/ contains jpg files with names such as 00000003_1.jpg, etc
tr.txt and val.txt contain strings like that:
00000003_1 въ доме московскаго универ
00000003_10 валентина
(<image filename without .jpg> <string that corresponds to the image>)
Where syms.txt contains
⊗ 0
) 1
+ 2
/ 3
0 4
1 5
2 6
3 7
4 8
5 9
6 10
7 11
8 12
9 13
[ 14
] 15
i 16
k 17
l 18
| 19
× 20
ǂ 21
а 22
б 23
в 24
г 25
д 26
е 27
ж 28
з 29
и 30
й 31
к 32
л 33
м 34
н 35
о 36
п 37
р 38
с 39
т 40
у 41
ф 42
х 43
ц 44
ч 45
ш 46
щ 47
ъ 48
ы 49
ь 50
э 51
ю 52
я 53
і 54
– 55
. 56
<space> 57

What I get is
[2021-03-03 16:08:03, ... ERROR laia.data.transforms.text.transforms] Could not find ... in the symbols table
[2021-03-03 16:08:03,856 ERROR laia.data.transforms.text.transforms] Could not find "20" in the symbols table
[2021-03-03 16:08:04,049 CRITICAL laia] Uncaught exception:
Traceback (most recent call last):
  File "/home/pchela/TextRecognition/PyLaia/laia/engine/engine_exception.py", line 27, in exception_catcher
    yield
  File "/home/pchela/TextRecognition/PyLaia/laia/engine/engine_module.py", line 115, in compute_loss
    batch_loss = self.criterion(batch_y_hat, batch_y, **kwargs)
  File "/home/pchela/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/pchela/TextRecognition/PyLaia/laia/losses/ctc_loss.py", line 102, in forward
    y = torch.tensor(list(itertools.chain.from_iterable(y)), dtype=torch.int)
TypeError: an integer is required (got type NoneType)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./laia/scripts/htr/train_ctc.py", line 204, in <module>
    main()
  File "./laia/scripts/htr/train_ctc.py", line 200, in main
    run(**args)
  File "./laia/scripts/htr/train_ctc.py", line 128, in run
    trainer.fit(engine_module, datamodule=data_module)
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 468, in fit
    results = self.accelerator_backend.train()
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/cpu_accelerator.py", line 61, in train
    results = self.train_or_test()
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 66, in train_or_test
    results = self.trainer.train()
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 490, in train
    self.run_sanity_check(self.get_model())
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 697, in run_sanity_check
    _, eval_results = self.run_evaluation(test_mode=False, max_batches=self.num_sanity_val_batches)
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 613, in run_evaluation
    output = self.evaluation_loop.evaluation_step(test_mode, batch, batch_idx, dataloader_idx)
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 178, in evaluation_step
    output = self.trainer.accelerator_backend.validation_step(args)
  File "/home/pchela/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/cpu_accelerator.py", line 77, in validation_step
    output = self.trainer.model.validation_step(*args)
  File "/home/pchela/TextRecognition/PyLaia/laia/engine/htr_engine_module.py", line 72, in validation_step
    result = super().validation_step(batch, *args, **kwargs)
  File "/home/pchela/TextRecognition/PyLaia/laia/engine/engine_module.py", line 150, in validation_step
    batch_loss = self.compute_loss(batch, batch_y_hat, batch_y)
  File "/home/pchela/TextRecognition/PyLaia/laia/engine/engine_module.py", line 119, in compute_loss
    return batch_loss
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/pchela/TextRecognition/PyLaia/laia/engine/engine_exception.py", line 29, in exception_catcher
    raise EngineException(
laia.engine.engine_exception.EngineException: Exception "TypeError('an integer is required (got type NoneType)')" raised during epoch=0, global_step=0 with batch=['00000004_20', '00000003_22', '00000003_8', '00000004_34', '00000003_5', '00000003_36', '00000004_48', '00000003_0']`

It's probably because you are missing the CTC symbol
<ctc> 0
Try putting it in your syms.txt file

Also notice the two errors at the top
[2021-03-03 16:08:03, ... ERROR laia.data.transforms.text.transforms] Could not find ... in the symbols table
[2021-03-03 16:08:03,856 ERROR laia.data.transforms.text.transforms] Could not find "20" in the symbols table

Which means either you didn't add them to syms.txt or there is an issue in your tokenization

I replaced "⊗ 0" with " 0" in the syms.txt file. However, the problem still persists.
Yeah, this script is looking for strings of text (which correspond to the images) in syms.txt for some reason (probably because letters are Cyrillic). Is there a way to fix it?
It is also suspicious that batch contains image string ids instead of image tensors.

It's also a pity I can't see how exactly the dataset is organized in the IAM dataset, because of this error:
2021-03-05 13:54:24 (501 MB/s) - ‘data/original/words.tgz’ saved [4720/4720]

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
ERROR: Failed extracting IAM ascii files.

I replaced "⊗ 0" with " 0" in the syms.txt file. However, the problem still persists.

I'm saying it should be
<ctc> 0
⊗ 1
) 2
+ 3
/ 4

Yeah, this script is looking for strings of text (which correspond to the images) in syms.txt for some reason (probably because letters are Cyrillic). Is there a way to fix it?

Is your text properly tokenized? as in, every token is present in the syms.txt file?

It is also suspicious that batch contains image string ids instead of image tensors.

That's on purpose so it's easy to see which images produced an exception, as sometimes the failure comes due to images being too small or not correctly labeled

because of this error:

What did you run that produced this error?
You can try running a benchmark to see the general structure
https://github.com/jpuigcerver/PyLaia/blob/master/benchmarks

I'm saying it should be
<ctc> 0
⊗ 1
) 2
+ 3
/ 4
...

Yeah, sorry, I meant that, but forgot about Markdown escape symbols.

Is your text properly tokenized? as in, every token is present in the syms.txt file?

I thought only letters and other chars should be in syms.txt, because Handwritten recognition recongnizes them in the CTCLoss neural networks. Does it require words?

What did you run that produced this error?

I ran run.sh from https://github.com/carmocca/PyLaia-examples/tree/master/iam-htr

I thought only letters and other chars should be in syms.txt, because Handwritten recognition recongnizes them in the CTCLoss neural networks. Does it require words or strings?

So if your image contains The red house with id img-01.jpg
You'd have the line text:
img-01 T h e <space> r e d <space> h o u s e
and your syms.txt
<ctc> 0
T 1
h 2
e 3
r 4
d 5
h 6
o 7
u 8
s 9
<space> 10

I ran run.sh from https://github.com/carmocca/PyLaia-examples/tree/master/iam-htr I specified correct username and password, and I already encountered problems downloading and extracting words.tgz before.

Unfortunately, I don't have crendentials to test it, but if you have the local files, you can adapt https://github.com/carmocca/PyLaia-examples/blob/master/iam-htr/src/download.sh to skip the actual download part.

Thanks, now I see that strings of text should contain <space> instead of spaces, which are used here as char separators. That solved the error, but I got another one: 'The loss is NaN or ± inf', that seems not connected with the previous error, so I'll close this issue and open another


chore(deps): rm some rlp deps

Codecov Report

Merging #1839 (55f2d4b) into main (85076fb) will decrease coverage by 53.86%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff             @@
##             main    #1839       +/-   ##
===========================================
- Coverage   73.55%   19.70%   -53.86%
===========================================
  Files         410      410
  Lines       50509    50509
===========================================
- Hits        37152     9952    -27200
- Misses      13357    40557    +27200

Flag
Coverage Δ

integration-tests
19.70% <0.00%> (-0.02%)
:arrow_down:

unit-tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

crates/rlp/rlp-derive/src/lib.rs
96.77% <ø> (-3.23%)
:arrow_down:

crates/rlp/src/lib.rs
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 275 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Update wood gross energy cost per kwh to use consumer.nz numbers

Taking an average of gross energy cost per kwh over 4 typical wood fuel types using consumer.nz numbers
Note that these costs refer to the wood energy content rather than the final heat delivered (which will be ~1/0.7 more expensive per kWh for a typical wood burner).

Closing: we need to include ratio of volume of thrown wood to solid wood (which is 2:1 according to https://ec.europa.eu/eurostat/documents/38154/4956233/Quality_standard_statistics_wood_fuel_consumption_households_CA-RES_2012.pdf) and should discuss which prices and NCVs to use.