Include reimplemented methods in whats_left.sh, and skip inherited methods

This is so that in e.g. #358 , bool methods that are inherited from int aren't displayed. But methods that have are currently implemented as inherited from int, but should instead be reimplemented with special behavior, are displayed.

Codecov Report

Merging #919 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #919   +/-   ##
=======================================
  Coverage   64.16%   64.16%
=======================================
  Files          89       89
  Lines       15415    15415
  Branches     3452     3452
=======================================
  Hits         9891     9891
  Misses       3233     3233
  Partials     2291     2291

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee2b2a8...c38a371. Read the comment docs.


Bugfix/DNN-33796_Invalid_Username_Email_Address_When_Containing_onfocus

Invalid Username/Email Address When Containing "onfocus"
Fixes #DNN-33796  https://jira.devfactory.com/browse/DNN-33796
Summary
Solution to replace RegEx
DNN-33796_fixed
Not UT's for open source

@sergeydryomin please create a proper GitHub issue describing the bug/enhancement in detail.
Then reference the issue using the proper GitHub syntax.  Then please modify the title for this PR to something that does not contain a JIRA issue number.  This is confusing to the open source community because we have no visibility to JIRA issues.  Thanks.


Anonymous endpoints are not working if web.ignoring().antMatchers is used

Description of the bug
In a secured application with Spring security, I want to be able to use anonymous Endpoint as anonymous.
If the SecurityConfig contains web.ignoring().antMatchers("/images/**"); as described in the documentation I have a 401 error when calling the anonymous Endpoint.
Minimal reproducible example
I used a simple fusion starter. I've added a annonymous endpoint.
Then Spring Security.

I have a client side error:

Log server:
2021-05-22 14:23:25.108 DEBUG 3597 --- [nio-8080-exec-5] o.s.security.web.FilterChainProxy        : Securing GET /
2021-05-22 14:23:25.108 DEBUG 3597 --- [nio-8080-exec-5] s.s.w.c.SecurityContextPersistenceFilter : Set SecurityContextHolder to empty SecurityContext
2021-05-22 14:23:25.108 DEBUG 3597 --- [nio-8080-exec-5] o.s.s.w.a.AnonymousAuthenticationFilter  : Set SecurityContextHolder to anonymous SecurityContext
2021-05-22 14:23:25.108 DEBUG 3597 --- [nio-8080-exec-5] o.s.s.w.a.i.FilterSecurityInterceptor    : Authorized filter invocation [GET /] with attributes [permitAll]
2021-05-22 14:23:25.108 DEBUG 3597 --- [nio-8080-exec-5] o.s.security.web.FilterChainProxy        : Secured GET /
2021-05-22 14:23:25.113 DEBUG 3597 --- [nio-8080-exec-5] w.c.HttpSessionSecurityContextRepository : Did not store anonymous SecurityContext
2021-05-22 14:23:25.113 DEBUG 3597 --- [nio-8080-exec-5] w.c.HttpSessionSecurityContextRepository : Did not store anonymous SecurityContext
2021-05-22 14:23:25.113 DEBUG 3597 --- [nio-8080-exec-5] s.s.w.c.SecurityContextPersistenceFilter : Cleared SecurityContextHolder to complete request
2021-05-22 14:23:25.286 DEBUG 3597 --- [nio-8080-exec-6] o.s.security.web.FilterChainProxy        : Securing GET /vaadinServlet/?v-r=push&refresh_connection
2021-05-22 14:23:25.286 DEBUG 3597 --- [nio-8080-exec-6] s.s.w.c.SecurityContextPersistenceFilter : Set SecurityContextHolder to empty SecurityContext
2021-05-22 14:23:25.287 DEBUG 3597 --- [nio-8080-exec-6] o.s.s.w.a.AnonymousAuthenticationFilter  : Set SecurityContextHolder to anonymous SecurityContext
2021-05-22 14:23:25.287 DEBUG 3597 --- [nio-8080-exec-6] o.s.s.w.a.i.FilterSecurityInterceptor    : Authorized filter invocation [GET /vaadinServlet/?v-r=push&refresh_connection] with attributes [permitAll]
2021-05-22 14:23:25.287 DEBUG 3597 --- [nio-8080-exec-6] o.s.security.web.FilterChainProxy        : Secured GET /vaadinServlet/?v-r=push&refresh_connection
2021-05-22 14:23:25.288 DEBUG 3597 --- [nio-8080-exec-6] w.c.HttpSessionSecurityContextRepository : Did not store anonymous SecurityContext
2021-05-22 14:23:25.288 DEBUG 3597 --- [nio-8080-exec-6] s.s.w.c.SecurityContextPersistenceFilter : Cleared SecurityContextHolder to complete request
2021-05-22 14:23:29.763 DEBUG 3597 --- [io-8080-exec-10] o.s.security.web.FilterChainProxy        : Securing POST /connect/HelloEndpoint/hello
2021-05-22 14:23:29.763 DEBUG 3597 --- [io-8080-exec-10] s.s.w.c.SecurityContextPersistenceFilter : Set SecurityContextHolder to empty SecurityContext
2021-05-22 14:23:29.764 DEBUG 3597 --- [io-8080-exec-10] o.s.s.w.a.AnonymousAuthenticationFilter  : Set SecurityContextHolder to anonymous SecurityContext
2021-05-22 14:23:29.764 DEBUG 3597 --- [io-8080-exec-10] o.s.s.w.a.i.FilterSecurityInterceptor    : Authorized filter invocation [POST /connect/HelloEndpoint/hello] with attributes [permitAll]
2021-05-22 14:23:29.764 DEBUG 3597 --- [io-8080-exec-10] o.s.security.web.FilterChainProxy        : Secured POST /connect/HelloEndpoint/hello
2021-05-22 14:23:29.765  INFO 3597 --- [io-8080-exec-10] c.v.f.s.c.a.VaadinConnectAccessChecker   : Invalid CSRF token in endpoint request
2021-05-22 14:23:29.766 DEBUG 3597 --- [io-8080-exec-10] w.c.HttpSessionSecurityContextRepository : Did not store anonymous SecurityContext
2021-05-22 14:23:29.766 DEBUG 3597 --- [io-8080-exec-10] w.c.HttpSessionSecurityContextRepository : Did not store anonymous SecurityContext
2021-05-22 14:23:29.767 DEBUG 3597 --- [io-8080-exec-10] s.s.w.c.SecurityContextPersistenceFilter : Cleared SecurityContextHolder to complete request

Versions:
- Vaadin / Fusion version: 20 beta3
- Node version:
- Java version: 11
- OS version: mac

As a workaround (or solution), you can remove the configuration to ignore the images path and use:
http.authorizeRequests().antMatchers("/images/**").permitAll();

I can reproduce it in Safari. I think I can't in Chrome, FF,  ...  anymore

After checking with @Artur- , it's not related to Spring Security but to Safari.
I close it as duplicate #22


default value for SPARK_VERSION and spark_session

Hi!
Why can't specifying the Pyspark version in environment variables be optional?
like this
import os
import pyspark
os.environ['SPARK_VERSION'] = str(pyspark.__version__)

or
@lru_cache(maxsize=None)
def _get_spark_version() -> str:
    try:
        spark_version = os.environ.get("SPARK_VERSION")
        if not spark_version:
            spark_version = str(pyspark.__version__)
    except KeyError:
        raise RuntimeError(f"SPARK_VERSION environment variable is required. Supported values are: {SPARK_TO_DEEQU_COORD_MAPPING.keys()}")

    return _extract_major_minor_versions(spark_version)

And default spark version ...
like this
class VerificationResult:
    """The results returned from the VerificationSuite
    :param verificationRunBuilder verificationRun:  verification result run()
    """

    def __init__(self, spark_session: Optional[SparkSession], verificationRun):
        self._spark_session = self._setup_spark_session(spark_session)
        self.verificationRun = verificationRun

    def _setup_spark_session(self, session=None):
        if session:
            return session
        potencial_session = SparkSession.getActiveSession()
        if potencial_session:
            return potencial_session
        else:
            msg = "Spark session not found, init with `VerificationResult(my_session, ...)`"
            raise AttributeError(msg)

maybe i can do this and create PR?

There were some efforts to get runtime Spark version before, but it was reverted back because it caused issues https://github.com/awslabs/python-deequ/issues/111. You could take a look at what happened with the last attempt.
We would welcome ideas and PR to simplify this.


dlib library

The dlib library works only on the lower versions of Python. Make sure to include this in the readme file.

Many thanks to you!


Add missing functions in Q# example

Some functions were missing on Develop with Q# and .NET tutorial, making it unfeasible to execute the examples. Related to the #474 and the PR #503.

Docs Build status updates of commit 8431762:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/install-csharp-qdk.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@bradben
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

This issue has been addressed in PR 936 of the private repo.


Transform Service Not Propagateing  user credentials  and/or dbsync "discovery"

Direct transform service request
If I'm logged in as an elevated user, I can get a resource, but not the transformed resource; eg:
http --headers get  http://localhost/fcrepo/rest/expert/51d37fccfdb86059c507a75b46cb0f66 Authorization:"Bearer $jwt" | grep ^HTTP
#HTTP/1.1 200 OK

But
http --headers get  http://localhost/fcrepo/rest/expert/51d37fccfdb86059c507a75b46cb0f66/svc:node Authorization:"Bearer $jwt" | grep ^HTT
#HTTP/1.1 500 Internal Server Error

The response is:
{
    "details": {
        "message": "403 <!doctype html><title>HTTP Status 403 – Forbidden</title>
HTTP Status 403 – Forbidden

Type Status Report

Description The server understood the request but refuses to authorize it.

Apache Tomcat/9.0.65
",
        "stack": "Error: 403 <!doctype html><title>HTTP Status 403 – Forbidden</title>
HTTP Status 403 – Forbidden

Type Status Report

Description The server understood the request but refuses to authorize it.

Apache Tomcat/9.0.65
\n    at TransformService.exec (/fin/services/gateway/models/transform.js:659:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async parsed (/fin/services/gateway/models/service-proxy/transform-service.js:24:23)"
    },
    "error": true,
    "message": "Unable to render from transform service node"
}

This Error is thrown here:
https://github.com/ucd-library/fin/blob/8e73bfa637398d21db336da9186e444ecfbc332e/services/fin/gateway/models/transform.js#L657
Where I don't see the original request's either directAccess or jwt token being propogated.
Default Model
The model code for dbsync has similar problems.  However, there may be an additional problem in the model as well.  Here, I don't see how dbsync propagates a (What I would guess to be a direct access as discovery) request to the transform service.
https://github.com/ucd-library/fin/blob/8e73bfa637398d21db336da9186e444ecfbc332e/services/fin/dbsync/lib/model.js#L512

From @jrmerz
How about this, we allow admins to override their roles with a header.  so if the JWT token is authenticated and has role admin AND there is a fin-principal header, (ex: fin-principal: foo, bar ) then the request to fcrepo will ONLY have the supplied principles.  This would allow admins in impersonate users for testing as well.
3114e79a0f0f5d878b426f250ea50fef746ff9dc
You need to make sure the finPrinciples array includes fedoraUser... Forgot to do that.
Then update the DB sync get container function to use the service account token (see the keycloak module) and set the fin-principle header to discover

@jrmerz I have been testing sandbox, and this is still not working.  In short, dbsync calls transform services, and these services are calling the underlying fcrepo setup as public.
Here's the long description:
The gateway proxy divides fcrepo requests into proxyRequests or middleware.
Dbsync calls a transform service, so it goes to the middleware, to the transformService
to the renderTransform which execs the transform.
But here, we are calling a simple api.get(options) to  fetch the fcrepo data.
I'm not sure how you think best to solve this.  I can see that for (some) middleware, all the way back in the proxy,  it makes sense to call the normal proxyRequest, get the container, and then call the transform.  Alternatively, you could propogate the original request to that api.get request, the issue there is that request has already been run through the gateway, so I'm not sure if at that point you want to make a direct request, or just start the whole process over, and if the latter, do you just forward the fin-principal and Authorization, or a larger header set?
Examples:
This works, since we've set the principal
http http://localhost/fcrepo/rest/expert/66356b7eec24c51f01e757af2b27ebb8 Authorization:"Bearer $jwt" fin-principal:discover

This sets the principal to an unallowed user, and properly fails
http http://localhost/fcrepo/rest/expert/66356b7eec24c51f01e757af2b27ebb8 Authorization:"Bearer $jwt" fin-principal:foo

This should work, but also fails, since the service makes a public request.
http http://localhost/fcrepo/rest/expert/66356b7eec24c51f01e757af2b27ebb8/svc:node Authorization:"Bearer $jwt" fin-principal:discover


density_heavy test should create a load balancer per iteration

The density_heavy test currently creates a single load balancer at the beginning of the test. On each iteration, this load balancer has further VIPs/backends provisioned on it.
In actuality, we should not be creating a single monster load balancer, but instead we should be adding a new load balancer during each iteration of the test. This way, we will end up with a number of load balancers equal to the number of iterations, and each of those load balancers will have a small number of VIPs/backends.

Fixed by https://github.com/dceara/ovn-heater/pull/83


[Bug]:  test_develop UT failed

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Run UT failed
Expected Behavior
No response
Steps To Reproduce
No response
Environment
- Towhee version(e.g. v0.1.3 or 8b23a93):
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:

Anything else?
No response

The test try  to create an egg-link in the site-package path, which the user has no writing permission.

towhee develop should support creating the link in user's path instread of system path;
the unittest should use user's path by default;


Color Palettes for Ordinal Categories

As described in https://github.com/visualize-admin/visualization-tool/issues/101#issuecomment-973001152, we should offer sequential color palettes for ordinal dimensions.
Not sure if the selection has to be strictly limited to ordinal dimensions, as they may also work for dimensions that are sorted manually.
For reference, d3 offers these single- and multi-hue palettes out of the box: https://observablehq.com/@d3/color-schemes?collection=@d3/d3-scale-chromatic

Closing this as it is being tackled in https://github.com/visualize-admin/visualization-tool/issues/1722


Error while evaluating a Resource Statement, Duplicate declaration: Package[unattended-upgrades] is already declared

Affected Puppet, Ruby, OS and module versions/distributions

Puppet: 7.23.0
Ruby: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux-gnu]
Distribution: Debian GNU/Linux 12 (bookworm)
Module version: 8.0.0

How to reproduce (e.g Puppet code you use)
# Basic config for all Debian nodes
class profile::debianapt {
  # install basic packages
  class { 'apt':
    update => {
      frequency => 'daily',
    },
    purge  => {
      'sources.list' => true
    },
  }
  apt::source { "deb.debian.org-${facts['os']['distro']['codename']}":
    location => 'http://deb.debian.org/debian/',
    repos    => 'main non-free-firmware non-free contrib',
    include  => {src => true}
  }
# noch mehr apt sources

  class { 'unattended_upgrades':
    auto                   => {
      clean  => 'always',
      remove => true
      },
    enable                 => true,
    origins                => [
    'origin=${distro_id},suite=${distro_codename}',
    'origin=${distro_id},suite=${distro_codename}-security',
    'origin=${distro_id},suite=${distro_codename}-backports',
    'origin=${distro_id},suite=${distro_codename}-updates',
    ],
    remove_new_unused_deps => true,
  }
}

What are you seeing
error  about duplicate declaration when trying to run puppet agent --test
What behaviour did you expect instead
successfully configuring unattended upgrades
Output log
Any additional information you'd like to impart
I'm fairly new to both Puppet and Ruby, so this might just be an error I made and don't know how to correctly google.

What is the full error message? It should give you filenames and line numbers. Seems like this is coming from your code, because the package resource is only declared once in this module.


Fix broken twitter avatars (for real this time)

didn't notice avatarOptions object last time

Thanks @saimeunt!

Thanks! Seems to work :)


update hebrew, hindi, and multilingual goldens

Google fonts has updated the fonts.
TBR=@hterkelsen

Landing to unbreak the build.


Add local read benchmarks for PagedBlockStore

What changes are proposed in this pull request?
Added benchmarks for PagedBlockStore that read from local storage rather than UFS.
Why are the changes needed?
This piece is missing as PagedBlockStore didn't support creating local blocks then.
Does this PR introduce any user facing changes?
No.

@beinan @dbw9580 Would you like to take a look? Should be pretty quick.

Any benchmarking number you can share?

alluxio-bot, merge this please


load tests interfere with eachother

e.g.
Running the following passes:
poetry run pytest langkit/tests/test_injections.py -o log_level=INFO -o log_cli=true --load

but this fails some of the test_injections.py tests:
poetry run pytest langkit/tests -o log_level=INFO -o log_cli=true --load

Suspect we need some test setup/tear down helpers to reset udfs between tests. Might need more isolated ways of testing the various UDF configs so they don't affect other tests.

I think this is an actual LangKit/whylogs... bug? design issue?
The regexes.py module registers a UdfMetric for all columns of string type. injections.py registers a UdfMetric for the _prompt column (by name). A column can only have one UdfMetric instance, so there's a race condition for which UdfMetric gets applied to the _prompt column. The registration mechanism will unify all the metric UDFs registered to the same column name or type, but I don't think it can do anything to fix column name/column type races since the type is not known at registration time.
I'd suggest making injections a dataset UDF.

2 things here: 1. we should expect more conflicts like this, targeting specific columns is very useful in many cases and not just for langkit udfs, and 2. whylogs should support multiple udfs targeted to the same column, this seems like a bug on whylogs that needs to be resolved asap.

2 things here: 1. we should expect more conflicts like this, targeting specific columns is very useful in many cases and not just for langkit udfs, and 2. whylogs should support multiple udfs targeted to the same column, this seems like a bug on whylogs that needs to be resolved asap.

It supports multiple UDFs (all flavors) on a column. It does not support multiple instances of the same metric on a column, in this case the UdfMetric. The single UdfMetric instance can host lots of metric UDFs, but there can only be one instance of the metric itself per column. The problem is we were trying to stick 2 instances of the UdfMetric on the column, one by column name and one by column type. You'd have the same problem if you tried to create a schema that attached DistributionMetric by both column name and column type.

I'm increasingly leaning towards deprecating UdfMetric and metric UDFs as the other flavors of UDF grow more capable.

The load tests now pass when run altogether, there are some noisy errors/warnings logged which is related to this issue, but we can track the noisy output separately.


Use gnused in all scripts

Looking at sed -i compatibility between the GNU implementation and the BSD implementation it turns out there is no simple trick to get this right (see https://stackoverflow.com/questions/5694228/sed-in-place-flag-that-works-both-on-mac-bsd-and-linux).
I wondered why sed -ie 's/^cors_allowed_origins =.*$/cors_allowed_origins = ["*"]/' "${TMP_DIR}/config/config.toml" works in the Tendermint start script. Turns out this is pure luck. The e is interpreted as a value to -i and creates a config.tomle in the same folder. And the expression is not working because it is a value to -e but a positional argument.
So in order to avoid unnecessary confusion, we should use gnu sed for all scripts that use in-place and should work on Mac and Linux: https://github.com/CosmWasm/cosmwasm/blob/v0.14.0-rc1/devtools/set_version.sh#L5

Sound good to me


PWA-252 - fixes for the filter chips

correct max-height for filter chips wrapper
filter chips "more" button visibility is evaluated more often

Pull Request Test Coverage Report for Build 1088

2 of 7 (28.57%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 43.096%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pages/FilterAttribute/connector.js
0
1
0.0%

components/ChipLayout/index.jsx
2
6
33.33%

Totals

Change from base Build 1085:
-0.03%

Covered Lines:
1739

Relevant Lines:
4043

💛  - Coveralls


Add support for some RAW MIB Quad formats

For now, we support 1x1 and 2x2 layouts, with 1bit and 6bit counter depth. Support for other layouts and bit depths can be added on demand. Fixes #1135.
TODO

[ ] Compare performance (current master branch without the re-ordering vs. this one) and probably fix!

Contributor Checklist:

[x] I have added or updated my entry in the creators.json file
[x] I have added a changelog entry for my contribution
[x] I have added/updated documentation for all user-facing changes
[x] I have added/updated test cases
[N/A] I have included the rebuilt production build of the client (only if changes were made to the GUI)

Reviewer Checklist:

[x] /azp run libertem.libertem-data passed

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

/azp run libertem.libertem-data

Discussion with @sk1p: Performance could be  improved still, but "good enough" for now and not slowing down anything else. Merging, thx! :-)


Splitted image support?

And can it be like in a Halfix Emulator?

No plans at the moment, but a PR would be welcome

This is what I want, how I can split image like this?

The code for the requests starts here (but I don't think it's compatible with halfix): https://github.com/copy/v86/blob/5b79e47bc910d33507a9652e551db6cd9e21c7fb/src/browser/lib.js#L463
The server-side code for creating the part files is a simple cgi script: https://gist.github.com/copy/c8f6c27e090752f28aa318bec0719bab

The code for the requests starts here (but I don't think it's compatible with halfix):
https://github.com/copy/v86/blob/5b79e47bc910d33507a9652e551db6cd9e21c7fb/src/browser/lib.js#L463
The server-side code for creating the part files is a simple cgi script: https://gist.github.com/copy/c8f6c27e090752f28aa318bec0719bab

Bad question, but I can't run this CGI script. Is possible to rewrite script on NodeJS or Python?

Bad question, but I can't run this CGI script. Is possible to rewrite script on NodeJS or Python?

I don't have time, but PRs would be accepted. The spec is very simple: In order to read from byte i (inclusive) inclusive to j (exclusive) in file foo.img: foo-$i-$j.img.

Alternatively, you could add fixed-sized buffer support to AsyncXHRPartfileBuffer and generate files using split (more http requests, but doesn't require scripting on the server).

I wrote script on python (not tested on v86):
filename = 'boot.img'

ext = filename.split('.')[-1]
no_ext = filename[:-len(ext) - 1]

opened = open(filename, 'rb')

readf = opened.read()

file_length = len(readf)

step = 100

i = 0

while i < file_length - step:
    temp_file = open(f'out\\{no_ext}-{i}-{i + step}.{ext}', 'wb')
    temp_file.write(readf[i:i + step])
    temp_file.close()
    i += step

temp_file = open(f'out\\{no_ext}-{i}-{file_length}.{ext}', 'wb')
temp_file.write(readf[i:i + file_length])
temp_file.close()

opened.close()

This script works bad, so, I'm created realtime creating server:
from flask import Flask as FlaskApp
from flask import request as req
import os

PORT = 5000
DEBUG = True
HOST = '<IP_ADDRESS>'
folder = os.getcwd()
fname = os.environ['img_to_split'] if 'img_to_split' in os.environ else input('Enter filename: ')
os.environ['img_to_split'] = fname
temp_file = open(fname, 'rb')
readf = temp_file.read()
temp_file.close()

if not os.path.isdir('out'):
    os.mkdir('out')

app = FlaskApp(__name__, static_folder=folder, template_folder=folder)

def render(filename):
    temp_file = open(filename, 'r')
    read = temp_file.read()
    temp_file.close()
    return read

@app.errorhandler(404)
def error404(e):
    url_split = req.url.split('/')
    filename = url_split[-1]
    ext = filename.split('.')[-1]
    if not ext == 'img':
        return ''
    no_ext = filename[:-len(ext)-1]
    splitted = no_ext.split('-')
    temp_file = open('out/' + filename, 'wb')
    temp_file.write(readf[int(splitted[1]):int(splitted[2])])
    temp_file.close()
    print('Writed ' + filename)
    return app.send_static_file('out/' + filename)

@app.route('/')
def main_index():
    return render('index.html')

if __name__ == '__main__':
    app.run(HOST, port=PORT, debug=DEBUG)

Is possible to load files like:
start - only divisible by {step}
len - only {step}

I did it!
    const step_ = 4096;

    /**
     * @param {number} offset
     * @param {number} len
     * @param {function(!Uint8Array)} fn
     */
    AsyncXHRPartfileBuffer.prototype.get = function(offset, len, fn) {
        console.assert(offset + len <= this.byteLength);
        console.assert(offset % this.block_size === 0);
        console.assert(len % this.block_size === 0);
        console.assert(len);

        var block = this.get_from_cache(offset, len, fn);
        if (block) {
            if (ASYNC_SAFE) {
                setTimeout(fn.bind(this, block), 0);
            } else {
                fn(block);
            }
            return;
        }

        const fake_offset = parseInt(offset / step_) * step_;

        const m_offset = offset - fake_offset;

        const total_count = parseInt(len / step_) + 2;

        var blocks_ = new Uint8Array(m_offset + (total_count * step_));

        var finished = 0;

        for (var i = 0; i < total_count; i++) {
            const cur = i * step_;
            const part_filename = this.basename + "-" + (cur + fake_offset) + this.extension;
            v86util.load_file(part_filename, {
                done: function done(buffer) {
                    const block = new Uint8Array(buffer);
                    blocks_.set(block, cur);
                    const tmp_blocks = blocks_.slice(m_offset, m_offset + len);
                    finished++;
                    if (finished == total_count) {
                        fn(tmp_blocks);
                    }
                }.bind(this),
            });
        }
    };

This is merged now. Parts can be created using gnu split, see https://github.com/copy/v86/pull/483#issuecomment-947862665


[User Story] Balance SNV-calling inputs

Need
As a pipeline user, I want to better balance the input to DeepVariant, to allow for faster and more parallelised processing of samples. Currently, regions are input as whole to DeepVariant, e.g. chr1 is run in one process, chr2 in another. The bigger regions, e.g. chr1, will take longer time to process than smaller regions, e.g. chr22. This effectively means that the time of variant calling is limited by the size of the biggest region.
Suggested approach
Suggest we further divide chromosomes into smaller chunks, meaning the variant calling can be split into more even (and also just more) processes. This has the additional benefit up speeding up the CI test without BED file, since we could run 4 parallel DeepVariant processes.
Can be closed when

[x] #263
[x] #220

Blockers
No response
Anything else?
Changes the results of the SNV-calling ever so slightly. Sharding already happens inside DeepVariant so this should be no problem. No change in F1-score in HG004 chr22 against truth set.
A similar (although maybe more sophisticated) solution is implemented in sarek.

This would probably not compatible with genmod (?). Could consider making a custom bed for hg38 where regions are split for example at centromeres.


Feature/cleanup dashboard system basics

This PR takes care of #8557. Details are listed on the issue page.
It requires a companion Bedrock PR https://github.com/concrete5/bedrock/pull/50
Things that were done:

replaced all .btn-default with .btn-secondary
replaced all .pull-left and .pull-right with .float-left and .float-right
changed some markup on single pages
made sure success messages were set as "success" and not "message"
fixed icons for text and date/time attributes (FontAwesome changed their names and the polyfill was not working)
modified the getListIconElement() method of the FontAwesomeIconFormatter class to make all icons have the same width using the class name .fa-fw
Made the CKeditor plugin preview fancier with a "loading" text and spinner and a debounce function when adding extra plugins

Pages that were modified
/dashboard/system/basics/name
/dashboard/system/basics/attributes
/dashboard/system/basics/attributes/edit
/dashboard/system/basics/accessibility
/dashboard/system/basics/social
/dashboard/system/basics/social/edit
/dashboard/system/basics/icons
/dashboard/system/basics/editor
/dashboard/system/basics/multilingual
/dashboard/system/basics/multilingual/update
/dashboard/system/basics/timezone
/dashboard/system/basics/reset_edit_mode
Deprecated CSS class names
First buttons were using deprecated BS class names: pull-left, pull-right and btn-default so they were not aligned properly and not styled properly. All these class names were replaced with the new float-left, float-right and btn-secondary. Secondary buttons now have their border, and save buttons are on the right as should be.

The deprecated "hide" class name was used as well on the attribute edit page to show and hide a warning. So the hide/show functionality was not working. I replaced it with the proper d-none class name.
Markup
I had to modify the markup here and there to follow the new design and make sure things were aligned properly. For instance some headings were using label instead of fieldset legend.
Attribute dropdown & other issues
In the attribute page /dashboard/system/basics/attributes the attribute selection dropdown was not styled so I styled it.

To make sure it looked as good as possible I also had to modify the FontAwesomeIconFormatter  class to make use of the fa-fw (fixed-width) class name. This ensures all icons use the same width so they are all properly aligned and the text on the right as well. For instance, the icon for the "User Group" attribute is normally wider than the others so the text was pushed further to the right.
Another problem was that the text and the data/time attributes were still using deprecated FontAwesome icons so they were not being displayed. I could see the FA 4 shim was used but it didn't seem to work so I replace the icon names wit their new FA 4 names in each attribute's controller.
Success and error messages
Another problem was with success messages on single pages. Several of them were being set as "message" instead of "success" so they were blue instead of green. I fixed that and made sure error messages were displaying correctly.
I also had to replace the way error messages were set sometimes. For instance, I removed $this->set('error', [$this->token->getErrorMessage()]); and replaced it with $this->error->add($this->token->getErrorMessage());
Editor preview
The editor preview on page /dashboard/system/basics/editor had a few problems.
First, after clicking on the preview button, which turned green, there was no indication of anything happening while the editor was loading. I added a loading text with a spinning FA 4 icon.

Second, once the preview is active, checking extra plugins refreshes it. When clicking several checkboxes in rapid succession that was just too many server calls and the editor was flickering being reloaded on each click. So I used Underscore's _.debounce() function with a delay of 1500ms to only call the refresh once.
When clicking on checkboxes the loading messages is also displayed right away so the user knows something is happening.
I think both the loading indicator and the denounce make for much better user experience.
Other fixes and additions
Form elements were given a 20px left padding but non-form elements were not aligned the same way. So I added a new class name ccm-dashboard-content-indented that adds a 20px left margin and can be used on dashboard pages to align non-form elements properly.
While working on the social icons page I noticed all 3 concrete5 social links were using HTTP instead of HTTPS. Not a great look. So I modified elemental's content.xml to fix that.
Question
On the install languages page /dashboard/system/basics/multilingual/update, by default the list is not displayed. We have to click on the link at the top to show it.
In my opinion that makes for one needlessly really empty page and that link is not really a great indicator of what should be done here.
Couldn't we just display the list on page load and remove the hide/show functionality that's really not needed?
Weird Stuff
It seems the page /dashboard/system/basics/images was removed but the single page and controller files are still there. Should I delete them?
It seems BootStrap collapse is broken all over. Try it on the Install languages page for instance /dashboard/system/basics/multilingual/update
This is the error I'm getting
main.js:4237 Uncaught TypeError: Cannot convert object to primitive value
    at RegExp.test (<anonymous>)
    at HTMLDivElement.<anonymous> (main.js:4237)
    at Function.each (jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2)
    at S.fn.init.each (jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2)
    at S.fn.init._jQueryInterface [as collapse] (main.js:4231)
    at HTMLDivElement.<anonymous> (main.js:4291)
    at Function.each (jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2)
    at S.fn.init.each (jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2)
    at HTMLElement.<anonymous> (main.js:4286)
    at HTMLDocument.dispatch (jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2)
(anonymous) @ main.js:4237
each @ jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2
each @ jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2
_jQueryInterface @ main.js:4231
(anonymous) @ main.js:4291
each @ jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2
each @ jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2
(anonymous) @ main.js:4286
dispatch @ jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2
v.handle @ jquery.js?ccm_nocache=2af556aaa1bab5ebc0174f18be041b6bf3d4575b:2

Thanks @mnakalay - this is looking really really good. Just got a couple bits of feedback on this and then I'm sure we can get it merged.
Name and Attributes, Accessibility, Social Links, Bookmark Icons all look good.
Rich Text Editor: I like the Preview button, it looks nice and seems to work well. However there's something off about the list of Editor plugins. The checkboxes seem misaligned and don't have enough padding between them and the labels. Also something about the tooltip is going crazy when I hover over it; check out this screenshot:

Mutilingual: The checkbox under Login on this page doesn't look aligned properly and doesn't have the correct padding next to the label.  The Update Languages page looks strange (see this screenshot):

Time zone page looks great.
Reset edit mode page looks great.
Thanks! Here are some answers to your additional questions:

Yes, we removed the images page - can you remove the controller and single page like you asked? That sounds like a good idea.
Yes, this is a new bug because we introduced jQuery 3.5, which apparently has a conflict with bootstrap 4. I expect a BS4 update will come soon to fix this issue (hopefully?)

@aembler here are the changes and a few comments and interrogation.
Concerning the tooltips, there was a problem sitewide. The tooltip holder #ccm-tooltip-holder had position: relative set on it and all tooltips were misaligned everywhere. I removed it and it' working now.
There is a problem with checkboxes everywhere. There is no padding on them at all. When you see them with spacing between the checkbox and the label that's because there's a space in the markup not because of padding.
If we were to add padding we would have to make sure there is never any space in the markup between the checkbox and its text.
In the present situation, we have to make sure to add a space in the markup which is what I did for the editor page.
Really no perfect solution here.
Here's a screenshot of what it looks like now

Concerning the update languages page, I am not sure what the problem is? I added BootStrap classes to make it look better but that's all I could think of (screenshot below). What else should change?

About the checkbox spacing question: I'd avoid forcing some padding via CSS but use spaces (it's more natural: do you want a space? Add a space)

@mlocati I totally agree with you. It's better to say add a space when you need it rather than asking to make sure no extra space is added. Especially since indenting code with spaces creates spaces in the page anyway.

Oups sorry about that. I'm on it

I did the same mistake in #8648 (fixed by https://github.com/concrete5/concrete5/pull/8648/commits/ad21639928f52cc83f02424537c1cc1801243013).
The problem with this new Bootstrap 4 syntax is that you have to know the value of the id attribute of the <input> element since you have to explicitly write it in the for attribute of the <label> element.
This may not be the case for <input type="radio">, where the value of the id attribute is dynamically build; BTW thanks to https://github.com/concrete5/concrete5/pull/8645 you can now explicitly define this value explicitly (see for example https://github.com/concrete5/concrete5/pull/8648/commits/ad21639928f52cc83f02424537c1cc1801243013#diff-8ef2aee1732d4b7b364ef44a6ce1f465R53

@aembler I changed all the checkboxes to conform to BS4 rules.

Can you reopen a bedrock pull request with just your tooltip position relative changes?

I did it: https://github.com/concrete5/bedrock/pull/65

Hi, @aembler Sorry I got sidetracked on another project that got time-sensitive out of the blue.
I fixed the issue.
@mlocati thank you for taking care of that.

@aembler I'm sorry but I have absolutely no idea why all checks fail. A little help please.

Now that we've moved over to the develop branch, can you reopen this against the develop branch?

Yes I just saw your message in Slack about it. Will do shortly


Tests

Are there any tests?

Would love to figure out a way to do tests, but all this extension really does it emit events over websockets. It's up to the client side JS or extension to actually do anything.

I'm doing some issue-gardening 🌿🌷🌾 and came across this issue.
Sorry that this has lingered for quite a while!
If you have thoughts on how to improve testing, I'll happily review a PR! For now I'm going to close this issue though.
(this repo is maintained by a few people contributing their free time, unfortunately we can't solve every issue on our own, so help from others is needed for some of the more exotic issues that crops up)


Given a domain name, how do I find it's token id?

I want to take a domain name and find its corresponding token id.

With resolution-cli:
resolution namehash --decimal -d brad.crypto

If you want the token ID in hex format, just remove --decimal from the command

ah nice thank you so much!!!


chore(release): v1.1.0 🎉

:robot: I have created a release *beep* *boop*
1.1.0 (2022-04-12)
Features

hash: support hashes (70ee1aa)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/uladkasach/url-fns/releases/tag/v1.1.0 :sunflower:


Return local time if binance server time is not available

Return valid datetime object instead of None if binance failed to provide the server time
When the bot reads the time from the binance server and fails, it returns None, which breaks the downstream logic with time processing (because None is not datetime). Returning local time in case when the server time is not available should fix the problem.
Fixes #333
Type of change
Please make your selection.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] This change requires a new dependency
[x] Code Maintainability / comments
[ ] Code Refactoring / future-proofing

How Has This Been Tested?
Works well on my PC.
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added pytest unit tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ ] I have checked my code and corrected any misspellings

I was not sure, maybe the correct behaviour would be to keep requesting time again and again until I get one, but OTOH, the original failure might have been a result of temporary ban, and sending more requests will unlikely get things better. Using the local time instead was at least better than returning None.

@hebawom could you please approve again, I've found another piece of copy/pasted code of the same meaning, had to change that as well.


location of wslpath binary

I was trying to convert an Ubuntu installation to WSL2 and kept getting error
./bin/wslpath: Can't create 'bin/wslpath'
bsdtar: Error exit delayed from previous errors.

Your script moves the wslpath binary to /usr/bin/ folder. I had to move it to /bin/ to succeed.

It's not necessary to use this script anymore since there is a built-in wslpath tool in WSL2 distro.


WebSession: missing unique constraint for user.(nickname|email)

Prevent user creation with same nickname or email address that another existing user has.

There is no PR for this issue, hence closing it as per the legacy code base freeze; it is addressed in master code base differently.


Add 2.14.0 manifests

Description
Add 2.14.0 manifests
Issues Resolved
N/A
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@peterzhuamazon @gaiksaya @bbarani folks could you please take a look? Other repos stuck on 2.14.0 distribution availability ..

Hi @reta
OpenSearch has not bump the version yet https://github.com/opensearch-project/OpenSearch/blob/2.x/buildSrc/version.properties#L1
Thats why even automation did not create the manifests https://build.ci.opensearch.org/job/manifest-update/349/console
Core engines need to bump the version first so that new artifact is built.

Mind if I merge https://github.com/opensearch-project/opensearch-build/pull/4556 instead of this PR?
Looks like dashboards isn't ready yet.

Mind if I merge #4556 instead of this PR?

Sure!

Thank you!


changed the copyright message and also added apple store link

PR for fixing issue #124

Printed the year dynamically and also added the apple store logo for redirection along with the link in footer for issue #129

Failing to compile on local.

[ ] Ved Vyas Foundation should be a hyperlink to https://vedvyas.org
[ ] App store icon is too small

will check and update again

i have added a commit for the error and the hyperlink but couldn't check for the small icon issue as i cant run it on my machine

@eht-ck is picking this up in #150 PR so closing this for now.


Duplicate bind test fix: moved test to transport.wseb and exception assertion via ExpectedException rule

Duplicate bind test fix: moved test to transport.wseb and exception assertion via ExpectedException rule

Closing current PR.
The fixes addressed by the current pull request will be submitted within the following PR


Nerfs and adds the double bladed esword back to traitor uplinks

About the PR

This PR nerfs the reflection chance of the double bladed esword and adds it back to uplinks for traitors to purchase again for 20TC.
Why / Balance

Prior to it's removal, the most oppressive part of the double bladed esword was it's incredibly high reflection chance of 80%. With the recent reflection chances rework it was given a minimal reflect chance of 65%, which was a step in the right direction but probably wasn't quite enough of a nerf. I believe that nerfing the reflect chances further to 60% max. and 45% min. makes it balanced enough for a grand re-introduction into the traitors' arsenal.
Technical details

Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

:cl:

tweak: Nerfed the double bladed esword's reflection chances and added it back into traitor uplinks.

changelog is broken, needs 🆑 emoji

20% less reflection is not going to make this weapon less broken, it just makes it a little more rng.

this thing is 20 tc purchase and dual wield, sounds pretty balanced for me

this thing is 20 tc purchase and dual wield, sounds pretty balanced for me

that's the state it was in before it got removed because it was too strong.

this thing is 20 tc purchase and dual wield, sounds pretty balanced for me

that's the state it was in before it got removed because it was too strong.

-you're gonna wanna be moving with it since its a melee weapon so the relfection chance isnt always gonna be 60%
-its two-handed so you cannot just use it for passive protection while firing a gun or healing
-you cant use it to just stand still and cheese reflect since its 20tc so you cannot really buy any additional armor (unless you put in the effort to get it from elsewhere) and 60% reflect chance alone (unlike the previous 80%) isnt gonna keep you alive for long

Give him 100% reflect all or nothing melee weapon

IIRC it does >40 DPS so is incredibly strong.

IIRC it does >40 DPS so is incredibly strong.

its 36 (12 slash + 12 heat * 1.5 attack rate), which is only 6 higher than the regular esword

will there be bug fixes to the wielding of the double bladed esword too?

will there be bug fixes to the wielding of the double bladed esword too? or was that not one of the reasons the dsword was removed

There weren't any bugs iirc, it was just because 80% reflect at all times was bs. The wielding works just like with any other weapon (and the sword turns on/off automatically when wielded/unwielded)

will there be bug fixes to the wielding of the double bladed esword too? or was that not one of the reasons the dsword was removed

Iirc the real reason was the structural damage rework, and it doing too much damage against walls and windows. It was removed in that PR along with the Syndie Fire Axe

Spending all your TC for an esword that has 6 more dps and a little bit more reflect doesn't seem worth it imo.
You get so much more versatility out of getting an esword and having TC to spare.
Reflect is fine where it is but it needs more dps. Maybe bring It to 40-45.

until game will have melee blocking stuff doubleesword will be "overpowered"

We had a vote among the maintainers and admins. The poll options and results were the following:

merge the PR, adds it back to the uplink for 20TC (8)
make it surplus crate only (and worth 20TC) (10)
close the PR, no double esword (5)

The discussion was far to long to completely summarize, but the opinions ranged from "give people their edgdgelord weeabo sticks back!" to "it is a weapon that has historically been used exclusively to murderbone and been designed exactly for that", with everything in between. So I guess we will need some more time to think of an idea how to properly balance it before a second poll can be made. One suggestion was that it should be a sidegrade to the esword, not just a better version of it. If you have any ideas, feel free to let us know either here or on our Discord server.

Hey, we had another vote can came to the conclusion to give the new balance a try by making it only available randomly through the surplus crate (and worth 20TC). Could you adjust your PR for that?

Closing this as derelict.
Also we got another PR #32414 which replaces the dsword.

@slarticodefast would there still be an interest in having this in the surplus crate, or are we going for the replacement?

@Boaz1111
The last maintainer/admin vote on this had the following results:

merge (10 votes)
make it suplus crate only and worth 20TC (14 votes)
close PR (3 votes)

It might be best to wait and see if #32414 gets merged (and with what balance) since it serves as a replacement for the dsword.

@Boaz1111 The last maintainer/admin vote on this had the following results:

merge (10 votes)
make it suplus crate only and worth 20TC (14 votes)
close PR (3 votes)

It might be best to wait and see if #32414 gets merged (and with what balance) since it serves as a replacement for the dsword.

alrighty! I'll ask again after that PR's resolved!


[TF-TRT] Allow some unittests to run on both CPU and GPU environments

This PR allows some tests to run both on CPU and GPU operations.
Primary objective, not having them skipped with --test_tag_filters=gpu,-no_oss

@bixia1 can we get a review please ?

@DEKHTIARJonathan I updated the PR description and the PR title, please check.

@DEKHTIARJonathan Can you please check @bixia1's comments and keep us posted ? Thanks!

@DEKHTIARJonathan Any update on this PR? Please. Thank you!

@DEKHTIARJonathan Any update on this PR? Please. Thank you!

@bixia1 all comments addressed please review


invalid value encountered in cast

🐛 Bug

RunTime Warning after opening a new layer (Windows OS)
To Reproduce
Steps to reproduce the behavior:
1.Open a new layer by using the layer buttons
2.
3.

Expected behavior
No warning

Environment

Please copy and paste the information at napari info option in help menubar here:

Any other relevant information:

Additional context
Windows OS

Could you provide the output of napari --info or from the Help > napari info menu item?

Hmm, seems similar to:
https://forum.image.sc/t/adding-empty-shapes-layer-after-adding-rgb-image-causes-runtimewarning/75676

@VictorDidier We have a 0.4.18 release candidate! Please try to install it from pypi and see if this is fixed for you:
Either create an new env and then:
python -m pip install --pre "napari[all]"
Or you can try to update in place:
python -m pip install --pre napari -U

@psobolewskiPhD did you think that we could close this?


improvement: shell script improvements

Description
Shell scripts improvement with refactoring duplicated code.
Azure DevOps PBI/Task reference

AB#10887
Checklist

[x] Vehicle App can be started with dapr run and is connecting to vehicle data broker

[x] Vehicle App can process MQTT messages and call the seat service

[x] Vehicle App can be deployed to local K3D and is running

[ ] Created/updated tests, if necessary. Code Coverage percentage on new code shall be >= 70%.

[ ] Extended the documentation in Velocitas repo

[ ] Extended the documentation in README.md

[ ] Devcontainer can be opened successfully

[ ] Devcontainer can be opened successfully behind a corporate proxy

[ ] Devcontainer can be re-built successfully

[ ] Release workflow is passing

This is not relevant anymore and will be closed.


Add Alexa channel configuration documentation

Add documentation on how to set up the Alexa channel.

Change applied to the private branch for //build 2020. Closing this PR.


raidboss: TOP add config option + rotation to omega dodge

Because the rotation direction depends on which side you have dodged to, it's impossible to write this PR without having the user pick which dodge they are doing.
This promotes the dodge direction to a new option (defaulting to far, since that seems to be the PF approach) and adds in the rotation direction as well.
Because there's an option, there's no longer any combo output strings for staff sword, and the correct one is picked.

cc @Legends0 if you wanted to test this out too

cc @Legends0 if you wanted to test this out too

I'll try this out next time my group runs.

I had it call left then far E, but that was actually right. Using far dodge.

After resetting my config, it seems to have resolve itself as I haven't been able to replicate the issue.

This works well since the 5372 fix. Haven't had an issue with it.

Ok, I think I'll land this and do a release.  Thanks for all the testing.


JS: Loops in Javascript

Introduction to loops in Javascript:
This will describe the usage of different loops in Javascript
Directory:
Topics/4.Syntax/5.Loops

Description:

What are loops?
Different types of loops in Javascript

for
for/in
for/of
while
do/while

Formats:

[x] Document
[ ] Video
[ ] Audio

I want to work on this!! Please assign it to me

I want to contribute with documentation. Please assign it to me !!

May I have the honour of contributing to this issue, please?

I want to contribute with documentation. Please assign it to me !!

I would like to contribute to the documentation part of this task.
js batch 1
preeti kumari yadav


Fails parse of developer_data_id record without application_id

When trying to parse a fit file I'm running into trouble. This is what I get back running fitdump on master:
Traceback (most recent call last):
  File "./scripts/fitdump", line 92, in <module>
    main()
  File "./scripts/fitdump", line 86, in main
    for n, message in enumerate(messages, 1):
  File "/media/BigHDD/Home stuff/Dropbox/Dropbox/Personal/Programming/python-fitparse/fitparse/base.py", line 450, in get_messages
    message = self._parse_message()
  File "/media/BigHDD/Home stuff/Dropbox/Dropbox/Personal/Programming/python-fitparse/fitparse/base.py", line 163, in _parse_message
    add_dev_data_id(message)
  File "/media/BigHDD/Home stuff/Dropbox/Dropbox/Personal/Programming/python-fitparse/fitparse/records.py", line 367, in add_dev_data_id
    application_id = message.get('application_id').raw_value
AttributeError: 'NoneType' object has no attribute 'raw_value'

I believe this is due to fitparse trying to read a nonexisting field in a developer_data_id record. This file has only the developer_data_index and not application_id which fitparse seems to always want to read.

Thank you! Just out of curiosity (I'm not familiar with the code) - are you
are not fetching which blocks to read from the definition records in this
case or is this done automatically in some other place of the code?
On Mon, Aug 21, 2017 at 2:10 PM, Carey Metcalfe<EMAIL_ADDRESS>wrote:

Closed #35 https://github.com/dtcooper/python-fitparse/issues/35 via
bb54ac8
https://github.com/dtcooper/python-fitparse/commit/bb54ac80e54f47f9535136846634612dd3ab1181
.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dtcooper/python-fitparse/issues/35#event-1214036733,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALIdGNQbyxNt-hwbphMxLSMOrkB-_sXks5saXOrgaJpZM4O6w7i
.

Disclaimer: I'm not the original developer (that would be @dtcooper) or the person who wrote the developer extensions code (@nall) so I'm not the best person to be answering this but I'll give it a shot.
When parsing messages, if the type name is developer_data_id, a dev type is recorded with the data index (developer_data_index) and application id (application_id). If the type name is field_description, the data index is used to look up the existing dev type and define the fields of that type. The data is the automatically parsed using those fields in the same way as all the other predefined fields in the spec.
Does that answer your question?


feat(build): add htmlTerserOptions to minify html

close #3536

Description
Add build.htmlTerserOptions config which pass on to html-minifier-terser.
If this option is configured, the build output html will be minified.
Additional context

What is the purpose of this pull request?

[ ] Bug fix
[x] New Feature
[ ] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the Pull Request Guidelines and follow the Commit Convention.
[x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[x] Ideally, include relevant tests that fail without this PR but pass with it.

As the team discussion, the problem here is that this introduce terser and a bunch of other dependencies, while Vite is trying to be as lightweight and fast as possible, we probably don't want to have this.
While this feature makes sense, we would like to have a lightweight HTML minifier that only minify HTML but not other languages like JS/CSS. If you are interested in contribution, we are happy to have another PR for it. Thank you.

Got it!  I also think it is not such appropriate to introduce a bunch of dependencies.  Now we can also use minifyHtml from https://github.com/anncwb/vite-plugin-html .


Meta: Stream Manager

Already done in https://github.com/singularity-data/risingwave/pull/2563.

I see, are you working on #2557? If not, I am available to do it.

I see, I will close this


chore: Add NebulaGraph in Database category

Not sure if putting it as a green new badge is OK :).
Adding it to the database category helps expose its dd extension code repo(that may help the community!).
Thanks!

Thanks for the PR. LGTM.


Code cleanup

Fixes #1406

Move method calls out of the timer Stopped() event handler
Call Stop() instead of setting the timer's state via a property

Is this what you were thinking @DocMoebiuz ?


Copy and Update Logistic Information

@osg-cat
Unfortunately my grammar falls short of knowing where the curly quotes should be.
Otherwise I have made the suggested changes.


simp regression: simp? returns unuseful lemmas

Prerequisites
Please put an X between the brackets as you perform the following steps:

[X] Check that your issue is not already filed:
https://github.com/leanprover/lean4/issues
[X] Reduce the issue to a minimal, self-contained, reproducible test case.
Avoid dependencies to Mathlib or Batteries.
[X] Test your test case against the latest nightly release, for example on
https://live.lean-lang.org/#project=lean-nightly
(You can also use the settings there to switch to “Lean nightly”)

Description
universe u

class MyClass (α : Type u) extends LE α where
  sup : α → α → α
  le_refl : ∀ a : α, a ≤ a
  sup_of_le_left : ∀ a b : α, b ≤ a → sup a b = a

instance : MyClass Prop where
  le_refl _ := id
  sup_of_le_left _ _ h := propext ⟨Or.rec id h, Or.inl⟩

example : MyClass.sup False False = False := by
  simp? [MyClass.sup_of_le_left, MyClass.le_refl]

The above code gives ge_iff_le as one of the simp lemmas, despite this lemma not being useful or needed in the call.
Context
This is another instance of a regression in simp from Lean 3.
https://leanprover.zulipchat.com/#narrow/stream/287929-mathlib4/topic/simp.3F.20giving.20nonsense.20lemmas
Steps to Reproduce

Code as above.

Expected behavior: simp? returns the lemmas needed in a simp only call.
Actual behavior: simp? returns nonsense lemmas.
Versions
Version: 4.9.0-rc2
OS version: MacOS Monterey. Reproduces on live.lean-lang.org.
Additional Information
[Additional information, configuration or data that might be necessary to reproduce the issue]
Impact
Add :+1: to issues you consider important. If others are impacted by this issue, please ask them to add :+1: to it.

When we are inside a discharger, for some reason the if e == rhs then test in tryTheoremCore is not rejecting the rewrite by ge_iff_le. Outside of a discharger, it correctly rejects it.
I changed that test to
if e == rhs then
        trace[Meta.Tactic.simp.rewrite] "{← ppSimpTheorem thm}, == rejected {e} ==> {rhs}"
        return none

but even with pp.all true I don't see anything weird in the terms.

Sorry yes, my explanation was a bit garbled. Adding that extra trace line let me observe that ge_iff_le was tried twice, and rejected because of == only the second time.

Did you try pp.raw true? Maybe some .mdata or something like that.

Good suggestion, but there's nothing extra revealed by pp.raw.


feat: improved unlocking display

What this does
This PR resolves https://github.com/blockstack/explorer/issues/362 and https://github.com/blockstack/explorer/issues/361.
There is some logic on the API endpoint i've made to determine if an address has vesting data to fetch its true balance from the v2/account node endpoint, and the components will use that value when applicable. Note: this does not apply to if a user searches for the address in the search bar. The updated, correct information is only displayed on the Address page. Once the fixes to the API go into effect, then everything should be the same.
The lockup data is static, and should not change. You can view the data for it here.

Address view
Unlocking schedule view

Staging & example addresses

SP31AJM5WD8N9WB3HRCN5M2MA5QB14RF4ZYGEY9D7
SP2WPBNG1XR3HG95R91RVBYN4SADTBYZFFAV7Y4BQ
SPH4BRKR7J2BTHRZE2CPVYCSKPDXWP9EWSV61QBA
SP2Y1FXTR9BJH29VKQ35FKFRXC40149RB5T9TVN6J

todo

[x] Mobile view for unlocking schedule

the staging looks good to me! LMK if any other feedback would be helpful.

:tada: This PR is included in version 1.8.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Option to specify the source folder

Is your feature request related to a problem? Please describe.
Our source folder is called app instead of webapp. Therefore, we cannot use the ui5-linter.
Even with a direct file path, ui5-linter refuses to lint the file:
ui5lint --details --file-paths /home/xxx/dev/project/app/folder123/business-objects/business-objects.controller.js
Error Message:
File path /home/xxx/dev/project/app/folder123/business-objects/business-objects.controller.js is not located within the detected source or test directories of the project
Describe the solution you'd like
There should be an option to specify the source directory. Or at least a way to forcefully lint a file using a direct file path without being patronized by the ui5-linter.

You should be able to workaround this issue by creating an ui5.yaml configuration file in /home/xxx/dev/project (parent of the app folder) and specifying the alternative webapp folder name in there:
specVersion: '3.2'
metadata:
  name: app
type: application
resources:
  configuration:
    paths:
      webapp: app

In general, UI5 linter is intended to work with most projects out of the box without the need for a ui5.yaml or usage of UI5 Tooling. However, this only applies when the expected folder structure (webapp for applications, src / test for libraries) is used. Beyond that, we recommend to create a ui5.yaml configuration as outlined above to make use of UI5 linter.

I will try that, thank you.

@xLexip did my proposal work for you?

@matz3 it did, thanks.


Continuous Integration Infastructure

Currently, crystal uses Travis CI for continuous integration. This works well, but has some limitations. Travis  currently allows us to test on our major architectures: linux 64 and 32 bit, and macos. However, in the past year we have gained ARM support in 32 and 64 bit, as well as support for freebsd/openbsd. These architectures would be difficult to test using travis. Without continuous integration on a target triple, that triple is essentially unsupported and could break at any time. In addition, travis lacks the ability to do automated releases. This makes the release process more error-prone and precludes the ability to do nightly releases.
I have been working on setting up Jenkins as a replacement for travis. Jenkins is a much more flexible system, as it allows connecting your own nodes with their own customised build environment. For example we could test crystal on an actual raspberry pi for every commit. We could also schedule jobs to create nightly builds, and authorised users on the web interface could kick off an automated release process.
Currently I have a test jenkins instance running at https://crystal-ci.rx14.co.uk/, here is a (nearly) passing build. Jenkins builds can be configured by a Jenkinsfile in the repository like travis. Here's the one I made for crystal. I've documented the setup for the master and slave instances here. Currently i'm thinking of running every slave in qemu/kvm on a x86_64 host for consistency between slaves. Automating slave installs using packer seems trivial.
There are quite a few different options for jenkins slaves however. It's possible to create jenkins slaves on the fly by integrating with different cloud providers. This has the added benefit of the environment being completely from scratch on every build. It also may be cheaper, depending on build length, commit frequency, and hardware constraints. It might also be wise to mix this and the previous approaches, for example using some raspberry pis for arm, a long-running VM for openbsd, and google compute engine for the x86 linux targets (musl in docker?).
Rust seems to use buildbot instead of jenkins, but Jenkins has really surpassed buildbot in the last year in terms of being a modern tool suitable for non-java builds (released 2.0, added jenkinsfile, seamless github integration like travis). I also have 3-4 years experience working with jenkins, but have never worked with buildbot before.
The problem I have in proceeding is that I don't know the options and preferences @asterite and Manas have in terms of infrastructure and how they would like this set up.
TL;DR: CI on every target triple? Nightly builds? Yay! Now how do I proceed?

@RX14 This is awesome! Every point you say is a +1 in comparison to running things in travis, at least for a project this big and with many platforms to support. I'll try to discuss this at Manas next week and see how we can proceed.

In addition, if we have macos install media, it seems we can use https://github.com/boxcutter/macos to create a macos VM, so it looks like we really can test on every triple using jenkins.

Any progress on this issue?

@RX14 We didn't have time to review this yet, some of the team is on vacation at this time of the year (it's summer here ^_^) but this is something that we'll definitely take a look at, and probably switch to, once we discuss it.

The analogy in Ruby is rubyci.org running builds tests and specs on multiple platforms in the background' whilst travis and appveyor work in the foreground' on marker versions giving `immediate' feedback to pull requests and branches. This works well since edge cases turn up constantly across the different platforms whilst travis usually catches low lying fruit. Ruby use chkbuild - a CI server written in Ruby. Apologies if you know all of this - I'm a contributor to Ruby/Spec and thought I would share. TLDR :+1:

I think I would strongly prefer running builds for every platform for every commit. Whether the PR is "okayed" back to github after only the faster builds have completed is a question which I think will have to be answered after it's all set up.

Hey @RX14, so sorry for the delay in the reply. We discussed this internally at Manas, and we agree that having a Jenkins (or equivalent) environment for managing the builds in multiple platform would be ~nice~ awesome to have. If this is something you'd like to work on, and feel comfortable in Jenkins, then Jenkins it is; we do have some experience in Jenkins and none in buildbot, so it seems the best choice to go.
The first thing to work out would be hosting. We'd prefer to handle as much as possible of the infrastructure in-cloud, having a master Jenkins node running in Amazon (where we host most of our assets), and slaves covering all the required platforms.
So step one is to build a list of all targets, and figure out the best place to run them. I guess that a node in EC2 running qemu (I understand there are no limitations for running qemu on an EC2 machine, right?) would be a good choice for most architectures, though I'm not sure if we can cover all of them this way.
What do you think?

Architectures should be tested on real hardware if possible. I discovered bugs when running Crystal on a AArch64 server (provided by Packet) that didn't happen in QEMU for example. It's also very, very, slow.
Scaleway has cheap ARMv7 servera, Packet have expensive but incredibly powerful ARMv8 servers (Cavium ThunderX × 2).
Same for Alpine, running in a container will be different than running in a VM, because the Alpine kernel is patched (grsecurity, ASLR, ...).

@spalladino absolutely no problem with the delay in reply.
Obviously i'd like to automate as much of the deployment of the master as possible. Docker would be my first choice as I have experience with it (and already made a container). The master node doesn't need to be that powerful. Needs probably only 2gb of ram and somewhat decent processor. We can assess if it needs more resources as the project grows.
The list of targets is here: https://github.com/crystal-lang/crystal/tree/master/src/lib_c, we'll want something for every one of those in an ideal world. I don't want to run any targets in qemu if possible, but acquiring hardware for every single target in the future is impossible. In hindsight every non-x86 slave is going to end up being a special snowflake, so there's not much point in attempting qemu for every slave.
Build slaves don't really need a particularly good connection, so they are fine to be run at home if it comes to it. For example I'd like to run the ARM targets on a raspi as that's realistically going to be the most common device they're run on. Raspi 1 is ARMv6, raspi 2 is ARMv7 and raspi 3 is ARMv8, and I know that aarch64 distros exist for the raspi. Raspis are quite a cheap single-cost and racking solutions for raspi exist so the slave hardware for ARM seems to look bright in that direction.
x86_64 slaves should be able to be run in the cloud with ease. Even if they need to be virtualized, KVM should ensure that they're running on a real CPU most of the time using VT-x. Architectures which you can get an AMI for can be run directly on ec2. AFAIK ec2 doesn't use containerized virtualisation so we should have full control over kernel versions etc for alpine.
Another interesting question is that of LLVM versions? Which LLVM version do we test with? Testing with every LLVM version on every architecture seems like a waste of time. We could pick a random LLVM architecture for each run (maybe deterministically from the commit hash).
@spalladino This is a topic which would probably benefit from real-time discussion, so don't hesitate to ping me on IRC/gitter if you want to chat.

For LLVM, we should test a few versions. It's usually overkill, but we may break compatibility when supporting a new LLVM version or when the compiler uses more features. So we may:

use the stable branch by default (3.8, soon 3.9);
have 1 VM with the qualification branch (3.9, soon 4.0);
have 1 VM with the oldest supported release (3.5, soon 3.8?);

Note that ARM / AArch64 have best results with 3.9 (maybe 3.8 is enough); older versions lead to crashes in release mode for example.
BTW I couldn't find any AArch64 distributions for Raspberry 3 (only ARMv6 or v7). It can boot in 64bits mode, but when I searched a few months ago, there was no kernel (only preliminary attempts). I'd love to see a distribution, thought. Note that Packet was willing to sponsor us with an ARMv8 server, I can ping them back (or send a DM to @packethost on Twitter).

How about only running tests on all LLVM versions nightly? I'm a little wary of build times for PRs slowing down development.

What about nightly tests for all config's, and 'live' tests [per PR or per merge-to-xyz branch] only on one/few select config's?

@RX14 What about primary builds that will report a green state as quickly as possible, then additional builds that will report more feedback? Or maybe have a trigger in commit messages, to enable LLVM builds (eg: matching /LLVM/i)? Usually we shouldn't care about LLVM except for particular branches / pull requests (supporting new LLVM version, using new LLVM features, ...).

@ysbaddaden That could work. Trying things out should be quite easy once we have the infrastructure set up.

I agree on having primary builds that can report a green state as quickly as possible. I'm not sure though about how to handle the additional builds: I think I'd start with nightly builds of master (as @RX14 suggested). Later, we could set something up to auto-merge a PR if all builds have passed and a committer has given a thumbs-up, having a quick primary build report an initial ok state.
Anyway, first we need to have the builds running somewhere, so let's go back to the hosting.
I wasn't aware of potential issues when running in QEMU as @ysbaddaden mentioned, so I guess that rules QEMU out (speed would not be much of a problem if we are using them for nightlies or "extra" builds).

For any Linux on x86_64, an EC2 node sounds like the best option. Should we use Docker for generating different isolated different environments within the same node?
For Windows on x86_64 (when it's ready), a Windows EC2 node should do the trick; though we could check if the cost in Azure is lower.
For Mac OSX, there seem to be cloud providers for Mac, such as this one, we should look into them and choose the best in terms of pricing
For ARMv8, if we can have support from Packet, that would be great. @ysbaddaden would you mind contacting them again? Feel free to CC me or @bcardiff to discuss sponsorship details. Also, if they are open to offering support for other architectures and we can reduce the cost from Amazon, all the better.
For other ARMs, I'd like to avoid having a rack here at the office; but if there is no cloud provider available, we'll go down that route.

Regarding LLVM versions, I'd pick a primary LLVM version for each platform for the primary builds, and then add other combinations (as mentioned by @ysbaddaden) as additional builds.
Am I missing something? What do you think? I'll be in the IRC in a few minutes if you want to follow up there, though I'd rather keep the conversation here (just for the sake of keeping the history more organised).

Maybe branches like [but possibly renamed if desired]:

master [for releases]
nightly [for nightly scans across all configs], which would be merged into master if all pass
(all others) [for 'live' scans on primary configs and (as @RX14 suggested) other configs when trigger in commit messages or in PR], which would be merged into nightly if all pass

@drhuffman12 I don't think that changing the git repo setup would be a good idea.

Note that the only reason we keep compatibility with older LLVM versions in the source code is because for our release process we are stuck with LLVM 3.5 for the moment, but that should be upgraded to the latest LLVM version (and upgraded each time LLVM lands a new version)... but that's kind of tricky to do, as far as I know.
So in my opinion, I wouldn't have a matrix of LLVM versions to test against. In fact, once we upgrade the omnibus build to the latest LLVM version I would directly remove Crystal code that deals with older LLVM versions.

Also, old LLVM versions have bugs, so shipping Crystal with support for an older LLVM version means shipping a buggy version of Crystal... so that's another good reason to drop support for older LLVM versions.

Supporting the LLVM stable (3.8) and qualification (3.9) branches simplifies building on many distributions. Alpine and OpenBSD ship LLVM 3.8 for example.
It's not that hard to have compatibility for 2 or 3 LLVM versions; despite the breaking changes, the C API goes through a deprecation release before removal in the following release. The current complexity is that we now support 5 versions (3.5, 3.6, 3.8, 3.9, 4.0pre) which totals many breaking changes...

@spalladino we can still use QEMU, it's very good. I wouldn't have ported Crystal to ARM without it, but it's still an emulation, not real hardware, and there may be some quirks that it doesn't exhibit. Maybe not that much, though.

Got it. Should we take longer than expected to set up real hardware, we can rely on it for the time being then.

@ysbaddaden I think for now we should be able to get real hardware for each build slave (raspi+aws), but if we port to more unconventional architectures in the future, I think qemu will be required for those.

For now the first step would be to make a few tests with a master node and a slave node in a rather standard architecture, and see whether we want to keep slaves running or use jcloud to start and terminate them on the fly, and check if it works to use multiple AMIs for isolating configs (or we need to rely on Docker or similar). Chris will be kindly making a few experiments during the weekend and we can pick up from there.

I couldn't find any AArch64 distributions for Raspberry 3

I stand corrected, it appears openSuse has one: https://en.opensuse.org/HCL:Raspberry_Pi3

@ysbaddaden Thanks for the link! I think that archlinux-arm has an AArch64 distro for raspi 3 too (scroll to bottom): https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-3

Just a note here - CircleCI offers macOS builds for Open Source projects. We should probably ping them when times come.

I've done a bit more work on the CI, and got the AWS slaves launching. For example this build was performed on a tempoary build slave provisioned on AWS. The build slaves stick around for a configurable time period before being terminated.
The AWS builds are run on t2.medium instances, utilizing a cutom AMI built using packer. Currently the AMI only includes llvm 3.5, so the build fails. Over the weekend I'll install multiple llvm packages (3.5, 3.6, 3.8, 3.9, 4.0 seem to be available) from the more cutting edge debian repositories, and then set up a large matrix build. I also need to figure out multiarch to run 32bit builds.
Therefore, next steps seem to be:

[ ] Provision AMI with multiple llvm versions + 32bit environment
[ ] Move Jenkins master + AWS account from my personal account/domain
[ ] Set up nightlies to build on larger matrix (release + non-release builds for every architecture, possibly on every LLVM version)
[ ] Enable side-by-side with travis on crystal-lang/crystal!

I'm now building in parallel with LLVM 3.8, 3.9 and 4.0! See a working build here.
Unfortunately, I've had to upgrade the instances again to t2.large, so that they have 8gb ram. compiling all_spec with only 4gb of ram seemed to get OOM killed. Travis's sudo: required builds seem to have 7.5gb ram available, and maybe OSX builds are just more efficient.
As for moving the Jenkins master and slaves, I'm ready for that to happen. The Jenkins master needs a box with say 2gb of ram and not much CPU at all. This doesn't have to be on EC2, but if it was I'd recommend a t2.small. For the slaves, I just need a AWS user with this IAM role (however a full user account may be useful for debugging).
Again, just ping me on IRC/gitter if you want a chat about any details I may have missed out :)

Awesome, thanks Chris! I'll contact you during the week to get everything
set up. Cheers!
Sent from my mobile
On Feb 25, 2017 3:05 PM, "Chris Hobbs"<EMAIL_ADDRESS>wrote:

I'm now building in parallel with LLVM 3.8, 3.9 and 4.0! See a working
build here
https://crystal-ci.rx14.co.uk/blue/organizations/jenkins/crystal/detail/feature%2Fjenkinsfile/37/pipeline/
.
Unfortunately, I've had to upgrade the instances again to t2.large, so
that they have 8gb ram. compiling all_spec with only 4gb of ram seemed to
get OOM killed. Travis's sudo: required builds seem to have 7.5gb ram
available, and maybe OSX builds are just more efficient.
As for moving the Jenkins master and slaves, I'm ready for that to happen.
The Jenkins master needs a box with say 2gb of ram and not much CPU at all.
This doesn't have to be on EC2, but if it was I'd recommend a t2.small.
For the slaves, I just need a AWS user with this
https://gist.github.com/RX14/2922b8c6f96fc64a41644cc03a015917 IAM role
(however a full user account may be useful for debugging).
Again, just ping me on IRC/gitter if you want a chat about any details I
may have missed out :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/crystal-lang/crystal/issues/3721#issuecomment-282501169,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAaOJDcjvsfMujJtIuwz78OxheRweGMMks5rgG1MgaJpZM4LP4_F
.

@RX14 a few questions:

Is it possible to configure the jcloud plugin to request EC2 spot instances? These work by bidding on compute time, thus making them more affordable though not immediately available, but it could work well for nightlies for instance.
Would it be feasible to set up a complex build matrix for nightlies, though test in fewer configurations in a per-commit basis?
Since the build is really fast (10min compared to Travis' 20), and since EC2 bills each fraction as a full hour, is there a chance to run more configurations in serial instead of parallel? For instance, is it easy to reuse the Docker images used now for testing to check for different LLVM configurations, and use different AMIs for changing just the platform?

Again, thanks a lot for all the work with this! It's looking really awesome!!

There are other cloud providers that are much more affordable than AWS EC2. Take a look at https://www.ovh.ie/vps/vps-ssd.xml for instance. This listed pricing is whole-month price. But they also charge per hour: monthlyprice*2 / 720 per hour. It could be worth it.
They have an API, but this docker-machine plugin could be used: https://github.com/yadutaf/docker-machine-driver-ovh

@lbguilherme we could look into other cloud providers in the future, however ec2 is a good start because it has an existing mature Jenkins plugin so it's quite easy to set up. The actual machine provisioning script only assumes a basic debian 8 base, so could be used to create images on numerous clouds. A cheap and simple option for decreasing some load on the Jenkins would be to run builds on containers on developers machines (thinking more desktop than laptop though). Although that route is likely to cause more pain on the communication and operations side than it saves on the money side.

@spalladino If you have the keybase app (kbfs) installed, my current ssh public key is at /keybase/public/rx14/id_rsa.pub. If not, I'll send you an email.

@RX14 I refuse to start this message without thanking and clapping out loud your work, so here it is: 🙇👏🙇👏🙇👏🙇👏🙇👏🙇👏🙇
Now, back to business 😛
Do you know if it's possible/difficult to hook different providers side by side with EC2?
Last week I've tested CircleCI for OSX builds. It's a 14-day trial, but it seemed to work really nice. More specifically, it didn't have any queue time when building - and that's what's killing us in Travis.
We've been discussing with @spalladino the possibility of running the per-commit builds using Travis and Circle (Linux x86_64 with LLVM 3.5 & 3.9 on Travis, OSX with LLVM 3.9 on Circle), and maybe run nightlies on EC2. By splitting the workload between the providers we get quick feedback - Travis doesn't usually queue the Linux builds, and neither does Circle for OS X -, and we get lighter bills from EC2.
But, of course, this depends on a couple of things - mainly, in the feasibility of hooking your Jenkins setup with Circle/Travis, and a little bit on Circle's kindness in giving us some nice free plan.
We're working on the monetary side of this - we'll get back to you soon, I hope. If you can give us some thoughts about using Travis/Circle mixed with EC2, that'd be awesome :)
Thanks, once again!

In regards to running 2 workers per slave, I've found that even just running a compile and a spec process at the same time can run out of memory. In addition, I was getting some weird errors (1) (2), even when setting CRYSTAL_CACHE_DIR.
I'm working on a reply/thoughts to what you said @matiasgarciaisaia, but CircleCI/Jenkins integration will be minimal if possible at all. Can you expand on what sort of "hooking up" you would want between Circle/Jenkins.
One thing major thought is that the vast majority of the builds will either fail on all platforms or none, so having a quick build script which simply runs make std_spec crystal, tests the examples, and checks formatting would be great for the 99% case. The question is then whether we should allow the 1% type of commit onto master and catch it in the nightly, or test every commit on an expanded set of architectures. Maybe we should "opt in" some PRs which touch a lot of architecture stuff to run every commit on a large matrix. However with such a system we gain a lot of complexity.
Just some thoughts.

We were thinking about Jenkins not performing the Linux x86_64, Linux i386, and OSX builds, but consuming its OK/FAIL status from TravisCI/CircleCI jobs as the ones currently running. So Jenkins runs the jobs for the other platforms, and provides something like a dashboard to see how the builds are going, but delegates the actual build to third parties for a couple of platforms.
We want Jenkins, also, because we should eventually plug some (physical) RasPi's and things like that to also build on those platforms, so Jenkins would give us the flexibility of having whichever slaves we want, while we keep outsourcing the hardware requirements for platforms that can easily be outsourced.
But my 30-second Google-search suggests there's no such thing as a Jenkins/Travis connector or whatever I was dreaming of :/

The question is then whether we should allow the 1% type of commit onto master and catch it in the nightly, or test every commit on an expanded set of architectures. Maybe we should "opt in" some PRs which touch a lot of architecture stuff to run every commit on a large matrix. However with such a system we gain a lot of complexity.

How about looking for a keyword on the commit message(s) for triggering a full matrix build, as is done with the [ci-skip]? So, if a commit is particularly sensitive, we can add a [fullbuild] (or a better name) to the commit msg. If we miss it, it would still be caught as part of the nightly.

@spalladino I think it's better done using github comments by team members. That used to be how Jenkins approved building PRs before 2.0 and pipelines. It doesn't seem to be possible now in the pipeline plugin. Also, encoding metadata in commit data (especially the title) is quite ugly.

Absolutely, I agree that github comments are much better; I was aiming at
commit messages as I expect them to be easier to implement.
On Fri, Mar 3, 2017 at 1:42 PM, Chris Hobbs<EMAIL_ADDRESS>wrote:

@spalladino https://github.com/spalladino I think it's better done
using github comments by team members. That used to be how Jenkins approved
building PRs before 2.0 and pipelines. It doesn't seem to be possible now
in the pipeline plugin.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/crystal-lang/crystal/issues/3721#issuecomment-284004508,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAaOJI-6mb7O7H1P-PZansPWIt6tRrKOks5riELggaJpZM4LP4_F
.

--
Santiago Palladino
Manas Technology Solutions
[ar.phone] 4796.0232
[us.phone]<PHONE_NUMBER>
[email]<EMAIL_ADDRESS>[web] man.as/spl

I think that the best thing to do is to use CircleCI or travis for an "initial smoketest" which runs on every single PR. This initial smoketest would be as simple as make std_spec crystal, followed by testing samples and crystal tool format --check.
A small bot can then be written (in crystal!) which looks at github issue comments, and schedules jenkins builds using the jenkins API. Jenkins builds would run a full matrix validation suite. I'd argue that we should run full validation on every commit to master as well. Using a custom bot has the advantage of somewhat decoupling the interface to CI from the implementation so we can possibly make the CI more fine-grained (run just windows/osx/mac) in the future. This kind of setup seems very similar to what swift has.
But first, we should get something simple and useful working, I suggest this: build every push to master, on several LLVM versions, using EC2. From there we can evaluate the cost of running on EC2 and what optimisations are worth the effort. I do somewhat feel we're getting lost in the details and optimizations instead of getting something working...

Ok, I've set up the Jenkins master on the new server, and written some reproducible instructions on how to set it up here: https://github.com/RX14/crystal-jenkins/tree/master/master. The Jenkins is now live at https://jenkins.crystal-lang.org/. Next steps: get EC2 credentials and configure Jenkins with them (and document), and set up the actual job (and document).
I'll try to focus on good documentation for this infrastructure, because I want the infrastructure to be well-understood even if i'm busy. If you have any questions or suggestions, don't hesitate to ask me to improve the documentation. Also, do you think moving RX14/crystal-jenkins to the crystal-lang organization would be a good idea?

@ysbaddaden - there is also a new ARMv8 Debian build for Raspberry Pi 3 at https://blog.hypriot.com/post/building-a-64bit-docker-os-for-rpi3/ which you might find of interest.
I'm working with Packet on getting their ARM infrastructure together for CI builds, and will bring this issue to the attention of the team here that's doing this work.

Build is set up on the new infastructure and (somewhat) working: https://jenkins.crystal-lang.org/blue/organizations/jenkins/crystal/detail/feature%2Fjenkinsfile/1/pipeline
Next steps include fixing https://github.com/crystal-lang/crystal/issues/4089 and merging the completed Jenkinsfile (i'll PR it soon). Once this is done we can start running status checks on master and PRs.

:)

I've very nearly got 32bit support working on the CI, but i've hit this problem: https://jenkins.crystal-lang.org/job/crystal/job/feature%252Fjenkinsfile/11/execution/node/13/log/. It appears to me that my -m32 link flags aren't being passed to cc on macro runs. This is exactly what you want when cross-compiling typically, but as my libcrystal.a has been compiled using -m32, this simply doesn't work. It looks like we need another kind of link flags which get passed to every cc invocation. @asterite what are your thoughts?

Hey @RX14!
I think I agree with you, at least in this case - the link flags should get forwarded to the macro run. I'm not sure, however, if there's any scenario in which you want those flags to not be passed to the macro run.
I'm still trying to reproduce this issue - have to set the environment up and whatever - because I can't see it from the code. The only mentions there are to CC or cc are in compiler.cr, and they all include the @link_flags.
But please do fill the issue, so we can track it down 👍

@RX14 Something I don't get is, in the current ci there is a 32 bit environment and the specs are passing there. Why a change in the infrastructure would trigger the behavior of passing the link flags? I am not discussing if it should or not (I am not fully convinced), but I wouldn't expect that to be an issue since there a 32 bits environment in travis running. What has changed?

I think Jenkin's a 64 bit build, and so the issue would be "Cross compiling from x64 to i386 is broken".

I would mimic the release process. i.e.: using the latest 32 release to compile the next 32 release. That also match what the user will be doing if contributing to the compiler from a 32 platform.
I don't think the flags (at least the -m32) should be forwarded, because the macro run will execute the compiled program in the original environment so, in this case, the compiler used for macro run be setup for 64bits.
From the Jenkins log I guess you are trying to cross-compile the specs, there the macro run is used, which end up calling the just cross compiled compiler with a libcrystal.a for 32 bits. Again, I would just compiler the compiler and the specs in 32 bits without cross compilation. But, if not, do not cross compile the the specs at least. I think that should work.

@bcardiff In the current CI there are 2 docker containers, one containing a 64bit filesystem, one containing a 32bit filesystem. This means that the linker itself is 32bit by default, you don't need to pass -m32. It still executes in a 64bit kernel on travis.
In the new environment, I didn't want to spin up 2 VMs for 32 and 64bit, or use containers (added complexity), so I chose to use debian's multiarch features. This seems to work very well, apart from when macro runs are required. It turns out though that this is impossible anyway because libevent isn't correctly packaged for multiarch so 2 VM images are required.
Using a 32bit crystal release of crystal in the 32bit builds would be a good idea. I think that in the future I would like to make downloading a crystal release part of the build process, instead of baking in the crystal version to the VM image. But I don't think that that should require 2 VMs, or to create a chroot. Crystal should be able to utilise multiarch features to cross-compile for 32bit. Someone will want to do it in the future, so I think we should support it.
I ended up passing -m32 by setting CC=cc -m32, which is probably actually the recommended solution to this problem, however it seems the macro runs don't pass the target to the new compiler instance. I added a quick commit which probably fixes the problem (https://github.com/RX14/crystal/commit/ad941fb28e4cf057f027821ddf3e0f05dfb02e4f) but it's a hack I can't test and it's looking more and more like this is a dead end.
Unfortunately there are no official debian 32bit AMIs, so i'm going to have to think about how else to do this :(

Debian don't provide 32bit AMIs, with a message telling you to use multilib, which is what I tried to use in the previous comment and failed. Getting it to work would require compiling the compiler for x86_64 before compiling for 32bit. This is difficult as it would require both 64 and 32bit development versions of the libraries used, which is difficult in certain cases as some packages still aren't fully multilib compliant. In general debian's 32bit support seems a huge mess right now.
I could try creating a 32bit AMI using bootstrap-vz but there isn't an official manifest for it. If that doesn't work out, I'll have to think of something else.

Ok, building a 32bit AMI was easier than expected, I even nearly completed a whole 32bit build: https://jenkins.crystal-lang.org/job/crystal/job/feature%252Fjenkinsfile/18/console. It fails to build the compiler due to some linking errors, however. Any suggestions?
I think the next steps are really to merge the Jenkinsfile and get nightly builds going, gradually explanding the available architectures. After that I think i'll try and get PR builds working, which I will probably end up building a bot very similar to rust's bors, but using the jenkins api, passing targets to be built and git sha as build parameters. Thoughts?

I've added swap space and used --threads 1 to control the memory usage on 32bit. It seems to be working. We now have a full matrix of 32 and 64bit debian using llvm 3.5-4.0. I've created a crystal-nightly job which currently builds my fork nightly.
Next steps are to merge the jenkinsfile to get nightly builds kicked off and get notifications sending on failed builds so that the nightly results don't get ignored. That includes deciding which matrix we should include in the nightly builds.
A weird bug seems to have been exposed though: both 32 and 64bit LLVM 4.0 builds failed with linker errors. A build log is here.

A weird bug like https://github.com/crystal-lang/crystal_lib/issues/25 or https://github.com/crystal-lang/crystal/issues/1269 ?

Arrgh, this is why I need to finish crane and use it on the CI, so that we have a native crystal install and ditch the omnibus.

Is there any progress on this? It would be great to have nightly builds. I think it would help avoiding releases requiring an immediate followup bugfix release when changes in master can be more easily tested in the wild ;)

Totally agree. One thing that's certainly missing is a wiki page to specify what kind of support we're providing for each target. I'd like to basically "translate" this page to one in the Crystal repo wiki: https://forge.rust-lang.org/platform-support.html
This page should state our initial goals and not where we currently are, so we can use it as guidance to inform this issue.
I won't have time to start that until the end of next week, so if anyone wants to take a stab at it before, we can iterate from there.

For the linking problems #4825 ...

It is probably too late for this but in case we want to consider Azure DevOps as a cloud hosted CI options, do let me know. I work in the DevOps team.

We have GitHub Actions and Circle CI. I propose it is time to remove Travis (1 check remaining), and duplicated Circle CI checks. (note: same story for shards).
There are currently 36 checks running!

Yeah, Crystal CI's good now :)