Continuous query odd performance issue

Hi all,
we are evaluating InfluxDb for an internal project, and we came across some odd behavior when evaluating continuous query performance. Here is our setup:
Server specs:
CPU: i7 4770 4 core (8 thread)
RAM: 32 GB
Drives:
sda <- SSD Samsung EVO 850 250GB (OS + swap + influxdb data directory + influxdb meta directory)
sdb <- SSD Samsung EVO 850 250GB (influxdb WAL directory)
OS: Centos 7 x64
InfluxDB:
Version 0.12.2
linux Go 1.4.3 (arch amd64, maxproc 8)
logs disabled
wal directory on dedicated SSD (sdb)
data & meta directory on main disk (sda)
http "max-row-limit" set to 0
Schema:
at this moment we have only 2 type of data, a RAW input and its minute aggregation. The RAW data point (measurement "raw") ingested by our system has a nanosecond timestamp, one tag (called "topic") and several fields, of which only 2 are used for the aggregation process (let's call it val1 and val2, both numbers).
The second measurement, "minute", is produced with the CQ by downsampling the data with 1 minute window.
We decided to split this two measurements in two different databases, dbRAW and dbMIN.
The first has as default retention policy "raw_policy", with duration 7d and shard duration 30m, while the second db has as default RP "minute_policy" with duration 90d and shard duration 1w
That said, the query we use to produce the "minute" data is:
SELECT sum(val2) as tot_v2, min(val1) as min_val1, max(val1) as max_val1, first(val1) as in_val1, last(val1) as out_val1, mean(val1) as avg_val1, count(val1) as event_num INTO dbmin.minute_policy.minute FROM dbraw.raw_policy.raw GROUP BY time(1m), topic fill(none)
Ingestion rate:
In our tests we tried to simulate the "production" rate. In our case we have a population of "topics" that produces the raw points. This population is divided in 4 categories, called bottom, low, mid and high. The distribution is the following:
Bottom: 98000 topics, producing in total ~27 raw points per second
Low: 8000 topics, producing in total ~347 raw points per second
Mid: 500 topics, producing in total ~417 raw points per second
High: 140 topics, producing in total ~1540 raw point per second
which in total gives an ingestion rate of roughly 2331 raw point/sec and 106640 different series (just for the "raw" measurement). (note: With this ingestion rate the machine "sits" comfortably with very little RAM and CPU taken by the influx process)
The "Issue":
Now, we wanted to evaluate how fast the CQ are for the aggregation process, so we created the CQ (with the above select query) and monitored its processing time. With the above ingestion rate, the average CQ execution time was ~26sec, a time which is more than enough to keep up with the data production.That said though, we noticed something odd happening to the server during the CQ execution: during the first 8-9 seconds the CPU was used (1 core @~100%), after which influx starts to write very intensively for the remaining 17-18sec over the WAL directory. We noticed this by using the "iostat" command, which reported 90-95% utilization for the sdb partition (the one containing the WAL partition) with more or less 1000 write requests per second (!!). Take in account that in average the aggregated points are more or less 10000 every minute (~10000 different topics writes at least a raw point per minute), not a huge amount of data!
With this in mind, we wondered what happens if we ask from an external process the aggregated data and then write the result back in the dbmin database (I know, the idea seems odd but the results were more than surprising!).
So, we built a little Java program that every minute:
sends the following query to influxdb:
SELECT sum(val2) as tot_v2, min(val1) as min_val1, .... (same field as the first query) FROM dbraw.raw_policy.raw WHERE time >= start and time < end GROUP BY topic fill(none)
where "start" and "end" differs for exactly a minute, and end equals to the current minute
uploads the result data in the same influxdb istance, writing the data in dbmin.minute_policy.minute
By doing this, we went from ~26 seconds to ~9!! In particular, we have
~8-9sec for the data retrieval
~400-500ms for the write in influx
"normal" disk usage (no 90-100% spikes)
Are we doing something wrong with the CQ configuration? Why this totally different behavior?
Thanks for answering,
Davide Soldi

Fixed via #6732


Release 2.0

No changes from main except for removed pair alerts feature

Renamed the PR because we actually should release for GitHub, too


Bug doesn't play well with samples names starting with a number.

N E X T F L O W  ~  version 19.01.0
Launching `zuberlab/crispr-mageck-nf` [jovial_engelbart] - revision: 3b2575bbe5 [master]
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$mageck = <value>` with a process selector
[warm up] executor > slurm
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$postprocess = <value>` with a process selector
[74/c8674e] Submitted process > mageck (48hIFNG_d04_HLA_high_vs_NoIFNG_d0)
ERROR ~ Error executing process > 'mageck (48hIFNG_d04_HLA_high_vs_NoIFNG_d0)'

Caused by:
  Process `mageck (48hIFNG_d04_HLA_high_vs_NoIFNG_d0)` terminated with an error exit status (1)

Command executed:

  prefilter_counts.R            counts_mageck_ctrls-grouped.txt                 "pmax(48hIFNG_d04_HLA_high,NoIFNG_d0)"          50 > counts_filtered.txt

  VERSION=$(mageck -v 2>&1 >/dev/null)

  if [ $VERSION = "0.5.5" ]; then

   mageck test                  --output-prefix 48hIFNG_d04_HLA_high_vs_NoIFNG_d0                       --count-table counts_filtered.txt                       --control-id NoIFNG_d0                         --treatment-id 48hIFNG_d04_HLA_high                     --norm-method median                    --adjust-method fdr                     --gene-lfc-method mean                         --normcounts-to-file
  else
   mageck test                  --output-prefix 48hIFNG_d04_HLA_high_vs_NoIFNG_d0                       --count-table counts_filtered.txt                       --control-id NoIFNG_d0                         --treatment-id 48hIFNG_d04_HLA_high                     --norm-method median                    --adjust-method fdr                     --gene-lfc-method mean                         --normcounts-to-file
  fi

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.stub: line 45: ps: command not found
  .command.stub: line 45: ps: command not found
  .command.stub: line 45: ps: command not found

  Attaching package: ‘dplyr’

  The following objects are masked from ‘package:stats’:

      filter, lag

  The following objects are masked from ‘package:base’:

      intersect, setdiff, setequal, union

  Parsed with column specification:
  cols(
    .default = col_integer(),
    id = col_character(),
    group = col_character()
  )
  See spec(...) for full column specifications.
  Error in parse(text = x) : <text>:1:8: unexpected symbol
  1: pmax(48hIFNG_d04_HLA_high
             ^
  Calls: %>% ... eval_bare -> .Call -> parse_expr -> parse_exprs -> parse
  Execution halted

Work dir:
  /groups/zuber/zubarchive/USERS/Kimon/markus/HLA1_staggered/work/74/c8674eaf6ff6d7d550c57c867ad3d3

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
FAILED

If I edit the counts table and contrasts table to rename the offending sample as h48IFNG_d04_HLA_high instead, the pipeline works fine.

This is because MAGeCK can't deal with column names starting with numbers.

I happened to hit a very similar problem today using plain DESeq2.
It seems to originate in R's parsing of formulas from strings.
So definitely not pipeline-related or even mageck-related.


Introduce themable click actions + if condition for themes

Add themable click actions :
Applies on text & image components : <image onclick="back" [...]
Note : Please bump the theme, there are 2 working & useful samples in it.

Gamelist View onclick actions :

	back
	options
	gameoptions
	launch
	search
	savestates
	favorite
	random

System View onclick actions :

	netplay
	navigationbar
	search
	open
	random
	prev
	next

use "none" to disable default click action for an element.

Add if condition for theme elements, with a simple evaluator :
Applies to any element :  <pos if="{system.name} == 'collections'" [...]

extend themes variable list. Variables are now :
  global.architecture
  global.help					( bool )
  global.clock					( bool )
  global.cheevos				( bool )
  global.cheevos.username
  global.netplay				( bool )
  global.netplay.username
  system.cheevos				( bool )
  system.netplay				( bool )
  system.savestates				( bool )
  system.fullName
  system.group
  system.hardwareType
  system.manufacturer
  system.name
  system.releaseYear
  system.releaseYearOrNull
  system.sortedBy
  system.theme
  system.command

Thanks for your work, really. I've tried the new "if" method with variables and it works great, it's brilliant, congratulations! I think it will give us many new options for the development of layouts in the themes :). Just a suggestion, we can add to the list of variables some new ones like?:
global.cheevos.points
global.cheevos.rank
global.cheevos.friends
And to detect the screen resolution in ES, something like?.
global.aspect.ratio
Please
I'm very excited about the new environment that opens with this new "if" method. Thanks again!

Interesting for global.aspect.ratio. I'll think about it
Complex to write for global.cheevos.points/rank/friends, because it would require a connection before system & themes are loaded.

The aspect ratio sounds good. About the cheevos, do you mean the actual username variable is not pinging the retroachievements server? Is it "just" the string that the user fills in the field on the configuration screen?

Understood. Complex thing then... Thanks Fabrice!

Would it be possible to extend the system onclick action with retroachievements? this click would open the retroachievements window that is now accessed from the main menu

it already exists. Action is named cheevos

great! thanks. Please, is there a site or file where we can see the complete list of actions?


100% of Failed: 200 OK @ 21.0 MB / sec

Hi,
I installed and configured server and jquery components followed stepy-by-step your readme.
I used bootstrap template. But after upload I got:
100% of Failed: 200 OK @ 21.0 MB / sec
I don't have a clue, why.
Thx
P.

Can you post some more details, like the full error stack if possible?

Hi, i think i have the same issue : 100% of Failed: 0 error @ 8.99 MB / sec.
It's working fine on my localhost but not at all on my server (digitalOcean).
If i look at the console i can see i have these errors:
GET http://mysite/sockjs/info?cb=fgpzfrxeyv net::ERR_CONNECTION_RESET
POST http://mysite/upload net::ERR_CONNECTION_RESET
GET http://mysite/sockjs/info?cb=pe00pir_ns net::ERR_CONNECTION_REFUSED
So i guessed it's not working because of some security issues, but i have no clue how to solve it.
Do you have the same palpinter ?

I've had this issue as well. It is usually caused by the following problems:

The directories to not exist (make sure checkCreateDirectories is set to true in your options)
Your permissions are incorrect on your folder

Ensure that you have ownership (or your Meteor user does) of the uploads directory and that it has read/write permissions, e.g. 755 or if you're feeling like a dare-devil 777).

Closing for now. Please reopen if necessary,

I'm having the same problem. I don't get any error message or anything that can help me to debug this problem. I made sure to follow @onbjerg 's instructions

One guy fixed this problem and said this :
"So the permissions where the problem but my uploads-folder already did have chmod 755. What the problem was is that the folder was owned by the root user while the user trying to upload was the user made by meteor. So I didn't change the chmod but the owner of the folder with chown. The problem probably occurred because I deployed the uploads-folder to the server and didn't let the package create the folder."


dataset issue

Some lyrics include an interlude in the beginning and ending time. Such as:

Yep, It is actually a flaw in this work. Because the time span of each lyric is automatically extracted from each corresponding .lrc file .
In .lrc file, each lyric is annotated with the beginning time, we automatically use the beginning time of the next lyric as the ending time of the current lyric.

Thank you!


feat: integrate ExecutorProvider

This phases out EvmProcessor type and uses the new executorProvider abstraction
Due to a cyclic dep that needs to be fixed separately, the batch execution output currently mirrors BundleStateWithReceipts with From/Into impls.
this can be unified but requires followups
This extends the Provider Gats to the block inputs.
this makes the trait easier to use, and eventually unlocks something like:
trait ExecutorProvider<Primitives: NodePrimitives> {
      type Executor<DB: Database<Error = ProviderError>>: for<'a> Executor<
        DB,
        Input<'a> = BlockExecutionInput<'a, NodePrimitives::Block>,
        Output = BlockExecutionOutput<NodePrimitives::Receipt>,
        Error: Into<BlockExecutionError>,
    >;
}

this unlocks a ton of further cleanup
Followups

unified executor abstraction, with pre,tx,post sections
more reth-revm cleanup
unify BundleStateWithReceipts and BatchBlockExecutorOutput

few comments, would also want #7798 (comment)

this will be done in a follow, this is Receipts because this is used by BundleStatwWithReceipts


The maven component query return data is missing the publishedAt field.

https://api.deps.dev/v3/query?versionKey.system=MAVEN&versionKey.name=com.alibaba:fastjson&versionKey.version=2.0.47
There is no publishedAt field for the return result.

Hello and thanks for pointing this out.
Maven published timestamps are a bit tricky. Currently, for packages on Maven Central we only collect them from the index file which is only updated weekly. This means publishedAt might not show up for some time after the rest of the version data becomes available.
That said, this one is still missing and it's been around a month since it was published. I've had a look and found a bug in how we're handling these, which I hope to fix soon. I'll be sure to update here once the fix is in and the data is backfilled.


MIME4J-233 Fix ByteBufferInputStream read implementation

Why did you close this?

Because it appears to be in master already unless I'm very confused https://github.com/apache/james-mime4j/commit/4861043e44d34d8cfcf16990720031083792b178

Ok fair.
Somebody merged this work on master, and I did not get any notice about it.
Sorry for the confusion, everything is fine. Thanks for the contribution.
Cheers,
Benoit


Is there a promise version of geocodeByAddress?

Promise is widely used by now

@gutenye Great issue! Thank you for this suggestion. I'll look into this and adding support for that 👍


About VLM

Do we support multi-image input? Thanks

Yes but vLLM may not. cc @isaacbmiller

Complex image types (lists or arbitrary dicts or pedantic objects) aren't merged yet. You can just pass in multiple inputs as images to dspy.
Vllm has no issues afaik with multiple images.


Add clients to DNS log privacy by name not IP

I'm asked for an IP to add to the "Host Privacy IP List" but I'd like to be able to select a Wi-Fi client by name. Or just add a tag or something in the clients list, rather than use this interface separately?

@lts-po we have UI to do this, we should update that

Dev branch has a fix for this now.
https://github.com/spr-networks/super/commit/97c033bd9b7dd9f8191762f0698d9294e107c8bd


Build Always Pulls master Branch

I noticed that the build process appears to always pull master branch commits, and not develop branch even if that is specified in the project settings. Can someone confirm this is happening as well? (Want to make sure its not just me.)

Seems to be pulling the current branch for me on my gitlab and github projects

I think the issue might be the build button on the project overview page. If I click the build button on a new project that has not been run it will build master, yet display develop (as that was configured). This gets remedied once you select the branch you want to build from the build drop-down. After that it appears to build as expected. Could this be because the build button builds on whatever branch it is already on, until such time that a branch is switched (either via the drop-down menu or via a web hook).
Does that make sense? These are just observations so far, so I'm not sure if this is what the code reflects.

Ah, I tried with an existing project. I'll try a new one

Still seems to be working for me
I set up a new project and set the default branch to release, I clicked Build and it checked out the latest revision of release as shown by the commit hash shown in the log

Is release also your default branch in your repository?
I will try again on a new project as well and see if it happens again ... it may have been a perfect storm edge cases coming together to haunt me :)

Yeah release was the default branch I set in PHPCI

Actually yes I am having something similar
[http://cloud.rebelinblue.com/image/151N2J051J2A]

Ah, this seem to happen if I use the "Rebuild" button, if I build manually it is fine

at my git I have a master and a dev branch, and phpci build always the branch to who I commited last (commit is triggered by a webhock from gitlab), not the dev branch as specified in the config.

Uh, isn't that deliberate? Surely you want it to run the tests on the code you just changed? Not actually sure what the default branch is for...
Stephen Ball
<EMAIL_ADDRESS>
On 8 Aug 2016, at 13:08, Stefan<EMAIL_ADDRESS>wrote:
at my git I have a master and a dev branch, and phpci build always the branch to who I commited last (commit is triggered by a webhock from gitlab), not the dev branch as specified in the config.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

My interpretation of the Default Branch option was: Build only this branch because thats my default.


Does not work well with newer numpy versions?

Seems relatio is based on tensorflow which requires a numpy version downgrade -- however that unfortunately leads to all sorts of bugs/errors  w.r.t. arrays when installed on the base environment.
Of course you can do everything on a new environment, but wondering if there will be some version or way to make relatio not dependent on tensorflow? (sorry if the question sounds stupid, I'm  an undergrad relatively new to NLP and only started to pick it up a couple months ago due to my growing interest in political economy)

@psungho As you mentioned the way we highly recommend to use relatio is in a separate computational environment. The easiest way is to use either:

a Python virtual environment using venv or
a conda environment using miniconda or Anaconda

Did you experience issues using the recommended approach?

no not at all

Thanks for the update!
One should always use one or more virtual environments per project - this is the desired way (or using containers).
Unfortunately there is not much that we can do.
We will see in the future whether we keep the tensorflow dependency or not.
Can we close the ticket?


Swap out logrotate ansible role

This PR is to propose swapping out our custom ansible role for logrotate with one from Arillso that will be hopefully more standardized and that we don't need to maintain.

💯 👍


Elastalert Duplicate rules

HI Team,
Iam getting the below error message and container is not starting. Not sure where went wrong. All config looks good and no issues found with the rules also.
Please let me know if any facing the same kind of issue and solved it by any chance?
my configurations are:
/opt/config/elastalert_config.yaml/
/opt/rules/rules.yaml
Rules config looks like below:
rules.yaml: |-
name: ErrorLogs
es_host: "elalsticsearch-hostname"
es_port: 9200
es_username: "username"
es_password: "password"
type: any
index: logstash*
timestamp_field: "@timestamp"
num_events: 1
timeframe:
minutes: 5
realalert:
minutes: 0
filter:
- query:
query_string:
query: "log: error AND log: Journalerror"
alert:
- "email"
email:
-<EMAIL_ADDRESS>
Elastic Version: 7.4.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
Index elastalert_status already exists. Skipping index creation.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/elastalert/loaders.py", line 123, in load
raise EAException('Duplicate rule named %s' % (rule['name']))
elastalert.util.EAException: Duplicate rule named ErrorLogs
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/elastalert", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/elastalert/elastalert.py", line 2034, in main
client = ElastAlerter(args)
File "/usr/local/lib/python3.6/site-packages/elastalert/elastalert.py", line 138, in init
self.rules = self.rules_loader.load(self.conf, self.args)
File "/usr/local/lib/python3.6/site-packages/elastalert/loaders.py", line 125, in load
raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /opt/rules/..2020_04_13_09_16/rules.yaml: Duplicate rule named ErrorLogs

Hi @vinodhkumart , name field is required by rule and should in unique. The problem is, you have multiple rules with named ErrorLogs in rules folder.

Hi,@rehannali , I tried changing the name to different instead of ErrorLogs . still issue exists.

Show me your global config.yaml file.

apiVersion: v1
kind: ConfigMap
metadata:
name: elast-config
namespace: namespace
labels:
k8s-app: elastalert
data:
elastalert_config.yaml: |-
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /opt/rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: elastichostname

# The Elasticsearch port
es_port: 9200
es_username: elastic
es_password: password
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2
smtp_host: "smtp-serverurl"
smtp_port: port
smtp_ssl: false
from_addr<EMAIL_ADDRESS>smtp_auth_file: "/opt/elastalert/smtp_auth_file"
# If an alert fails for some reason, ElastAlert will retry

# sending the alert until this time period has elapsed

elastalert_supervisord.conf: |-
[unix_http_server]
file=/var/run/elastalert_supervisor.sock
[supervisord]
logfile=/opt/logs/elastalert_supervisord.log
logfile_maxbytes=1MB
logfile_backups=2
loglevel=debug
nodaemon=false
directory=%(here)s

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/elastalert_supervisor.sock

[program:elastalert]
# running globally
command = elastalert --config /opt/config/elastalert_config.yaml --rule /opt/rules/rules.yaml --verbose
# (alternative) using virtualenv
# command=/path/to/venv/bin/elastalert --config /path/to/elastalert_config.yaml --verbose
process_name=elastalert
autorestart=true
startsecs=15
stopsignal=INT
stopasgroup=true
killasgroup=true
stderr_logfile=/opt/logs/elastalert_stderr.log
stderr_logfile_maxbytes=5MB

@rehannali Above is the config yaml  configured in the config map.

Check in /opt/rules directory as well as it's sub directory, Make sure every rule name should be unique.

My pod goes into crashloopback and I am unable to check the folder

You need to stop supervisor and first make sure everything is working fine with manual testing.

@rehannali I had removed the supervisor configuration and deployed but still i can see the same error.

Issue has been fixed... Closing the issue.

@vinodhkumart , Good to hear. 😊

Hi so how to solved the issue?

@vinodhkumart how did you solve the issue?
We're getting the same error with rules mounted as a configmap

I assume this?
https://github.com/Yelp/elastalert/issues/2270#issuecomment-643079750


Question: TypeScript + external packages (JSPM/npm?)

Hi,
I just started playing around with this starter. In doing so, I tried to create a new component that uses Moment.js as an external dependency. I installed it over npm, linked it, also used tsd to add the typings (which got saved in tsd.json). However, Typescript complains it cannot find the module, the application runs without issues.

The application just runs fine. On the other side, if I change the moment import like this

...TypeScript stops complaining, but obviously also the app stops working.
I was also wondering how I could use jspm for installing my dependencies as that one usually wants to have a config.js somewhere. Is that currently a manual process? I found the tools/config.ts file, which I suppose should be the usual config.js one has with System and JSPM, right?
Thx for any insights into this. I'm new to TypeScript as well as JSPM+SystemJS, so... :smile:
(the source is here: https://github.com/juristr/angular2-seed/tree/simple-component)

@juristr You are probably missing moment definition file. try tsd install moment.

The seed does not use JSPM

@ludohenin No, I did: https://github.com/juristr/angular2-seed/blob/simple-component/tsd.json#L47
Used the command you mentioned and apparently it also placed the definition files here:

but still..

(and also in VSCode)
Not sure that's the right place.

@ludohenin Any idea why it doesn't leverage JSPM?

Ah I see, Some definition file do not support (the way they are written) the import syntax. My best guess is that this one has be written to use as a Global. using global moment in your code should not bother your IDE. if such is the case you need to either use the global function or make your own .d.ts file to support import syntax OR if you declare import ... 'moment' then you need to set the correct path so Systemjs can load the file.

@juristr You may also look at https://github.com/urish/angular2-moment , it looks like you may have when attempting to import moment_.
// under systemjs, moment is actually exported as the default export, so we account for that
const moment:moment.MomentStatic = (<any>moment_)['default'] || moment_;

@NathanWalker ya I did. In fact otherwise I couldn't even make it to run.
Currently my issue is more to get TypeScript recognize the module. The code itself runs properly.

I think that the problem is with moment.d.ts, not with jspm nor typescript... It seems that it should have a namespace called moment before the definition an it doesn't.
Looking at moment itself in tsd, it depends on moment-node... so you should install like this
tsd query moment --action install --resolve --save

but it won't resolve your problem, because the .d.ts exports the MomentStatic and not the module...

@Bigous Hmm...not sure.
Anyway, I got it to work with these adjustments.
I had already done an installation of moment's TypeScript definition files using tsd install moment --save.
Then in the main file, dateTime.ts, I adjusted it to this:
import { Component } from 'angular2/angular2';
import * as moment from 'moment';

@Component({
	selector: 'date-time',
	template: `
		<span>{{ dateTimeValue }}</span>
	`
})
export class DateTime {
	public dateTimeValue: any;

	constructor() {
		this.dateTimeValue = moment().fromNow();
	}
}

With these changes, TypeScript recognition and autocomplete started working. The app however broke. to fix that, I had to adjust how moment is registered with System.js in tools/config.ts:
...
export const SYSTEM_CONFIG = {
  defaultJSExtensions: true,
  paths: {
    '*': `${APP_BASE}node_modules/*`,
    'moment': `${APP_BASE}node_modules/moment/moment.js`
  }
};
...

Then everything worked. The src is here.
Note sure whether there's a simpler, nicer way to solve this or whether this is the intended way of importing external libs in this starter? @mgechev ?? Actually I'd like to make use of the placeholder config in config.ts:
...
'*': `${APP_BASE}node_modules/*`,
...

..but with that I couldn't get TypeScript to recognize the moment module.

Just for curiosity, what is the Typescript version that you are using?

@Bigous Should be the one referenced in the package.json: https://github.com/juristr/angular2-seed/blob/simple-component/package.json#L73

@juristr if you want to avoid to set SystemJS path and as node_module/moment/moment.js exist (you may have the case with other libs definitions) you can write your own adapter.
Like tools/typings/moment_adapter.d.ts
declare module 'moment/moment' {
  export = moment
}

then you can use it in the import statement
import * as moment from 'moment/moment';

:+1: thx for that hint

Hi @ludohenin I'm trying to do the same with Highcharts at tools/typings/highcharts_adapter.d.ts
/// <reference path="./tsd/highcharts/highcharts.d.ts" />

declare module 'highcharts/js/highcharts.src' {
	export = Highcharts;
}

and importing
import * as HC from 'highcharts/js/highcharts.src';

It gives me the error during watch compile at npm start
app\directives\ng2-highcharts\ng2-highcharts.ts(2,21): error TS2307: Cannot find module 'highcharts/js/highcharts.src'.

but it works... (just the compile error that is annoying me)
Do you have any hint?
[]s

same issue here with socket.io-client
i followed the steps from @juristr  and everything is working.
the only issue left is that in build.js.dev i get the following error.
[13:50:15] Starting 'build.js.dev'...
app/services/chatService.ts(1,21): error TS2307: Cannot find module 'socket.io-client'.
[13:50:17] TypeScript: 1 semantic error
[13:50:17] TypeScript: emit succeeded (with errors)
[13:50:17] Finished 'build.js.dev' after 1.85 s

do i need to tell gulp-typescript where to find the module?


Added Netlify instructions

corrected from previous PR
React Boilerplate
Thank you for contributing! Please take a moment to review our contributing guidelines
to make the process easy and effective for everyone involved.
Please open an issue before embarking on any significant pull request, especially those that
add a new library or change existing tests, otherwise you risk spending a lot of time working
on something that might not end up being merged into the project.
Before opening a pull request, please ensure:

[ ] You have followed our contributing guidelines
[ ] Double-check your branch is based on dev and targets dev
[ ] Pull request has tests (we are going for 100% coverage!)
[ ] Code is well-commented, linted and follows project conventions
[ ] Documentation is updated (if necessary)
[ ] Internal code generators and templates are updated (if necessary)
[ ] Description explains the issue/use-case resolved and auto-closes related issues

Be kind to code reviewers, please try to keep pull requests as small and focused as possible :)
IMPORTANT: By submitting a patch, you agree to allow the project
owners to license your work under the terms of the MIT License.

I'm following these instructions but struggling with service workers. Should something be added to the instructions for how to deal with that and Netlify? This is the best info I've found so far https://www.netlify.com/blog/2018/09/21/international-service-worker-caching-awareness-day/ but still have not resolved the issue

Coverage remained the same at 100.0% when pulling 2c43ce14aa2dd4a08342feb046a63131618ff1f7 on t3db0t:patch-2 into 54c233370dfc006a305c5c92337c706b1aae9afa on react-boilerplate:dev.


Set the default repo name on creation of a Project or Branch

Currently it is just nil which causes a confusing error when running build.

This is done.


New release please

Hi
Could you please do a new release asap ?
Best

Done https://github.com/google/cpu_features/releases/tag/v0.8.0


ExtendedSignedAttributes cannot be set by notation-go

ExtendedSignedAttributes is an arguments of notation-core-go SignRequest. It passes the extended attributes to be signed. However, we cannot set the argument by notation-go. The extended attributes filed is used by verification. Verify by plugin will be trigger if the signature extended attributes contain io.cncf.notary.verificationPlugin fields, so it is necessary to support setting ExtendedSignedAttributes in notation-go to help library user and plugin developer.

https://github.com/notaryproject/notaryproject/blob/main/specs/signature-specification.md#guidelines-for-notary-v2-implementors


Slack channel join message

We need to come up with a nice message for when davis joins slack channels and stuff. Cory came up with a good start but we can maybe extend it to have more useful info.

Ill be able to dive into this and make a good intro Jan 19th.

I'll close this for now.


Join Two Linked Lists

Aim :
Given two linked lists of size n1 and n2 respectively, you have to join the head of second list to the tail of first so that we can traverse both the lists using head of 1st list.
Please assign me this problem  under LGM SOC I will be performing in c++.
Programming language

[ ] C
[ ] C++
[ ] Java
[ ] Python

Can I do in Java? I'm a participant of LGM Soc21.

Can I do that in cpp? Please assign this issue to me.

Hi,
Can i try this issue in C.


Friendly overload should expose [In, Out] PWSTR in more friendly way

Today, accessing such a method requires:
private unsafe (string Path, int Index) PathParseIconLocation(string pathAndIndex)
{
    const int MAX_PATH = 260;
    Span<char> szIconFile = stackalloc char[MAX_PATH];
    pathAndIndex.AsSpan().CopyTo(szIconFile);
    fixed (char* pszIconFile = szIconFile)
    {
        int index = PInvoke.PathParseIconLocation(pszIconFile);
        return (new string(pszIconFile), index);
    }
}

With StringBuilder in its friendly overload it reduces to just this:
private unsafe (string Path, int Index) PathParseIconLocation(string pathAndIndex)
{
    const int MAX_PATH = 260;
    var szIconFile = new StringBuilder(pathAndIndex, MAX_PATH);
    int index = PInvoke.PathParseIconLocation(szIconFile);
    return (szIconFile.ToString(), index);
}

Based on feedback from @riverar.

Would you agree that Span<char> would be friendly and wouldn't demand an allocation?

Put all that gunk in the wrapper. I should be able to just pass in a StringBuilder and call it a day. Why does everyone have to do this extra  work? And why is everyone so focused on removing an alloc and not the ancillary effects like increasing expended cpu cycles or image size?

stackalloc assigned to span does not require unsafe. And it is very tempting to add Span<char> for mutable strings. The problem is that span implies a length, but where native code does not take a length, that implication is misleading. Suppose for example you provide a Span<char> with a length of 5, whose content is "hello". Sound good so far? Well, it's bad, because the native code that didn't take a length assumes a null-terminated string, and there is no justification in as assumption that two null  bytes (a UTF-16 null character) follows that span in memory, which would lead to malfunction in the native code.
So where the native code takes a pointer and a length we can and do take Span<T> or ReadOnlySpan<T>. Otherwise we take pointers (or a StringBuilder, after this issue) so that null termination requirements are more obvious.

And why is everyone so focused on removing an alloc and not the ancillary effects like increasing expended cpu cycles or image size?

None of these alternatives to StringBuilder involve additional CPU cycles. On the contrary: StringBuilder undoubtedly consumes the most CPU. Do not confuse less user code with less CPU time. The alternatives discussed here do only less than StringBuilder would do internally.
Image size would not increase appreciably either.
I suspect the reason @jnm2 is still suggesting alternatives to StringBuilder is that StringBuilder incurs multiple allocations whereas char[] would incur just one. However whether it's 1 or 3 allocations, it's all relatively minimal, so IMO we should produce the easiest API to get right, given we already offer the low-level method that takes pointers so if you really are counting allocations you can always do the stackalloc thing and pass in a pointer and have 0 allocations.

Does StringBuilder guarantee that the built string is null-terminated without exceeding the initialized StringBuilder capacity? If not, then I don't understand how it is less of a pitfall than Span<char> or char[].
I see StringBuilder as extra work. Usually it feels like I'm doing more work and writing more code than I really want. Unless I have at least two CopyTo calls, which is rare, StringBuilder always feels like it's less user friendly to me.

@jnm2  FYI: this is legal C# and would work if we offered a Span<char> overload. Note the lack of unsafe anywhere:
private (string Path, int Index) PathParseIconLocation(string pathAndIndex)
{
    const int MAX_PATH = 260;
    Span<char> szIconFile = stackalloc char[MAX_PATH];
    pathAndIndex.AsSpan().CopyTo(szIconFile);
    int index = PInvoke.PathParseIconLocation(szIconFile);
    return (new string(szIconFile), index);
}

@AArnott Yes, I picked up that you said that. That's nice.

It would be a plus if the friendly overload was also capable of avoiding allocations if desired, not just the pointer overload

I agree. In the dotnet/pinvoke project we auto-generated several friendly overloads in order to appease several use cases. However this became a real problem in that the overloads were sometimes so similar that arguments sometimes were not sufficient to guide the C# compiler to disambiguate which overload to call, which ironically made the method harder overall. So at the moment I am searching for a nice middle-ground between usability and efficiency that requires only 1 friendly overload. This is an area we're still learning from though, so we may add more friendly overloads or modify signatures on the existing ones in the future.

Does StringBuilder guarantee that the built string is null-terminated

You can read about StringBuilder interop here, where it documents it always allocates capacity + 1 in order to allow for native interop that will write null terminators.
I need to research however whether the string it represents is always contiguous when I get a pointer to it (some evidence suggests that is not the case) or whether the content when initialized by managed code is always null terminated.

To be clear, .NET supports marshaling StringBuilder natively, but as we have a goal to have just one extern method, and that takes pointers, our friendly overload is on the hook to reproduce any marshaling steps that the runtime normally would have done.

Just found this doc which says:

AVOID StringBuilder parameters. StringBuilder marshaling always creates a native buffer copy. As such, it can be extremely inefficient.

It elaborates further if you care to read about it. It pitches use of char[]. So I'll noodle on this a bit more. I wouldn't do char[] at least wherever Span<char> is an option (see #117) as the latter works everywhere char[] does as an input parameter and enables you to skip allocating an array on the heap for short arrays. And both arrays and spans share the same risk of implying a length that may not be communicated to the native side.

You can read about StringBuilder interop here, where it documents it always allocates capacity + 1 in order to allow for native interop that will write null terminators.

That page doesn't answer my question. What happens if the text you append exceeds the initial capacity? Then there might be no zero terminator within the buffer length that the API expects. Is that not a pitfall that puts StringBuilder on more even footing with Span<char>?

I know it doesn't answer your question. I already stated that I need to research...

whether the content when initialized by managed code is always null terminated

And anyway, as the StringBuilder does not guarantee its content is even contiguous, a copy will always be required, so it is the responsibility of the marshaler (or in this case, my friendly overload) to ensure that the copy target is null-terminated.

Ah, I didn't recognize your comment about contiguity as being in reference to my question. Sorry!

If we accept Span<char>, the marshaler (friendly overload) will have to ensure that '\0' shows up as the last character or somewhere else is the span. If it does not, a copy will have to be created to ensure the native code does not misbehave.

so it is the responsibility of the marshaler (or in this case, my friendly overload) to ensure that the copy target is null-terminated.

I wonder what the marshaler does. My preference would be to throw rather than truncate the string, since the consequences of proceeding with an unintentionally truncated string (especally with I/O) could be important.

My preference would be to throw rather than truncate the string

When/why would we ever truncate the string? I wasn't suggesting that. I was saying that if we accept a StringBuilder, and its length is 5, we would allocate a 6 character memory block, copy the 5 chars in, then ensure the 6th is the null char. No truncating is happening there.

To be clear, .NET supports marshaling StringBuilder natively, but as we have a goal to have just one extern method, and that takes pointers, our friendly overload is on the hook to reproduce any marshaling steps that the runtime normally would have done.

Can you change your goal instead? Why is that your goal? If you consider that off-topic then I can open a separate bug for general project documentation.

In an API like this, where there's a documented maximum length:
    const int MAX_PATH = 260;
    var szIconFile = new StringBuilder(pathAndIndex, MAX_PATH);
    int index = PInvoke.PathParseIconLocation(szIconFile);
    return (szIconFile.ToString(), index);

What happens if pathAndIndex is bigger than PathParseIconLocation expects it to be, and the null terminator is not within the first MAX_PATH + 1 chars?
Maybe the API detects this and errors. But then there is no concern that the use of Span<char> would cause the null terminator to be forgotten about, is there? The API would detect that and error.
I'm still trying to understand how StringBuilder succeeds where Span<char> fails, in terms of zero termination safety, from an earlier comment. I hope I'm not completely misunderstanding.

@riverar One extern method is our goal because every extern method comes with hidden runtime costs, including a slow first invocation. To support AOT environments, some of this runtime cost may eventually change to extra compile-time generated code, which translates to larger image sizes. The CLR team reviewed our design and were delighted to hear this was our goal for this reason.
It also allows us to fix marshaling that .NET would get wrong. For example the MSIHANDLE type in 32-bits instead of IntPtr-sized, so the way .NET marshals SafeHandle would be broken if we relied on it.

@aarnott I'm not convinced at all, but rather than be an impediment I'll kindly step back and watch from afar. Thanks for your patience.

It also allows us to fix marshaling that .NET would get wrong. For example the MSIHANDLE type in 32-bits instead of IntPtr-sized, so the way .NET marshals SafeHandle would be broken if we relied on it.

Not sure what this means. It's a typedef unsigned long MSIHANDLE; in the SDK. Why do you think this needs to be IntPtr sized? Traditionally, only the lower 32 bits of a HANDLE are significant (see HWND, GDI objects, etc.) regardless of platform architecture.

I think I see what I'm missing. It's not that the API wouldn't check, it's that there might be additional characters and then a zero terminator immediately following the span, so the API can't know that this is not what you meant to send.
The marshaling impl could solve this for Span by throwing if span.EndsWith('\0') is false.

@riverar Andrew doesn't think MSIHANDLE needs to be IntPtr sized. .NET thinks it does, if you use SafeHandle to wrap the handle.

What happens if pathAndIndex is bigger than PathParseIconLocation expects it to be, and the null terminator is not within the first MAX_PATH + 1 chars?

I don't know. That would be a per-API behavior and outside the scope of a general policy for generating overloads. Folks need to read the docs for these APIs and do the right thing.

Maybe the API detects this and errors. But then there is no concern that the use of Span would cause the null terminator to be forgotten about, is there? If you forget to null-terminate your span, the API would detect that and error.

We must not rely an that. We know nothing about the bytes after the last char in the span. It might include 50 garbage bytes and then two null bytes in a row, which may be enough for the API to report success since it found a null character although it isn't at all what was intended.
Or even more nefarious, we don't know when the memory beyond the span will change. So it's conceivable that a null may exist at one point in time during the call and then disappear later, just in time to make something very bad happen.
While we can certainly write it off as user error if they do not null-terminate, I think looking at the last char to see if it is null is cheap enough to be a reasonable mitigation to the risk we are introducing by use of span that the user assumes the span's length is passed to native code when it is not.

I'm still trying to understand how StringBuilder succeeds where Span<char> fails, in terms of zero termination safety

It doesn't. StringBuilder does not even guarantee contiguous memory, so even if it always terminates with null, we still have to copy to guarantee contiguous memory. IMO the only win StringBuilder offers is a slightly easier syntax than char[] or Span<char>.

Traditionally, only the lower 32 bits of a HANDLE are significant (see HWND, GDI objects, etc.) regardless of platform architecture.

Certain handles are documented to never use the high 32-bits, but this cannot be assumed to be true for all handles. Just to warn you.

it's that there might be additional characters and then a zero terminator immediately following the span, so the API can't know that this is not what you meant to send.

That's one possible end. Another is that it encounters an AV before it encounters a null character and crashes your app.

The marshaling impl could solve this for Span<char> passed to PWSTR by throwing if span.EndsWith('\0') is false.

Yes, that's what I'm proposing.

@riverar Andrew doesn't think MSIHANDLE needs to be IntPtr sized. .NET thinks it does, if you use SafeHandle to wrap the handle. Rather than missing out on the benefits of SafeHandle encapsulation, Andrew is passing the wrapped handle value as 32 bits since .NET won't.

But that doesn't cause any actual problems, right? The MSI APIs will truncate, it fits in a register on the appropriate platform, and long-standing compatibility guarantees sign off on this pattern.

I didn't know it was okay to define DllImport methods to use a 64-bit argument (which .NET's marshaler requires SafeHandle to be) to 32-bit API parameter. Then again, I haven't tried MarshalAs on a SafeHandle parameter either.

Just going to link to it from here as well since the discussion has moved: https://github.com/microsoft/CsWin32/issues/64#issuecomment-784400874
I gave an example of why string mutation is bad and where we blocked [Out] string in .NET Core.
https://docs.microsoft.com/en-us/dotnet/standard/native-interop/best-practices should be up to date with the modern best practices we recommend. Aaron Robinson (AaronRobinsonMSFT) and Jeremy Koritzinsky (jkoritzinsky) are good people to ask if there needs to be clarification on anything.

But that doesn't cause any actual problems, right? The MSI APIs will truncate,

Yes, it expect is would cause problems. When the .NET marshaler sees a SafeHandle parameter in an extern method, it acquires the IntPtr from the SafeHandle and pushes the IntPtr onto the stack. That is a 64-bit value in a 64-bit process. But if the native method signature takes a 32-bit value (such as MSIHANDLE), that means .NET assumed the stack should reserve 64-bits for the argument when the native method only expected 32-bits. If another value were pushed onto the stack after that, maybe the native method would start reading at a different position that .NET pushed it to.
Now, CPU architectures vary, and I don't know that memory alignment rules would end up always making this "just work" if all 32-bit values get 64-bits of memory on the stack anyway. But that seems like a detail I would not want to assume is true without a strong guarantee documented somewhere, and I am not aware of any.

Even if the alignments ended up correct, there is no guarantee that "padding" bytes are zero'd. They could be arbitrary memory and therefore be invalid if the underlying platform actually read 64-bits when a 32-bit value was pushed.
You should, in general, match the sizes/types exactly. So if the native takes a HANDLE which is strictly a void* (or something likeHWND__forHWNDin the "type safe" mode), then you really need to pass something that is going to besizeof(void)(such asnint, nuint, IntPtr, UIntPtr`, or a pointer itself).
.NET 6+ will have the new CLong and CULong types to match the C long and unsigned long type which are 32-bit on Windows and ptr-bits on Unix, but that isn't important for CsWin32 😉

In this case, given that the typedef is typedef unsigned long MSIHANDLE;, .NET needs to understand it is a 32-bit integer so it gets passed correctly (whether in the correct general-purpose register or on the stack).
The compat guarantees are for communicating between 32-bit and 64-bit processes, and is just calling out that you can do basically (uint)(handle) when going 64-bit to 32-bit, and then passing a uint or do (ulong)(int)(handle) passing a ulong (the conversion to int first being important so you can sign-extend).
It doesn't apply for doing 64-bit to 64-bit and 32-bit to 32-bit

In this case, given that the typedef is typedef unsigned long MSIHANDLE;, .NET needs to understand it is a 32-bit integer so it gets passed correctly (whether in the correct general-purpose register or on the stack).

In long mode (64-bit), none of that matters. The processor guarantees pushes and stack alignment are 8 bytes. So there's no (alignment or any other) issue here. The handle will always be passed via a single register, unless the param count exceeds 4 in which usual stack semantics apply. The receiving end (MSI API) will get their 8 bytes regardless and truncate if/as needed.
I can't see any issues with MSIHANDLE / SafeHandle here.

That is true on x64 today. It might not be true for say ARM64 or any future 64-bit platforms that are supported.
It is, in general, not good to try and depend on what is effectively implementation-defined behavior for correctness here.
If there is benefit that SafeHandle provides that is desired for MSIHANDLE then that's a good opportunity to expose something that helps handle that (either in box, out of box, or something that ships as both so NS2.0 and .NET 6+ can be supported).

For example, having some SafeHandle32 and SafeHandle64 variants may be beneficial. There are "handle" types in DX12/Vulkan that are always 64-bits because they refer to the GPU address space, which can be over 32-bits in length even on 32-bit platforms.

Also its worth noting that stack alignment and stack packing are two different things.
8-byte alignment just means that any frame start must be at least 8-byte aligned. However, if you have two sequential int32 values, they still take up only 8-bytes of stack space, not 16.

Which is not relevant in this MSIHANDLE case, correct?

Thanks all, think we've derailed this issue enough. Will move on.

but that isn't important for CsWin32 since its Windows only

@tannergooding That is almost true. What about the wine case? I wonder what we should do to run properly there. Also the calling convention used in C# 9 function pointers left me wondering how we would work there. It certainly isn't a top concern or supported. But it makes me wonder.

What about the wine case?

I'd imagine that's a case where WINE itself needs to be concerned with converting between Windows type and Unix type rather than the user code. As far as the Windows API and its implementation is concerned, long is 32-bits. If you happen to run on a platform where it isn't, that platform needs to adjust itself to account for that.

 Most of the old examples findable use StringBuilder as an easy way to get access to the string. I understand about it duplicating a buffer, but I sometimes care more about easily getting the data than performance at least when trying to find an initial solution to a problem. Can always use a better pattern after.
Would be nice to see some documentation and help to guide folks down the right path and be able to have different overloads which can use a simple StringBuilder or something more direct like span.
It is weird when the official docs have a nice sample here, but then say not to do that here.
Just want to point out when it's faster/easier to copy two lines of code to do the old PInvoke with a StringBuilder for GetWindowText than to use the source generator, there's a bit of a problem for discovery/usage.

Coming here a year later, has there been any progress on this in the interim? Like @michael-hawker alluded to, I'm using cswin32 for all pinvokes in my current project except for GetWindowText, as the manual pinvoke is just so much more readable

Not on this issue. I've been working on some other projects, but recently spent a bit more time on CsWin32. Not sure when I'll get to this, TBH. I hope soon.

The friendly overloads in the issue description are obviously hand-written and contain return types that would not be generally creatable by CsWin32 from their native prototypes.
Here is my thinking for the final solution:
// demonstrates calling pattern for the new friendly overload:
[Fact]
public void PathParseIconLocation_Friendly()
{
    Span<char> buffer = "hi there,3".ToCharArray();
    int result = PathParseIconLocation(ref buffer);
    Assert.Equal(3, result);
    Assert.Equal("hi there", buffer.ToString());
}

// proposed helper method:
private static unsafe int PathParseIconLocation(ref Span<char> buffer)
{
    fixed (char* pBuffer = buffer)
    {
        PWSTR wstr = pBuffer;
        int result = PInvoke.PathParseIconLocation(wstr);
        buffer = buffer.Slice(0, wstr.Length);
        return result;
    }
}

This issue was to cover the lack of a friendly way to call this API. I maintain that using a bunch of unsafe unidiomatic code with memory pinning is not friendly in any way, shape, or form.

There is no unsafe code or memory pinning in the [Fact] method, which represents user code in this example.

@AArnott Oh I misunderstood that. This is great then.

Here is my thinking for the final solution:

@AArnott this code looks incorrect. The native signatures is int PathParseIconLocationW([in, out] LPWSTR pszIconFile); and LPWSTR is meant to be null-terminated. ToCharArray does not include a terminating null and so the native method will end up reading past the end of the array.
Further, this function in particular requires that a buffer of MAX_PATH be passed in and so native code will likely end up writing outside the bounds of what's been allocated.

In general, there are many native APIs in this format and even the C headers do not provide enough context (even in the SAL annotations) to be able to automatically determine the "correct" handling in every scenario.
There are going to be many potential CVEs and other issues for APIs like this where the "safe" signature is detected via some heuristic.

Good catch, missed that. Am a little surprised CI didn't catch this, are those manual tests? (Not familiar with this project.)

Further, this function in particular requires that a buffer of MAX_PATH be passed in and so native code will likely end up writing outside the bounds of what's been allocated.

I see that in the docs, but it doesn't quite make sense. And looking at the impl., it doesn't appear this is true (anymore). Suspect this is a doc bug.

The MAX_PATH in this case may be a doc bug, its possible it was changed alongside many other MAX_PATH APIs when long path support was added.
The null-terminator being required is definitely still the case (there is no length passed in) as is the more general point of potential CVEs if these "helper signatures" aren't carefully reviewed and considered.

Thanks for reviewing, @tannergooding.

this code looks incorrect. ... ToCharArray does not include a terminating null and so the native method will end up reading past the end of the array.

Well, that's a bug in my example rather than a bug in the generated code. But I should still fix that. It'll be considerably more complicated to do it 'right' which undermines the simplicity of the use case, sadly.

In general, there are many native APIs in this format and even the C headers do not provide enough context (even in the SAL annotations) to be able to automatically determine the "correct" handling in every scenario.

That's why the helper method just takes a Span<char>, leaving it up to the caller to size and initialize the buffer as necessary.
I wonder though, to be more defensive around easy bugs like the one I made in the caller, should the generated code ensure that the span contains a null character somewhere? We could first check the last character to see if it's null, and only search the whole span if null isn't at the last position. The goal being to ensure that the PWSTR contract of a null terminator is honored somewhere in the buffer.

There are going to be many potential CVEs and other issues for APIs like this where the "safe" signature is detected via some heuristic.

I wonder if CsWin32 should inject an assembly-level attribute to effectively document the version of CsWin32 that was used in a particular compilation. That way if memory bugs are found in CsWin32-generated code, we can more easily identify possibly impacted assemblies.

@tannergooding So, possible doc bug aside, when an API says "a null-terminated string of length MAX_PATH", does that mean the buffer must be MAX_PATH + 1 so it can have MAX_PATH characters while still being null-terminated, or must the null terminator appear within MAX_PATH, effectively reducing the number of characters that can be there to MAX_PATH - 1?

I think there are lots of options and there has been discussion (on the Runtime team) on whether there could be some analyzer to help flag problematic cases around Span<char>, ReadOnlySpan<char>, and the byte equivalents for ASCII/UTF-8 text. CC. @stephentoub, @GrabYourPitchforks
In general, however, this is a large potential pit of failure, especially for users who aren't intimately familiar with interop code or the requirements.
In the case that a string literal is passed in, the Span/ROSpan won't include the terminating null-character and so having the marshaller check for this will have a lot of false positives for common scenarios. Likewise, you can't safely read one past the end of the Span/ROSpan since that may itself be an AV (access violation).

does that mean the buffer must be MAX_PATH + 1 so it can have MAX_PATH characters while still being null-terminated, or must the null terminator appear within MAX_PATH, effectively reducing the number of characters that can be there to MAX_PATH - 1?

The latter, as covered by https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd.
The premise is that MAX_PATH is 256 because you can have a drive + separator C:\, one or more path parts (a single path part can be up to 256 characters, even with "long path" support), and the null terminator.

In general I'd recommend either taking string in which case you can know that its null-terminated or always allocating and ensuring you include a null-terminator yourself (just as the DllImport P/Invoke Generator will do).
I'd also lean towards matching what the docs say and not looking at the implementation. Not only can the implementation change over time (and potentially differ on older or future versions of Windows) but not everyone can look at the implementation and so no one else can replicate or rationalize the "why".
If there is a case where the docs are out of sync, we should likely push to ensure those get fixed. Likewise we should push to ensure the SAL annotations are up to date and cover the relevant information where possible.

In the case that a string literal is passed in, the Span/ROSpan won't include the terminating null-character

A string literal couldn't be used where a Span<char> is required because string literals are immutable. This friendly overload we're talking about is specifically for [in, out] PWSTR, so string literals will never be accepted here.

In general I'd recommend either taking string in which case you can know that its null-terminated

We do take string for [in] PCWSTR parameters. And yes, we take the optimal path by knowing that .NET strings are always null-terminated.

I corrected my earlier example method to allocate a proper, null-terminated buffer. I'm thinking I'll go ahead and add a null-termination check to the friendly overload as well.

Hi! Pardon me but I think I'm missing something, I'd been under the impression the fix for this issue would include a friendly overload for GetWindowText, but all I'm seeing is this:
internal static extern int GetWindowText(winmdroot.Foundation.HWND hWnd, winmdroot.Foundation.PWSTR lpString, int nMaxCount);

Is that still coming in another issue or was it deemed impossible? It's hard for me to parse where y'all landed on that from the above discussion. Thanks!!

Ah I see, makes sense, thanks! I filed #614, though I'm not positive that either the current code or any of the proposals are correct, just trying to get the ball rolling. Thanks!


Parts of RA re-execute itself, making it unsuitable for use as a library

Specifically https://github.com/rust-analyzer/rust-analyzer/blob/ed4b312fa777ebb39ba1348fe3df574c441a485e/crates/rust-analyzer/src/cli/load_cargo.rs#L44-L63
I was surprised, because the comment says "this function is used by external tools that use rust-analyzer as a library" - but it then promptly re-executes RA.
I was able to work around via environment variables in my binary code, but I couldn't find this gotcha documented anywhere (in particular, in ARCHITECTURE.md).

I suspect the consumers using this function don't enable proc macros. IMO the rust-analyzer crate is not really intended as a public interface. But ping @matklad

Yeah: https://github.com/regexident/cargo-modules/commit/5c71c97a2d5be790392f96e560dc480f0b3383e8


Add ability to ignore imports that are empty files

In certain cases when creating new imports within a UI, a file might not have any content yet.  For instance, if you create a new Directory.Build.props within a project, Visual Studio will immediately re-evaluate the project and since the newly created file is empty, the evaluation will fail and the project will unload.
I propose a new ProjectLoadSetting like IgnoreEmptyImports that treats an empty file as an empty project.  It would also treat empty XML files as empty if they only contained the XML declaration like:
<? xml version="1.0" encoding="utf-8" ?>

Since its a new project load setting, it would be opt-in and probably only used by UIs.  I've taken a first stab at this here:
https://github.com/Microsoft/msbuild/compare/xplat...jeffkl:IgnoreEmptyImports?expand=1

It's not a breaking change (such a build would break today), why not make it the default behavior and not both with the setting?

@danmosemsft we've decided to have it off by default in MSBuild but are going to have it turned on by default when Visual Studio evaluates the project.  This way you'll still get build errors if you have empty files but the project won't fail to load in VS.


Bugs/minor improvements to the Resource Trackers

Disable the Maximum, Minimum, and Current fields if opening a tracker editor from the Tracker Manager.
Add all resources tracked by a character sheet to the "Apply To" dropdown.
Add custom damage types for resource trackers when clicking the Update button on the Tracker Manager so that folks don't have to restart Foundry.

Could be possible to add an option to delivery damage to diferents trackers? like HP and Armor?

@cefasheli -- Just pushed a commit such that any resource on the character sheet with both a name and alias, and "Use as Damage Type" checked, will allow you to put any damage type on it. Alias and names must be unique for the trackers on a given character (i.e., no duplicates).

Done: - Disable the Maximum, Minimum, and Current fields if opening a tracker editor from the Tracker Manager.


[Protected Var Request] - report 5855 "Implement Standard Cost Change"

protected var
    //A108F0062.Event20.00
    PostingDate: Date;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Explanation of what "safeonly" actually means is missing

This document does not really explain what "safeonly" actually means. What are "safety nullability warnings" compared to regular nullability warnings?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a690704d-da32-dca7-ea3e-66b84fb1d00d
Version Independent ID: 11f314de-dd3a-362e-b2bb-d0a2133cfc7b
Content: Nullable reference types
Content Source: docs/csharp/nullable-references.md
Product: dotnet-csharp
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

@BillWagner we should just remove any references to safeonly, as it's no longer a valid option.

Closing, as we're already tracking removing "safeonly" in another issue.

My mistake on closing this. It's not a duplicate.


Passing struct by value

Hi,
I see on the wiki that the structure function params can be only pointers for all tyeps (in, out, inout).
Is there a way to handle passing structure by value?

Hi currently only supported way is passing by address. In far future we were thinking to add support also by value.


check if can run test e2e-contracts without webServer config

reasoning is that now we run a crx build before the e2e test

it's needed to run the UI


SPyder 4.0.0b1 fails to install with latest Anaconda.

Issue Report Checklist

[ *] Searched the issues page for similar reports
[* ] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[ *] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[ *] Could not reproduce inside jupyter qtconsole (if console-related)
[* ] Tried basic troubleshooting (if a bug/error)

[ *] Restarted Spyder
[* ] Reset preferences with spyder --reset
[ *] Reinstalled the latest version of Anaconda
[* ] Tried the other applicable steps from the Troubleshooting Guide

[* ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
Spyder 4.0.0b1 fails to  install with newest Anaconda.
What steps reproduce the problem?

Download latest Anaconda
Follow install instructions for Spyder 4.0.0b1
Watch it fail with requirement  "spyder=4.0.0b1 -> python[version='>=3.6,<3.7.0a0']"

What is the expected output? What do you see instead?
Spyder 4.0.0b1 should install.
I think this issue is probably simple, but the latest version of Anaconda (5.3.1) fails when trying to install the beta (4.0.0b1) version of Spyder. It seems that this beta requires a Python version less than 3.7.0, and the new Anaconda has 3.7.1.
Downgradng Anaconda to 5.1 (Python 3.6) fixes the issue. I haven't tried other versions of Anaconda, but this seems a simple issue of version mismatch. If Spyder 4.0.0b1 runs fine with Python 3.7.1 maybe the dependencies could be changed? Otherwise you might want to add this issue to the beta install page.
Sorry if this has been addressed elsewhere. A quick google and forum search didn't turn anything up.

This happens because at the time we released beta1, Python 3.7 was not available so we were unable to create packages for it.
We'll fix that in a couple of hours by creating packages for it.

Awesome! I figured it was something simple like that.
Thanks very much for the SPyder IDE. Every time I try using something else it becomes clear how useful it is for scientific programming and analysis and I come straight back.

Thanks for your kind words!


⚠ Bofrost.de has degraded performance

In 09d25cc, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 990 ms

Resolved: Bofrost.de performance has improved in 0d918c8.


Int64 arithmetic with char

Int64 arithmetic broken if second operand is char (probably any other non-int64 type, should be tested)
Test case:
using System;

public static class Program {
    public static void Main ()
    {
        long l = 48;
        string s = "2";
        Console.WriteLine(s[0] + l);
    }
}

Translation:
    $T00().WriteLine(($T02().op_Addition(
          ("2"[0]).charCodeAt(0),
          ($T01().Create(48, 0, 0)).ToUInt64()
        )).ToInt64());

Expected: "98"
But was:  "0"
Problem: op_Addition expect both types are Int64.

This is the IL code I get using dotnetfiddle:
IL_0000:  nop
IL_0001:  ldc.i4.s   48
IL_0003:  conv.i8
IL_0004:  stloc.0
IL_0005:  ldstr      "2"
IL_000a:  stloc.1
IL_000b:  ldloc.1
IL_000c:  ldc.i4.0
IL_000d:  callvirt   instance char [mscorlib]System.String::get_Chars(int32)
IL_0012:  conv.u8
IL_0013:  ldloc.0
IL_0014:  add
IL_0015:  call       void [mscorlib]System.Console::WriteLine(int64)
IL_001a:  nop
IL_001b:  ret

It seams the conversion from char to uint64 (instruction conv.u8, IL_0012) has been ignored.


Provide a way to add custom headers to the default nginx.conf

I want to use the default nginx configuration, but add some custom CSP headers. The only way I can now do it is by adding my own nginx.conf as far as I know. However, I don't want to maintain my own nginx.conf file. I want to use the default one provided by this buildpack, and extend it with some headers.
Possible Solution
A possible way is to introduce BP_NGINX_HEADERS which you can pass in, or configure in a project.toml.

Related: https://github.com/paketo-buildpacks/nginx/issues/668 (similar feature request, but scoped to custom location directives)


Using OutputToReadStream to write into a HttpServerResponse

Hi Guss77,
First of all, thanks for your library. I need to write a large amount of data into a HttpServerResponse to return these data to an http client without overloading the server's memory. I tried to customize your code from OutputToReadStreamTest as follows:
HttpServerResponse response = requestContext.getResponse();
OutputToReadStream stream = new OutputToReadStream(vertx);
stream.pipeTo(response);
vertx.runOnContext(p -> {
  try {
    byte[] data = new byte[...];
    while (...) {
       readMyData(data);
       stream.write(data);
    }
    stream.close();
    p.complete();
  } catch (IOException e) {
   p.fail(e);
  }
  ...
  response.end();
});

The problem is that the "stream.write(data)" command is blocking on the first write. Is the OutputToReadStream class supposed to work with an HttpServerResponse as WriteStream? Dit I do something wrong in my implementation ?
Thanks
Eric

I think your implementation is correct and as far as I understand (some details are missing from your report) the implementation works as intended: please note that the way Java IO handles back-pressure is by blocking, which is what this library does - the write() waits until the client handles all the data before returning.
If you'd look at the code for OutputToReadStream.push() you'd note that it does not implement any buffering - each byte array is immediately offered to the data handler - using the Vert.x thread context - and waits for the data handler to finish.
So this may be the issue in your code - you call write() from the same Vert.x context that is supposed to run the HttpServerResponse data handler. The purpose of this library is to bridge Java IO semantics with Vert.x semantics, so it is assumed that one side (the Read/Write stream) is running on the Vert.x context, and the other (the Java IO reading/writing code) is running on standard Java blocking context - i.e. some kind of Executor thread or something. The Vert.x context thread must never be blocked, but Java IO is by definition blocking, so they are incompatible.
I suggest either changing the Java byte array read/write process to use a standard Java thread, or - if you want to run it on the Vert.x context - make it non-blocking, in which case you probably want to re-implement it as a ReadStream and not bother with this library :smile: .


Validate BoundingBox3D message

I have been working on testing the ground truth system of Isaac sim and have found some issues with their publishers of vision_msgs data. They are working on improving this tool, but while talking with their engineers they pointed me to a case where one dimension of the BoundingBox3D message can be set to zero. That is if a flat plane mesh is in the scene, the 3d bbox will have zero thickness. In this PR I have submitted a suggestion on how the Rviz plugin can support this.

LGTM other than adding brackets to your if/else statements

Sorry about the formatting errors, I've obviously been soiled with pre-commit on most of my repos.


Added selectattr value to select only nfs mounts

Overall Review of Changes:
Added selectattr value to select only nfs mounts
Issue Fixes:
https://github.com/ansible-lockdown/RHEL9-STIG/issues/40
How has this been tested?:
Tested for control RHEL-09-231065
Haven't tested for RHEL-09-231070 or RHEL-09-231075 yet

hi @layluke
Thank you for this PR and issues. feedback really does help to improve things. I have dropped you a chat on discord so we should be able to turn this around a little faster i hope.
many thanks once again
uk-bolly


fix: asset reference operation collisions

This is the PR that closes #34 and also introduces a breaking change that simplifies the Scene Manager implementation and unifies addressable and non-addressable workflows.
Todo List

[x] Implementation
[x] SceneManager Tests
[x] SceneLoader Tests
[x] Code documentation
[x] Update readme
[x] Write migration guide

:tada: This PR is included in version 3.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Cache only works, if you download the same thing from the same computer

Got the steamcache running, downloaded dota 2 on laptop, then downloaded started downloading it on desktop, but I still see speeds toward my normal (4-4.5MB/s by steam measuring), but if I deleted it on my laptop and redownloaded, I saw it install in few minutes (as it is supposed to), 2nd try on desktop worked aswell.
It seems like cache was for dota 2 machine spesific.
Repeated the experiment with unturned.

This sounds like the second machine is not pointing to the cache, Please can you confirm that the DNS settings on both machines are pointing to the steamcache-dns container, and can you try running ping valve20.cs.steampowered.com and send the output

Closing issue due to no reply


scanBarcode={true} results in the camera plugging to freeze on the emulator

Describe the bug
When i configure the camera like so:
        <Camera
          style={{height: '100%', width: '100%'}}
          cameraType={CameraType.Back}
          flashMode="auto"
          scanBarcode={false}
          onReadCode={(event: any) => {
            console.log('bar code scanned');
            console.log(event);
          }}
          showFrame={true}
          laserColor="red"
          frameColor="white"
        />

Everything works as expected and my image appears perfectly. But when i try to activate the bar code reading:
        <Camera
          style={{height: '100%', width: '100%'}}
          cameraType={CameraType.Back}
          flashMode="auto"
          scanBarcode={true}
          onReadCode={(event: any) => {
            console.log('bar code scanned');
            console.log(event);
          }}
          showFrame={true}
          laserColor="red"
          frameColor="white"
        />

The camera image stop showing and goes black and the animation of the bar code frame becomes very stuttery. The emulator becomes very slow as well.
I receive this error in the android console:
2023-07-14 12:04:39.527 174-15931/? E/ResolverController: No valid NAT64 prefix (101, <unspecified>/0)
2023-07-14 12:04:39.527 189-231/? D/gralloc_ranchu: gralloc_alloc: Creating ashmem region of size 4096
2023-07-14 12:04:39.528 13248-13321/com.snackinlogistica I/OpenGLRenderer: Davey! duration=895ms; Flags=0, IntendedVsync=6648436629550, Vsync=6648553296212, OldestInputEvent=9223372036854775807, NewestInputEvent=0, HandleInputStart=6648564397086, AnimationStart=6648564405586, PerformTraversalsStart=6648564446211, DrawStart=6648564544961, SyncQueued=6648564586669, SyncStart=6648621982211, IssueDrawCommandsStart=6648622260544, SwapBuffers=6648623245877, FrameCompleted=6649389386544, DequeueBufferDuration=0, QueueBufferDuration=125000,
2023-07-14 12:04:39.529 13248-13321/com.snackinlogistica W/OpenGLRenderer: reserveNext failed, error = -2147483648 (Unknown error -2147483648)
2023-07-14 12:04:39.529 13248-13321/com.snackinlogistica D/EGL_emulation: eglMakeCurrent: 0x7d51a25a80: ver 3 0 (tinfo 0x7d51b19a20)
2023-07-14 12:04:39.531 13248-13248/com.snackinlogistica I/Choreographer: Skipped 41 frames!  The application may be doing too much work on its main thread.
2023-07-14 12:04:39.535 182-15921/? I/EmulatedCamera3_QemuSensor: captureYU12: Dimensions for the current request (640x480) differ from the previous request (1280x720). Restarting camera
2023-07-14 12:04:39.901 182-15921/? I/EmulatedCamera3_QemuSensor: captureRGBA: Dimensions for the current request (1280x720) differ from the previous request (640x480). Restarting camera
2023-07-14 12:04:40.314 182-15921/? I/EmulatedCamera3_QemuSensor: captureYU12: Dimensions for the current request (640x480) differ from the previous request (1280x720). Restarting camera
2023-07-14 12:04:40.406 203-203/? W/BufferQueueLayer: [SurfaceView - com.snackinlogistica/com.snackinlogistica.MainActivity#0] Timestamp<PHONE_NUMBER>210 seems implausible relative to expectedPresent<PHONE_NUMBER>267
2023-07-14 12:04:40.408 377-395/? W/Looper: Slow dispatch took 508ms android.fg h=android.os.Handler c=com.android.server.-$$Lambda$LocationManagerService$LocationProvider$R123rmQLJrCf8yBSKrQD6XPhpZs@79e42 m=0
2023-07-14 12:04:40.408 377-395/? W/Looper: Slow delivery took 507ms android.fg h=android.location.LocationManager$ListenerTransport$2 c=null m=1
2023-07-14 12:04:40.686 182-15921/? I/EmulatedCamera3_QemuSensor: captureRGBA: Dimensions for the current request (1280x720) differ from the previous request (640x480). Restarting camera
2023-07-14 12:04:41.068 377-395/? W/Looper: Drained
2023-07-14 12:04:41.071 1045-1045/? D/BoundBrokerSvc: onUnbind: Intent { act=com.google.android.gms.safetynet.service.START pkg=com.google.android.gms }
2023-07-14 12:04:41.075 13248-15932/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getInt(Ljava/lang/Object;J)I (greylist, linking, allowed)
2023-07-14 12:04:41.075 13248-15932/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getObject(Ljava/lang/Object;J)Ljava/lang/Object; (greylist, linking, allowed)
2023-07-14 12:04:41.081 13248-15928/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getInt(Ljava/lang/Object;J)I (greylist, linking, allowed)
2023-07-14 12:04:41.081 13248-15928/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getObject(Ljava/lang/Object;J)Ljava/lang/Object; (greylist, linking, allowed)
2023-07-14 12:04:41.084 13248-15929/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getInt(Ljava/lang/Object;J)I (greylist, linking, allowed)
2023-07-14 12:04:41.110 182-15921/? I/EmulatedCamera3_QemuSensor: captureYU12: Dimensions for the current request (640x480) differ from the previous request (1280x720). Restarting camera
2023-07-14 12:04:41.477 182-15921/? I/EmulatedCamera3_QemuSensor: captureRGBA: Dimensions for the current request (1280x720) differ from the previous request (640x480). Restarting camera
2023-07-14 12:04:41.477 203-203/? W/BufferQueueLayer: [SurfaceView - com.snackinlogistica/com.snackinlogistica.MainActivity#0] Timestamp<PHONE_NUMBER>210 seems implausible relative to expectedPresent<PHONE_NUMBER>872
2023-07-14 12:04:41.861 13248-13248/com.snackinlogistica I/Choreographer: Skipped 35 frames!  The application may be doing too much work on its main thread.
2023-07-14 12:04:41.901 13248-15932/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getInt(Ljava/lang/Object;J)I (greylist, linking, allowed)
2023-07-14 12:04:41.901 13248-15932/com.snackinlogistica W/nackinlogistic: Accessing hidden method Lsun/misc/Unsafe;->getObject(Ljava/lang/Object;J)Ljava/lang/Object; (greylist, linking, allowed)
2023-07-14 12:04:41.907 174-15934/? E/ResolverController: No valid NAT64 prefix (101, <unspecified>/0)

But this only occurs on the emulator. On real hardware everything goes smooth.
To Reproduce
Steps to reproduce the behavior:

Just install the app as described in the documentation;
Configure the Camera view as described in this issue

Expected behavior
The camera to keep working with the barcode reading feature activated. And being capable to read bar codes.
Screenshots
Not a visual glitch, the camera view just goes black, and the animation of the bar code reader frame starts stutter. And the emulator becomes very slow;
Smartphone (please complete the following information):

Device: Emulator | nexus 5;
OS: androidVersionApi: 29

What camera are you using in the emulator? I have no issue with "VirtualScene" and "Emulated", while "Webcam0" is definitely unusable - even with barcode feature disabled on mine.

You are using an old emulator / nexus 5 is outdated. Please upgrade your emulator image and possibly even Android Studio. At the time of writing "Android Studio Giraffe | 2022.3.1" is the latest.
I also had no issue with your provided code. Scanner activates as expected.


Code review

Let me know when you're done with this package. Then I'll do a little code review and publish it on packagist.

@freekmurze I tested my code. It works. @matriphe please test your code if it works.

I've registered the package on packagist and started the external services.
I'm leaving for a holiday soon and cannot guarantee I can do a review before that. @mpociot can you take care of the code review?


Should fixed-length string (type 0x82) be 0-terminated?

fixed-length strings will not be 0-terminated


Video should display black screen

Hey guys,
We found one issue when we are uploading video on s3 bucket and generate .m3u8 video url at that time when we load video url on video player it is showing black screen in video player. so is there any specific version to not showing video and why its showing a black screen.
@armadilio3

@zchwyng
@milansusnjar
@jonthysell
@wildseansy
@asklar
Guys please help i am stuck here.

did you try with mp4 to see if you set it up correctly or not?

+1, adding types does not appear to fix the issue

@zchwyng @ShaneMckenna23 @nickfujita
@wildseansy Solution not working. please provide me solution as much as possible. It's a blocker for me to release the app.

Facing the issue in 5.2.0, has anyone found a solution?

@zchwyng @ShaneMckenna23 @nickfujita
@wildseansy Are we have any solution?

I faced the same problem :(

I found a solution. But how do I commit my bugfixes?

@codal-mpawar try the current version 6.0.0-alpha1 (latest master) and pass hideShutterView to the video component. That might fix it!

I found a solution. But how do I commit my bugfixes?

what is the solution?


HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal

This PR fixes the issue mentioned here.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 37s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  2s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
32m 41s

trunk passed

+1 :green_heart:
compile
1m 20s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 16s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
checkstyle
1m  2s

trunk passed

+1 :green_heart:
mvnsite
1m 20s

trunk passed

+1 :green_heart:
javadoc
0m 54s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 29s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
3m 32s

trunk passed

+1 :green_heart:
shadedclient
18m 15s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 17s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 17s

the patch passed

+1 :green_heart:
compile
1m  6s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
javac
1m  6s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 53s

the patch passed

+1 :green_heart:
mvnsite
1m 12s

the patch passed

+1 :green_heart:
javadoc
0m 48s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 17s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
3m  7s

the patch passed

+1 :green_heart:
shadedclient
16m 14s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
273m 43s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

-1 :x:
asflicense
0m 42s
/results-asflicense.txt
The patch generated 27 ASF License warnings.

361m 33s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.namenode.TestSaveNamespace

hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes

hadoop.hdfs.server.namenode.TestParallelImageWrite

hadoop.hdfs.server.balancer.TestBalancer

hadoop.hdfs.server.namenode.TestXAttrConfigFlag

hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA

hadoop.hdfs.server.namenode.TestFileTruncate

hadoop.hdfs.server.namenode.TestAddStripedBlocks

hadoop.hdfs.server.namenode.TestFSImage

hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication

hadoop.hdfs.server.namenode.TestStripedINodeFile

hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks

hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots

hadoop.hdfs.server.namenode.TestAuditLogs

hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy

hadoop.hdfs.server.namenode.ha.TestHAAppend

hadoop.hdfs.server.namenode.TestFileContextAcl

hadoop.hdfs.server.namenode.TestFSImageWithAcl

hadoop.hdfs.server.namenode.TestReconstructStripedBlocks

hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2790

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 2741c9243cba 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2d1dee5e44a79775dee66a80e43a449b8969a58c

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/1/testReport/

Max. process+thread count
3353 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 52s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
35m  9s

trunk passed

+1 :green_heart:
compile
1m 19s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
checkstyle
1m  1s

trunk passed

+1 :green_heart:
mvnsite
1m 20s

trunk passed

+1 :green_heart:
javadoc
0m 52s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 20s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
3m 13s

trunk passed

+1 :green_heart:
shadedclient
18m 48s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 55s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 46s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 16s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
3m 21s

the patch passed

+1 :green_heart:
shadedclient
18m 37s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
340m  5s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 37s

The patch does not generate ASF License warnings.

432m 37s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestBlockScanner

hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks

hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks

hadoop.hdfs.server.namenode.ha.TestBootstrapStandby

hadoop.hdfs.TestPersistBlocks

hadoop.hdfs.TestDFSShell

hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots

hadoop.hdfs.server.namenode.ha.TestHASafeMode

hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList

hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor

hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2790

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux b5180c5da691 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2d1dee5e44a79775dee66a80e43a449b8969a58c

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/2/testReport/

Max. process+thread count
2075 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Thanx @Nargeshdb for the contribution and @virajjasani for the review!!!


update mantine to v5.2.0

This is not a small change and I did my best to verify the majority of views visually. It might be that I missed something.
I'd appreciate any testing and feedback.

Thanks for implementing this! Found a couple of bugs:
CustomSearch:

Popup doesn't close if it is unfocused or enter is hit
Persons are missing from results
Instead of being one panel, it's multiple single entries with rounded corners

Faces:

On first view of the faces page the tab labeled is selected, but inferred faces are shown

Footer:

Clicking on an icon, switches the site and opens up the popover. It should instead either open up a site or open up a popover instead

WorkerIndicator:

When hovering over the indicator, the popup spawns correctly, but Popup Pointer isn't at the correct place

AutoAlbum:

Delete Button is directly on the image of the auto album and not within the dropdown menu

TopMenu:

The avatar menu is no longer shown on hover. That's fine, but there should be an indication, that the avatar is clickable, when you hover over it

Addressed all issues except the CustomSearch. This one seems more challenging

Do you need help? What's the issue? 😅

Take your time and get well soon! 🤗

Most things look like they are fixed :+1:
CustomSearch:

I can only sometimes click on a search result and find something. It looks like a focusing issue
If I am able to click on a search result, it's issuing a ton of request to query /list

Thanks for testing. I'll look into outstanding issues.


set preloadFileSize to 5000 in luarc

the default is 500 but this is not enough to load isaacscript-common

It is also generally recommended to not have code files that big, what is the reason here?

transpiling to multiple files would result in a worse UX, right?

I understand then. Still, that would likely make it load luarooms files too, which would decrease performance in mods that use StageAPI, and performance for the extension (the Lua extension in general) has already been a complaint for developers that use this mod I'd rather not make a worse situation of.
This setting can be changed anyways; the extension only sets it if the current setting is lower than the target setting, so if you increase it it'll stay increased.

I understand then. Still, that would likely make it load luarooms files too, which would decrease performance in mods that use StageAPI, and performance for the extension (the Lua extension in general) has already been a complaint for developers that use this mod I'd rather not make a worse situation of.

can we set sumneko to ignore anything in the "luarooms/**" glob?


TypeError: position.split is not a function

Seems like it could be related to https://github.com/aldeed/node-simple-schema/issues/33
I'm on 0.2.2, but was getting this same error on an older version. Just a simple insert. It's odd because I have a near identical insert that works fine.
TypeError: position.split is not a function
I20170224-14:37:07.570(-7)?     at MongoObject.setValueForPosition (/Users/scotttolinski/Sites/projects/atya/node_modules/mongo-object/dist/mongo-object.js:226:30)

This might be an issue with error reporting more than anything. I found I was attempting to insert a string into a Number field. I'm guessing that should throw an error of wrong type instead of this .split is not a function business.

@stolinski We'd need to see more code to reproduce this and fix. But yes, probably a small bug somewhere in the error handling.


Fixed compatibility with Symfony3

Can you also squash your commits please?

done

Thank you very much @kejwmen


validation loss dont decrease under 0.45

Hi dear,
thank you for your code.
but my both train and validation loss don't decrease under 0.45 with 400 epochs. May you help me for more config such as learning rate and so on?
regards

What dataset you are training on? if total-text or icdar2013/15, the configuration in README can reproduce the result in table. If not in two datasets above, please check dataloader you implement is correct first.


Check copyright

Copyright should be the same as OpenLiberty (apache 2.0)

The issue description is the licensing - the copyright to use is as follows:
/*******************************************************************************
 * Copyright (c) 2018, 2022 IBM Corporation and others.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors:
 *     IBM Corporation - Initial implementation
 *******************************************************************************/