Form generator and package roadmap

Hi @joanpablo
I have an idea to write generator which will help to overcome some disadvantages of JSON access to field values
described in this issue https://github.com/joanpablo/reactive_forms/issues/28
Something like
@ReactiveForm
User() {

}

Also I would like to discuss with you if there is a possibility to create some roadmap about form engine features to implement)
If you are ok with that ping me in discord vasilich6107#1453

Hi @vasilich6107,
Thanks a lot for your help, I've been quite busy recently.  Your idea sounds interesting. I would like to understand it better. So I will ping you as soon as I can. I cannot promise you it will be today, but I will let you know as asap.
Thanks a lot. Meanwhile, if you like to argue a little more here I can read a take a better understanding before we talk in discord.

Let me try to give a more detailed example.
Let's assume that we have Login model
@ReactiveForm
class Login {
  final String email;
  final String password;

  Login({
    @FormControl(
      validators: [], /*and other possigle params*/
    )
        required this.email,
    @FormControl(
      validators: [], /*and other possigle params*/
    )
        required this.password,
  });
}

This will generate class something like
class LoginForm {
  static const String emailControlName = 'email';
  static const String passwordControlName = 'password';

  late FormGroup form;

  LoginForm() {
    form = fb.group(_formElements);
  }

  Map<String, Object> _formElements = {
    LoginForm.emailControlName: FormControl<String>(
      value: null,
      validators: [Validators.email],
    ),
    LoginForm.passwordControlName: FormControl<String>(
      value: null,
      validators: [Validators.required],
    ),
  };

  String? get email => form.value[LoginForm.emailControlName] as String?;

  String? get password => form.value[LoginForm.passwordControlName] as String?;

  Login get data {
    if (form.valid) {
      // need to check if fields have validators Validators.required before using `!`
      return Login(email: email!, password: password!);
    }

    // trying to access non valid form
    throw Error();
  }
}

@tgbarker @marcos930807 join to our discussion of the generator

I suppose to start initial implementation in one or two weeks)
I think there will be a alpha version first with tons of breaking changes during implementation)
I think we could assume to achieve more stable beta release till the end of the year

I actually wrote a similar generator for internal use, so I can't share fully.
It started the same way, by defining the backing model and its fields. This proved to be more complicated than expected.
You can not always infer the correct FormControl type from the model value. A List<String> can be a FormControl<List<String>> or a FormArray<String>. Of course you can handle this with additional annotations.
But after the initial tests, I switched to defining the FormGroup with its controls and generating everything from there.
abstract class FormModel<M extends FormModel<M>> {
  Map<String, Object?> toMap();
}

abstract class FormGroupWithModel<M extends FormModel<M>> implements FormGroup {
  M getModel();
  void setModel(M model, {bool updateParent = false, bool emitEvent = false});

  /// Called after addAll(controls)
  void postProcessControls();
}

abstract class LoginFormModel implements $LoginFormModel {
  static const _$LoginFormModelBuilder builder = _$LoginFormModelBuilder();

  // Mapping business model to form model, can be done somewhere else.
  static LoginFormModel build({@required Login login}) {
    return LoginFormModel.builder.build(
      username: login.user,
      password: login.password,
    );
  }
}

abstract class LoginForm implements FormGroupWithModel<LoginFormModel> {
  FormControl<String> get username;
  FormControl<String> get password;

  factory LoginForm() = $LoginForm;

  @override
  void postProcessControls() {
    username.setValidatorsWithoutUpdate([
      Validators.required,
      Validators.minLength(8),
    ]);
    ...
  }
}

The controls are automatically added empty and can be updated afterwards with validators and so on in postProcessControls().
You construct a LoginForm(), you map your Login model to the LoginFormModel and call form.setModel(model) with it.
There are a couple more helper functions in play but this works pretty well.
There are no more string names for fields, you can just access the typed controls.
MagicFormConnector<LoginForm, LoginFormModel>(
  // magic that handles constructing form, model and its updates, depends on state management
  builder: (BuildContext context, LoginForm form) {
    return Column(
      children: [
        ReactiveTextField(
          formControl: form.username,
          ...
        ),
        ReactiveTextField(
          formControl: form.password,
          ...
        ),
      ],
    );
  }
)

Just some food for thought, hope it helps.

@kuhnroyal thanks for insights about FormControl<List<String>> or a FormArray<String>

It seems that code generation will require library split into packages.
The dependency on flutter sdk break the code generation.
Also for annotations validation functions should be static…

@joanpablo
I asked one guru and he confirmed that code generation requires no flutter deps
I’m working on code gen with mocks
when I will finish I will give you a link to repo.
Then I’ll prepare migration to separate packages

Hi @joanpablo
I'm writing generator.
It already produces from
            import 'package:reactive_forms_annotations/reactive_forms_annotations.dart';

            @ReactiveForm()
            class Login {
              @FormControl(
                value<EMAIL_ADDRESS>                validators: const [
                  NumberValidator.validate,
                ],
                asyncValidators: const [
                  uniqueEmail,
                ],
              )
              final String email;

              @FormControl(
                validators: const [
                  NumberValidator.validate,
                ],
                asyncValidators: const [
                  uniqueEmail,
                ],
              )
              final String password;

              @FormArray(
                validators: const [
                  NumberValidator.validate,
                ],
                asyncValidators: const [
                  uniqueEmail,
                ],
              )
              final List<String> categories;

              Login({
                this.email =<EMAIL_ADDRESS>                required this.password,
                required this.categories,
              });
            }

this
// GENERATED CODE - DO NOT MODIFY BY HAND

// **************************************************************************
// ReactiveFormsGenerator
// **************************************************************************

import 'package:reactive_forms/reactive_forms.dart';

class LoginForm {
  LoginForm(Login login) {
    form = fb.group(_formElements(login));
  }

  static String email = "email";

  static String password = "password";

  static String categories = "categories";

  late FormGroup form;

  String get emailValue => form.value[LoginForm.email] as String;

  String get passwordValue => form.value[LoginForm.password] as String;

  List<String> get categoriesValue =>
      form.value[LoginForm.categories] as List<String>;

  bool get containsEmail => form.contains(LoginForm.email);

  bool get containsPassword => form.contains(LoginForm.password);

  bool get containsCategories => form.contains(LoginForm.categories);

  Map<String, Object> _formElements(Login login) => {
        LoginForm.email: FormControl<String>(
            value: login.email,
            validators: [NumberValidator.validate],
            asyncValidators: [uniqueEmail]),
        LoginForm.password: FormControl<String>(
            value: login.password,
            validators: [NumberValidator.validate],
            asyncValidators: [uniqueEmail]),
        LoginForm.categories: FormArray<String>(login.categories,
            validators: [NumberValidator.validate],
            asyncValidators: [uniqueEmail])
      };
}

There is no big problem to generate other methods
I would like to discuss with you the splitting reactive_forms into separate packages.

Added support of bundled widget builder
class LoginFormBuilder extends StatefulWidget {
  LoginFormBuilder(
      {Key? key,
      required Login model,
      Widget? child,
      required Widget Function(
              BuildContext context, LoginForm formModel, Widget? child)
          builder});

  @override
  _LoginFormBuilderState createState() => _LoginFormBuilderState();
}

class _LoginFormBuilderState extends State<LoginFormBuilder> {
  late LoginForm _form;

  @override
  void initState() {
    _form = LoginForm(widget.model);
    super.initState();
  }

  @override
  Widget build(BuildContext context) {
    ReactiveFormBuilder(
      child: widget.child,
      form: () => _form.form,
      builder: (context, form, child) {
        return widget.builder(context, _form, widget.child);
      },
    );
  }
}

@joanpablo
here is initial repo https://github.com/artflutter/reactive_forms_generator

Hi @joanpablo
I create PR to illustrate how package split could be done.
The next step I will prepare real example with code generartion

hi @joanpablo @kuhnroyal
here is fully functioning example
https://github.com/artflutter/reactive_forms_generator/tree/master/packages/reactive_forms_generator/example

Please comment)
If you want to try it inside your project just refer to this setup
dependencies:
  flutter:
    sdk: flutter
  reactive_forms_annotations:
    path: ../../reactive_forms_annotations
  reactive_forms:
    git:
      url: https://github.com/artflutter/reactive_forms.git
      ref: package-split
      path: packages/reactive_forms

  # The following adds the Cupertino Icons font to your application.
  # Use with the CupertinoIcons class for iOS style icons.
  cupertino_icons: ^1.0.3

dependency_overrides:
  reactive_forms_core:
    git:
      url: https://github.com/artflutter/reactive_forms.git
      ref: package-split
      path: packages/reactive_forms_core

dev_dependencies:
  reactive_forms_generator:
    path: ../../reactive_forms_generator
  flutter_test:
    sdk: flutter
  build_runner: ^2.0.6

The example uses lates 10.5.0 version of reactive forms

Hi @vasilich6107,
Sorry for not answered before. I would like to make some tests in my local env and see the generator in action and see the advantages and disadvantages before including it as part of the official package. So I need to take a look at your great work, and examples.
Meanwhile, I have my first question: Should I be able to use Reactive Forms in a traditional way, as before the generator? Or it's going to be mandatory for the use of the generator and annotations to start using Reactive Forms? Is it going to be compatible with previous versions?

@joanpablo I don't think the generator is a requirement to use reactive_forms. And it shouldn't be.
@vasilich6107 I understand why you want to split the library but can you instead try using const TypeChecker.fromUri('package:reactive_forms_annotations/src/form_array_annotation.dart#FormArrayAnnotation') (not sure about the correct factory function). This does prevent the runtime dependency and doesn't pull in Flutter.
Same works for types in reactive_forms as well.

@joanpablo I answered to @kuhnroyal here https://github.com/joanpablo/reactive_forms/pull/189#issuecomment-885198972
The next comment will be an answer on @joanpablo questions

Hi @joanpablo
Thanks for getting along with my work.
Generator will be an assistive tool. You will be able to use ReactiveForms as before.
There will be no difference for the user.
The only difference is that on pub.dev will be two packages

reactive_forms_core - the part that do not require flutter as dependancy
reactive_forms - the same as before.

I described in this comment why I need this split https://github.com/joanpablo/reactive_forms/pull/189#issuecomment-885198972
Before writing this comment I even tried to move required typedefs and AbstractControl in separate files and import them directly via reactive_forms/src/path_to_file to avoid importing flutter as dependancy.
No luck.
Some words about the reasoning.
First of all json naming. Which I described here https://github.com/joanpablo/reactive_forms/issues/28
It is painful to write a separate class to have a single source of truth for names. Maybe this is not painful for you but I suppose it is better to have an ability to give boring work to code generator rather than writing it by myself.
As I mentioned earlier there still will be a possibility to use ReactiveForms as before, let's say in manual mode.
Second reason is this line of code
class FormGroup extends AbstractControl<Map<String, dynamic>>

it resolves into this
final document = DocumentInput(
      file: imageFile != null
          ? UploadFileInput(
              file: imageFile, contentType: imageFile.contentType.mimeType)
          : null,
      subTypeId:
          (form.value[CertificateFormFields.subType] as DocumentSubTypeMixin)
              ?.id,
      documentNumber:
          (form.value[CertificateFormFields.documentNumber] as String),
      countryIsoCode:
          (form.value[CertificateFormFields.country] as CountryMixin)
              ?.isoAlpha2,
      countryOfIssueIsoCode:
          (form.value[CertificateFormFields.country] as CountryMixin)
              ?.isoAlpha2,
      issueDate: (form.value[CertificateFormFields.issueDate] as DateTime)
          ?.toUtcWithOffset(),
      vesselId: (form.value[CertificateFormFields.vessel] as VesselMixin)?.id,
      score: score != null && score.isNotEmpty ? double.tryParse(score) : null,
      expiryDate: (form.value[CertificateFormFields.expiryDate] as DateTime)
          ?.toUtcWithOffset(),
      typeId: DocumentType.certificate.id,
    ).toJson();

It is very painful.
Cause I need to remember all the types behind the field names.
Cause there is no static analysis while coding.
Code generator aims to cover this two issues.
The naked truth is that form.value['login'] as String is ok for login screen or pet project but this is not ok for the big forms or enterprise projects.
What I'm trying to achieve with code generation is to have a model that describes form state with strong typing and to get this model back fulfilled with values.

There is also a question What if I fail to finish the generator?
One answer is "The package will still be usable without generator"
Another one "Package split could be reverted without any difference to the user. As far as user will be using reactive_forms as main dependency and all content of reactive_forms_core is reexported from reactive_forms

The roadmap is quite simple:

release splitted packages
add support for flat forms
release as beta for public use and bug fixes
nested forms support (aka nested FormGroups)

HI @vasilich6107,
Thanks again for all the work you are doing and all the help you give, not only to me, because I feel this package is not mine anymore, but of the community.
I understand now the issues you want to tackle with the generator. I'm just concerned about two things:
1-) I want the package Reactive Forms to be used as always, without the need for the generator (the generator as an optional independent package). As far as I understand from your previous comment, this is possible.
2-) I don't feel comfortable with the idea of splitting the actual Reactive Forms into two separated packages reactive forms and reactive forms core. I feel this will bring confusion to the actual and futures consumers of the library.
I read in the PR comments that @kuhnroyal has a solution that doesn't require splitting the package, is that right or I missed something?
I hope you can understand my concerns.
Best regards.

Hi. Thanks for your words!
You are totally correct. We do not need to split the package.

@kuhnroyal @joanpablo
beta version released https://pub.dev/packages/reactive_forms_generator

Added form groups support

Started the docs
https://github.com/artflutter/reactive_forms_generator

improved support of form arrays + readme sample
https://github.com/artflutter/reactive_forms_generator#dynamic-forms-with-formarray


RFLink: ValueError: invalid literal for int() with base 10:

Make sure you are running the latest version of Home Assistant before reporting an issue.
You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:
Home Assistant release (hass --version):
0.54.0
Python release (python3 --version):
python3.5
Component/platform:
RFlink
Description of problem:
Unclear. Error generated with RFLink Gateway Firmware release 47 when the following formatted data is received
20;06;OWL_CM113;ID=EACD;CURRENT1=0000;CURRENT2=007f;CURRENT3=0000;
This generates the following error in Home Assistant:
ValueError: invalid literal for int() with base 10: '007f'
Expected:
N/A
Problem-relevant configuration.yaml entries and steps to reproduce:
sensor:
  - platform: rflink
    devices:
      owl_cm113_eacd_current2:
        sensor_type: current_phase_1
        name: 'Unscaled House Consumption Current'
        unit_of_measurement: A

Traceback (if applicable):
2017-09-24 17:08:17 ERROR (MainThread) [rflink.protocol] failed to parse packet: None
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/rflink/protocol.py", line 99, in handle_raw_packet
    packet = decode_packet(raw_packet)
  File "/srv/homeassistant/lib/python3.5/site-packages/rflink/parser.py", line 255, in decode_packet
    value = VALUE_TRANSLATION.get(key)(value)
ValueError: invalid literal for int() with base 10: '002d'
2017-09-24 17:08:17 WARNING (MainThread) [rflink.protocol] no valid packet
2017-09-24 17:08:23 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 205, in async_update_ha_state
    "No entity id specified for entity {}".format(self.name))

Additional info:
Note: RFLink Rx and Tx commands are still working fine from Home Assistant despite the error.
Error repeats on every received data instance for OWL_CM113.

@aequitas FYI

Thanks for the highlight.
According to the Rflink documentation this value is not explicitly marked as hexadecimal: http://www.rflink.nl/blog2/protref Hence I'm parsing it as integer: https://github.com/aequitas/python-rflink/blob/master/rflink/parser.py#L167-L169
The current rating for this entity will probably be off when there is no error in parsing (hex number with only numbers no letters). And it will fail to read if it is a hex number with letters, throwing this error.
Unless there is official word from the Rflink developers about the exact type of these fields I'm hesitant to change it into Hex type. And currently my only source is the documentation.
Pulling this into HA as a configurable option might be a solution but this would break the abstraction between the two software packages and creates a bigger change and more complexity.


TypeError: Cannot use 'in' operator to search for 'signal' in /api/auth/stats/857656800173621308

Code:

const fetch = require("node-fetch");

module.exports = async client => {
  console.log(`Servers [${client.guilds.cache.size}] Users [${client.users.cache.size}]`);

  client.user.setActivity('Cachorrão do ddzin :)', {type: 'PLAYING'});
  console.log('online');

var requestOptions = {
  method: 'POST',
  headers: {
    "authorization": "my-token-censoured",
    "Content-Type": "json"
  },
  body: JSON.stringify({"server_count": client.guilds.cache.size})
};

fetch(`bot.sunlabs.gq`, `/api/auth/stats/857656800173621308`)
  .then(response => response.text())
  .then(console.log)
  .catch(console.error);

}

Console:

TypeError: Cannot use 'in' operator to search for 'signal' in /api/auth/stats/857656800173621308
    at new Request (/home/runner/api-bot/node_modules/node-fetch/lib/index.js:1219:16)
    at /home/runner/api-bot/node_modules/node-fetch/lib/index.js:1409:19
    at new Promise (<anonymous>)
    at fetch (/home/runner/api-bot/node_modules/node-fetch/lib/index.js:1407:9)
    at module.exports (/home/runner/api-bot/events/ready.js:20:1)
    at Client.emit (events.js:314:20)
    at WebSocketManager.triggerClientReady (/home/runner/api-bot/node_modules/discord.js/src/client/websocket/WebSocketManager.js:431:17)
    at WebSocketManager.checkShardsReady (/home/runner/api-bot/node_modules/discord.js/src/client/websocket/WebSocketManager.js:415:10)
    at WebSocketShard.<anonymous> (/home/runner/api-bot/node_modules/discord.js/src/client/websocket/WebSocketManager.js:197:14)

I guess you already found the culprit, but because I saw this among the first results in google for this error, I think an answer is good:
The issue is that instead of using the requestOptions object as the second param to the fetch call, you used a string.
The solution is to give an object as the second parameter. So something like this would work:

fetch(`bot.sunlabs.gq/api/auth/stats/857656800173621308`, requestOptions)
  .then(response => response.text())
  .then(console.log)
  .catch(console.error);


Notification on new issue

Hey @vladyslav-burylov
Within GitHub's repository settings, is there an option to enable notifications when a new issue is created? Because I neither got an email nor a notification for issue #9

Hey @mu88 - yep - it just want's e-mail. Can you send me something to<EMAIL_ADDRESS>so that I'll see your e-mail? I will add you.

Thx man! I just discovered a step to do this automatically in a GitHub Action and it seems to work well 👍🏻


multiple answers

Hello, there is a problem. set set up the bot. do it and request it to me at my request meets 1 10 times.
where is the problem?

I don't understand the question

i write command and get 10 some responses.  instead of 1

do a system reboot and try again.


benchmark error

Hi all， I run the benchmark_cubas.cu and find a really big problem.
when the N = 8192, the result is
success=1
success=1
that is, both xnor, gemm and cublas is correct
but when i change the N to other number(128，100,192,1192,2192,1000, 8195 and so on), the first is always wrong and the second is almost correct(only 100 is wrong), that is, the result is :
success = 0;
success =1
so, is this a problem??why only 8192 is possible??
anther auestion.
when i change the grid and block size to 1, the xnor result is also wrong(success = 0).
so, i think maybe the xnor code is not universal .It only works for some limited settings.  am i wrong?
looking forward to your reply

I think you have to keep N a power of 2 (e.g. 1024 and 2048 might work).
Our XNOR kernel indeed only works for some limited settings.
It was only meant to be a proof of concept.


WIP node server: don't persist vol

@sbezverk

@rootfs Create and Delete Volume gets from CSI volume name, but Node Publish/Unpublish operates with Volume IDs, if we do not persist volume information how you can build relation then? Also CSI spec expects in case of a plugin/kubelet failure to be able to recover previously created volumes, without persistent info, how it can be done? Even in k8s CSI core side, there is a persistent storage. Another thing about CSI RPC LIST_VOLUMES, with the persistent info it is fiarly easy to do.

controller still keeps the persistent json, but node doesn't need it. Node can figure out what device to unmap from mountpath.

@rootfs still how about the recovery scenario? For un-publish you get target_path and volume id, to get device for rbd unmap you will need to do an expensive search(on prod cluster there might be lots of mounted devices), instead of just a quick search in the persistent storage file.

@sbezverk searching device path is what kubernetes rbd/iscsi do in this case too.

@rootfs right, I know and I heard negative feedback of doing so, I do not see the optimization of going back to this way. Appreciate if you could provide the benefits.

@sbezverk we have to parse /proc/mount in either case - rbd images can only be safely detached if there is no more mountpoint.

/lgtm


License clarification

What's the license for code in this repo?

Under the MIT license, added a license file.


Defer loading of both nested children and subclasses

Nested children must be loaded immediately in development, because we
need to use require_dependency in order for reloading to work. But
that's not a concern in production, so there we can defer to
Module#autoload.
As for the STI loader -- instead of loading all the children as soon as
the parent is loaded, we can wait until the moment it matters: when
someone (most likely, ActiveRecord) calls #descendants.
This deferal is particularly valuable, because for most of the relevant
mid-level classes, we actually never do anything that would trigger the
load.

I like it... I think the giant comment should be kept in some form, but let's hash out any issues this might introduce first.

kicked off build again. if that doesn't pass, then rebasing should fix.
--K

@matthewd I'm good with this, please describe what we have to do in the build process so this data is pre-cached.  How do we walk the classes and pre-cache them all?


add secrets template

To aid in setting this project up as quickly as possible in the future, this PR adds a "template" for a secrets.yaml file that redacts sensitive values, but retains a few non-sensitive values and numerous comments. This should provide helpful context to future maintainers who want to use Brigade to build this project.

/brig run

/brig run


Create IgnoreAttribute

Addresses #14

Allows you to specify which properties of a class should not appear in URL
Add a test in TemplateTests to check
Rearrange things in TemplateTests for clarity

:+1:


Add support for videos

React-static-tweets doesn't current support videos. Here's an example tweet:
On Twitter: https://twitter.com/sassal0x/status/1364207094614749184
Rendered Version: https://react-static-tweets.vercel.app/1364207094614749184
Would love any pointers on how/where one might start to implement this! 🙂

Working on this already (as agreed with @iMuzz).
Here's an update on my progress:

The static-tweets ast parser is not finding the video media.
That's because it's not there in the HTML in the first place (likely Twitter API change).
The media URI is available through this URL:

syndication.twimg.com/tweet?id=1378452375296442371

Or, alternatively:
cdn.syndication.twimg.com/tweet?id=1378452375296442371

Here's the relevant part of the JSON returned:
{
  "video": {
    "aspectRatio": [
      249,
      220
    ],
    "contentType": "media_entity",
    "durationMs": 9934,
    "mediaAvailability": {
      "status": "available"
    },
    "poster": "https://pbs.twimg.com/ext_tw_video_thumb/1378452274570264576/pu/img/ytz1zFSu4mFOGGI_.jpg",
    "variants": [
      {
        "type": "video/mp4",
        "src": "https://video.twimg.com/ext_tw_video/1378452274570264576/pu/vid/498x440/s2-MGvS6ilwspHei.mp4?tag=12"
      },
      {
        "type": "application/x-mpegURL",
        "src": "https://video.twimg.com/ext_tw_video/1378452274570264576/pu/pl/bScvTVcujd7bUNjO.m3u8?tag=12"
      },
      {
        "type": "video/mp4",
        "src": "https://video.twimg.com/ext_tw_video/1378452274570264576/pu/vid/304x270/0xP7m72L6iuRzaht.mp4?tag=12"
      }
    ],
    "videoId": {
      "type": "tweet",
      "id": "1378452375296442371"
    },
    "viewCount": 473
  },
}

Currently debugging the static-tweets fetcher and parser to figure out how to fix it. Looks promising though. I'm only worried about CORS-type issues, but maybe there's still a workaround by using iframes or similar. Will continue this afternoon.
PD: @transitive-bullshit I might change the example a bit to debug this. Currently, it seems to be using an endpoint from your tweet-search project, so I might implement the API call directly into the example for easier debugging. With lerna linking the packages to the example now (thanks for implementing that btw!) it should be much easier to debug now.
Will keep updating on this issue, and create a draft PR once I start having actual changes to push.

That's because the video's not there in the HTML in the first place (likely Twitter API change).

One option would be to switch from using cheerio to puppeteer.. but I would really like to avoid this if at all possible since puppeteer is more heavyweight and introduces difficulties in serverless environments.

but maybe there's still a workaround by using iframes or similar

Let's avoid this if at all possible. What CORS issues are you referring to? I can't imagine there would be any issues playing the video from the client-side if we already have the video URL.
We circumvent all the CORS stuff in general by fetching the tweet info + metadata server-side where CORS doesn't exist 😄

I am gonna get the data from the new source. I've found the endpoint that provides that data now (I got it by reverse-engineering the official tweet embeds), so I'll just add it.

We circumvent all the CORS stuff in general by fetching the tweet info + metadata server-side where CORS doesn't exist

The video is precisely the one thing that isn't requested by the server, but it'll probably be fine.

One option would be to switch from using cheerio to puppeteer.. but I would really like to avoid this

Puppeteer is definitely too heavy, I agree. Cheerio is okay. Btw, I've been thinking about revamping the scraper with surgeon, I used to work on scraping and this tool is pretty amazing. But that's for a different time :P
Fortunately, this has a pretty straightforward solution that doesn't involve changing the HTML parsing :D

I got it, but doing this I realized that the code is indeed due for a bit of refactoring, especially since the changes in the Twitter API. We could release an experimental version or publish it under something like unstable_fetchTweetAst named export if we want to ship it right away.
I'll push the working code later and create a PR. @iMuzz you should be able to use it directly from GitHub temporarily if it is urgent.

@DaniGuardiola try to keep the refactoring and feature additions as separate as possible (ideally in separate PRs).

Any updates on this @DaniGuardiola

@kevcodez I got it working but as mentioned in my last comment, it'd be better imo to do a more global refactor of the AST fetching code, because my current solution duplicates the calls to Twitter's API and I think there might be a way to avoid it.
Maybe we could ship this under an option so that it's opt-in for users who don't mind the cost (extra requests could lead to problems with Twitter's API rate-limits). Sort of like an experimental feature flag. What do you think @transitive-bullshit?
@kevcodez if you need to use this right now, feel free to build the package from my branch yourself, but be aware of the trade-offs. Also, although it works just fine, it hasn't been tested thoroughly so it could break. Keep that in mind.

Gifs aren't rendered too, here is an example https://react-static-tweets.vercel.app/1461738430698860544

any updates?


Added app insights for tracking FATP metrics and data transformation error types

Changed the build steps around a little bit to facilitate cross-SDK builds


possible to unset an env variable

Is it possible to unset an env variable via a config, similar to unser VAR command ?

Not yet, but PR welcome 👍

Unset the environment variable = Assign null string
[pytest]
env =
    RUN_PATH=


Inject service for other model

I'm following the transaction example and I need to do something a bit more complex.
My models are:
WorkEffort <-> WorkEffortParty <-> Party

So, when I create a WorkEffort I want to associate it with two parties so I need the service for WorkEffortParty to be called from the WorkEffort service.
I already injected the repositories for Party so I can find the ones which will be linked.
But I can't figure out how to inject the WorkEffortParty service to create the links.

Ok, I managed to do it using the TransactionManager injected to my service, like below:
const wepClient = manager?.create('WorkEffortParty', {
      workEffort: workEffortRes,
      party: client,
      role: WorkEffortPartyRole.WeprClient,
      createdById: userId
    })
manager?.save(wepClient, {reload: true})

But I wonder if it's better to get an instance of the BaseService somehow

Have a look at the Feature Flag example.  FeatureFlagUser is similar to what you're trying to do: https://github.com/goldcaddy77/warthog/tree/master/examples/07-feature-flags/src/feature-flag-user

It looks more straightforward than what a I came up with the manager instance. I'll give a try, thanks!


rustup for error reform

This is the Miri side of https://github.com/rust-lang/rust/pull/69839.

:umbrella: The latest upstream changes (presumably #1227) made this pull request unmergeable. Please resolve the merge conflicts.

@bors r+

:pushpin: Commit 49051e05eaf91e97e106c56a4cbb0f59757494c6 has been approved by RalfJung

:hourglass: Testing commit 49051e05eaf91e97e106c56a4cbb0f59757494c6 with merge 0ff05c4cfe534321b194bf3bedf028df92ef519c...

:sunny: Test successful - checks-travis, status-appveyor
Approved by: RalfJung
Pushing 0ff05c4cfe534321b194bf3bedf028df92ef519c to master...


fix parameter description

Signed-off-by: Rishabh Singh<EMAIL_ADDRESS>Description
fix parameter description
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2996 (7e51c21) into main (b84c2df) will decrease coverage by 0.39%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2996      +/-   ##
==========================================
- Coverage   93.26%   92.87%   -0.40%
==========================================
  Files         167      167
  Lines        4602     4602
==========================================
- Hits         4292     4274      -18
- Misses        310      328      +18

Impacted Files
Coverage Δ

src/system/temporary_directory.py
64.44% <0.00%> (-28.89%)
:arrow_down:

src/system/process.py
93.54% <0.00%> (-6.46%)
:arrow_down:

src/system/os.py
94.73% <0.00%> (-5.27%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Question: A way to pass JWT not in a query params?

Hello there, creators. Awesome package, you've made, love it.
I was just wondering if there is any way to send JWT, not in the server query, but either in headers or by cookies? Also, was looking through the sources and somewhat related issues, though found nothing useful, unfortunately. Any leads? Maybe I'm missing something or not understanding subject correctly.
Thanks in advance for your work and attention.

Hi @lennoximus. I don't think this is possible. This package uses actioncable.js under the hood and from this StackOverflow answer it doesn't seem possible.

Hi! What about this strategy?
https://www.npmjs.com/package/actioncable-js-jwt
Looks promising! What do you think @mclintprojects @lennoximus?


Issue where plugin is not deleting old disk properly after resizing operation

Steps to reproduce:

Add vagrant-disksize plugin
Add section to Vagrantfile to resize disk

resize disk
if Vagrant.has_plugin?("vagrant-disksize")
config.disksize.size = '40GB'
end
3. run 'vagrant up' and wait for machine to come up
4. run 'vagrant destroy -f'
5. run 'vagrant up' again. At this point there should be an error message from Virtualbox similar to:
"Stderr: VBoxManage.EXE: error: UUID {9e51dfa3-b04f-4bf5-ac33-3358d8e30cff} of the medium 'Y:\VirtualboxVms\dev_box_x\box-disk001.vmdk' does not match the value {7d47d965-34d3-47ab-a738-e9cc4befb7c1} stored in the media registry ('C:\Users<usr>.VirtualBox\VirtualBox.xml')"
This is caused by not closing out the older disk in vboxmanage (closemedium) before deleting.
Created fork and PR to address issue.

@sprotheroe Any chance we can get this merged? I tested and changes work as expected. Without this, VirtualBox complains that the old .vmdk file is not accessible.

Many thanks for this. I accepted a PR with a very similar but slightly different approach that was more in line with the conventions used elsewhere in the code. Can you confirm this fixes the issues you were seeing?


WebSocket disconnection should not be an error

In any of the following cases:

user loses Wi-Fi range
user puts his laptop to sleep/hibernation
server restarts
server stops

The users get's an alert about WebSocket disconnection:

This should not be an error because it is possible that the server was restarted but the session will be kept between the restarts. The next Puppet operation should try to reconnect to WebSocket or use HTTP fallback.

@miyconst after doing current work, could you investigate this and propose changes?

Me and @warpech on skype call agreed to:

Disable the ugly error message.
Switch to http/json+patch if WebSocket connection drops.
Try to restore WebSocket connection in 5/10/20/40/80 seconds.
Display a small and not annoying icon when connection drops and hide the icon when connection restored.

I have played with restoring WS connection a little bit. Now facing such problem:

Client loads page.
Client establishes connection with server.
Client sends patch 1 to the server.
Server receives the patch and closes the connection (or connection drops).
Client establishes connection with server again.
Client sends patch 2 to the server.
Server responds with remote version 3, but client expects it to be 2, as patch 1 was sent, but not confirmed.

What to do in such situation? We may ask server whether the patch 1 was really received or may assume by version # that the patch was received.

Could you specify the versions that your are talking about in each step? - PuppetJS' local version == Server's remote vs. PuppetJS' remote version == Server's remote? (/_ClientVersion or /_ServerVersion$ )

1.1.  Out:
this.localVersion = 1;
this.remoteVersion = 1;

[{"op":"replace","path":"/_ver#c$","value":2},{"op":"test","path":"/_ver#s","value":1},{"op":"replace","path":"/workspaces/0/Images/CurrentPage/Images/Cancel$","value":1}]

1.2. In: none
this.localVersion = 2;
this.remoteVersion = 1;

2.1. Out:
this.localVersion = 2;
this.remoteVersion = 1;

[{"op":"replace","path":"/_ver#c$","value":3},{"op":"test","path":"/_ver#s","value":1},{"op":"replace","path":"/workspaces/0/Images/CurrentPage/Images/Save$","value":1}]

2.2. In:
this.localVersion = 3;
this.remoteVersion = 1;

[{"op":"replace","path":"/_ver#s","value":3},{"op":"test","path":"/_ver#c$","value":3},{"op":"replace","path":"/workspaces/0/Images/CurrentPage/Images/RedirectUrl$","value":"/images"}]

At this moment local remoteVersion is 1, but the one which come from the server is 3.

Ping @warpech and @tomalec.

So the connections gets closed after browser sent message, and after server received it, but before server responses to browser, and what is worse server did composed change to clients message, which was not send.
@chrhol chould you confirm, I would expect that server ma queue such pending change on server side, and either send it when reconnected, on will just send entire JSON to reset all versions

If it may help somehow:
void Handle(Input.Cancel Action) {
    WebSocket.Current.Disconnect();
}

This is how I close the connection on Cancel button press.

I rewrote Konstantin's problem including the version numbers, because I couldn't put my head around it:

Client loads page (_ver#c$: 0, _ver#s: 0)
Client establishes connection with server
Client sends patch 1 to the server  (_ver#c$: 1, _ver#s: 0)
Server receives the patch and closes the connection (or connection drops).
Client establishes connection with server again.
Client sends patch 2 to the server   (_ver#c$: 2, _ver#s: 0)
Server responds with (_ver#c$: 2, _ver#s: 2), but client still waits for _ver#s: 1. In result, client hangs indefinitely

A naive solution would be to time out and create a new request for the missing patch or just the current viewmodel.
But maybe there could be some pure-OT solution?

Only one correction:
7. Client does not hand, it throws an exception.

What is the exception message?

Pure OT solution would be that server responds with {op: "replace", path: "_ver#s", value: "1"} at 7.

Why? Because in step 6 the client said that it knows _ver#s: 0?

What is the exception message?

My bad, it actually hangs, the exception occurs if the received patch was already applied.

@warpech  because versions should be increased consecutively, and I se no {op: "replace", path: "_ver#s", value: "1"} in this flow.

I think that in Konstantin's case it is true that server closed the connection imperatively, which means that it SHOULD push the pending _ver#s: 1 patch before disconnection. If it doesn't do that then it is an error.
But what in case when the server increased _ver#s to 1, but that patch never reached the client, because it was eaten by the gateway, middleware, etc. How could we recover from that?

But what in case when the server increased _ver#s to 1, but that patch never reached the client, because it was eaten by the gateway, middleware, etc. How could we recover from that?

This is a good point. I think the client, when reconnecting, either need to ask the server for all patches from a version or just asks for the whole viewmodel. for the first option, we need to save all patches that the client have not confirmed that it have received.

the client, when reconnecting, either need to ask the server for all patches from a version or just asks for the whole viewmodel. for the first option, we need to save all patches that the client have not confirmed that it have received.

Totally agreed, as I mentioned, JSON Patch OT Agent  keeps a list of "pending" patches - patches that were send to the remote, but remote didn't confirm it get them.
So I would say that both peers should send such info to each other when re-connecting.

But what in case when the server increased _ver#s to 1, but that patch never reached the client, because it was eaten by the gateway, middleware, etc. How could we recover from that?

I don't think PuppetJs should solve the problem of someone filtering, and eating messages during communication.
Our working scenarios are:
A

client sends a patch that triggers a change on server,
server increases _ver#s to 1, and sends it to client,
server closes connection,
once connection is re-opened (server may return view-model with _ver#s==1),
if a change is made on server-side, server increases _ver#s to 2, and sends it to client.

B

client sends a patch that triggers a change on server,
server closes connection, and no change was made,
once connection is re-opened (server may return view-model with _ver#s==1),
if a change is made on server-side, server increases _ver#s to 1, and sends it to client.

I don't think PuppetJs should solve the problem of someone filtering, and eating messages during communication also.

So you say PuppetJs should not solve the problem but actually in C you propose a solution :) Or is it incomplete?
A problem that I see with point C7 is currently we treat is as an error: https://github.com/PuppetJs/JSON-Patch-Queue/blob/gh-pages/src/json-patch-queue.js#L61. We can change it to be a warning.

in C you propose a solution :) Or is it incomplete?
Yes i do. But this is not a solution for all missing messages, only for re-sending pending messages, so somehow extending existing feature.

It will not send an error for C7 as patches 1 and 4 are processed on server, but you are right if we have PuppetJS, and JSON-Patch-OT-agent.js working on server as well, it will be an error, as 1 was already applied, in such case we should make it as warning, and disregard the patch 1, and apply 4.

Anything new in this issue? What is the priority?

Discussion about session reconnection is moved to another issue. Thanks for fixing the issue in topic!


Regarding 3.0.2

I just noticed that there is a 3.0.2 version in Maven central : https://search.maven.org/artifact/io.github.microutils/kotlin-logging/3.0.2/jar
But we don't have any release notes for it
Could you please clarify what is the difference between 3.0.1 and 3.0.2? I could not find any tag or information about it

There wasn't significant changes between 3.0.1 and 3.0.2.


Secure persistence: Idea: Add fingerprint

Maybe it's possible to add a cryptographically secure fingerprint of the whole saved object. Then, during loading, we can compare whether the loaded object's fingerprint is identical. This could prevent certain types of attacks where the object that's being loaded is not identical to the one being advertised.

Are you talking about a hash of the stored object, or a fingerprint created by a private key of the person/entity creating the object?
We can add the capability to sign the stored object, if that's what you mean.

I'm not an expert in cryptography, so maybe I'm not always using the correct words.

a hash of the stored object

That's what I meant. I'm not exactly sure how to do that, if we just want a checksum and then the platform can provide some code to verify the checksum (which would be outside of the scope of skops).
Maybe we can also do something within skops, where skops checks that the checksum/fingerprint is as expected. My initial idea is that each attribute we store creates a hash of itself and then on the object containing these attributes creates a hash of these hashes, recursively up to the root (which would be the schema), like a Merkle tree. Then the user only needs to verify this top level hash against a secure source and skops can check the hashes of all nodes when loading.

We can add the capability to sign the stored object

Not sure how useful that is, maybe?

I don't think that's very useful, since whoever can tamper the artifact, can tamper with the checksum as well.
Signing would be nicer since we can trust certain people who have signed artifacts, the same way we can do for git commits.

I don't think that's very useful, since whoever can tamper the artifact, can tamper with the checksum as well.

That's true, but say, I download the model from a trusted website but get MITMed. Then I can still check that the final checksum is identical to the one on the website and decide not to open it if it differs.
Whether there is an additional benefit in skops checking internally, I'm not really sure.

Signing would be nicer since we can trust certain people who have signed artifacts, the same way we can do for git commits.

Could be, I would consider this to be a separate issue though.

That's true, but say, I download the model from a trusted website but get MITMed. Then I can still check that the final checksum is identical to the one on the website and decide not to open it if it differs.

If you get a MITM for the file, you can also get a MITM for the checksum.
Generally I'm not sure how much checksums add to the security, I see them more like "check if a file is corrupted" kinda thing.
Maybe @narsil or @mcpatate would have other ideas.

If you get a MITM for the file, you can also get a MITM for the checksum.

I think there can be situations where the model file was tampered with but I can still trust the website. But maybe that's too narrow, I don't know.

Usually if the https certificate is valid, it's pretty unlikely that you got MiTM-ed.
As Adrin says, imo signing is the best way to do this, but I don't see added value from signing commits - do you want to do something separate from the Hub ?

Since github uses SHA-1 to create ids for files, I you could always compute it when you download a file from a mirror and then compare that to the blob's id with this :
sha1("blob " + filesize + "\0" + data)

I don't see why people would need to go through mirrors when interacting with the hub, but I may be mistaken :)

I don't see why people would need to go through mirrors when interacting with the hub, but I may be mistaken :)

If skops persistence finds adoption, it could be used completely independently of the hub. Though of course we can wait and see before implementing a feature that's not needed.

@adrinjalali is on point.
Checksum is for verifying validity. It could be just flaky network random flips, or actual MiTM attacks.
In order to check against MiTM you need two different channels for the file and the checksum (so that the attacker needs to handle BOTH channels to change things under you).
If you really want to protect against MiTM attacks, then signing files is more efficient. That's because usually the certificate/public key distribution mecanism is by design on a different channel. There is a 3rd trusted party that handles the management of these certificates, usually hardcoded in your browser/package manager. For iOS apps, Apple is responsible for handling the signed binaries.
It's still a pretty big endeavor if you have ever done iOS/MacOS dev you would know that the signing/registering of apps is a bit of a pain.


Elasticsearch startup fails with 'failed to obtain data locks'

Pod startup fails with the following:
[2017-10-26T22:24:41,544][INFO ][o.e.n.Node               ] [elasticsearch-master-4227219301-s0dr9] initializing ...
[2017-10-26T22:24:41,591][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-master-4227219301-s0dr9] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/data/data/monitoring]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.3.2.jar:5.3.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/data/data/monitoring]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:260) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:262) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.2.jar:5.3.2]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /data/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:239) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:262) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.2.jar:5.3.2]
	... 6 more
Caused by: java.io.IOException: Mount point not found
	at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) ~[?:?]
	at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65) ~[?:?]
	at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) ~[?:?]
	at java.nio.file.Files.getFileStore(Files.java:1461) ~[?:1.8.0_121]
	at org.elasticsearch.env.ESFileStore.getMatchingFileStore(ESFileStore.java:107) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.env.Environment.getFileStore(Environment.java:351) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.env.NodeEnvironment$NodePath.<init>(NodeEnvironment.java:108) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:227) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:262) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.2.jar:5.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.2.jar:5.3.2]
	... 6 more

Same issue as https://github.com/elastic/elasticsearch-docker/issues/44
Workaround for those of us on the pires images:
Use https://github.com/goldmann/docker-squash to reduce the size of overlay in /proc/mounts then push your own image and use that.
😢

If quay.io supports multi-stage builds, I can try and work on reducing the layers needed.

Squashing worked alright (grep overlay /proc/mounts | wc -c went down from 1870 to 358) and elasticsearch started. I don't actually know if the latest images have the same issue. I think it would depend on when you last built docker-jre.

I remember there was some issue with Alpine Linux containers and layers. Anyway, I can't find a way right now to squash this without overwriting automatically tagged images. Using your own is fine by me.

:heart_eyes:  Thanks for providing the workaround, using docker-squash could solve the problem. Another approach is updating the base image from alpine to new version (Alpine >= 3.6.2) or other linux distro as suggested by @AtzeDeVries in this issue.
just a warm reminder, if u want to keep using the same image version tag when deploying to k8s, make sure u have imagePullPolicy: Always in the deployment .yaml file. Otherwise k8s will always keep using the old image. (i got stuck for two days! :persevere: )

I was having the same problem with the java.io.IOException: Mount Point not found using the quay.io/pires/docker-elasticsearch-kubernetes:2.4.0 image (as of June 1, 2018) on Azure Kubernetes Service with https://github.com/pires/kubernetes-elasticsearch-cluster (for both es-master.yml and es-data.yml). Surprisingly, the exact same Kube specs worked on the first attempt on GKE and AWS (with KOPS). Also, notably, if I move to quay.io/pires/docker-elasticsearch-kubernetes:5.6.0, the configuration worked on all three providers equally.
To workaround the issue, as explained by @artushin in the original post, I followed these steps:

Pulled quay.io/pires/docker-elasticsearch-kubernetes:2.4.0 on my Linux VM
Installed docker-squash as described in https://github.com/goldmann/docker-squash
Ran: docker-squash 852fb8f4a5e1, where 852fb8f4a5e1 is the Docker image ID of quay.io/pires/docker-elasticsearch-kubernetes:2.4.0
Tagged the newly produced image (9c4ca802e8b9) as myrepo/pires-docker-elasticsearch-kubernetes:2.4.0
Changed my es-master.yml and es-data.yml to refer to myrepo/pires-docker-elasticsearch-kubernetes:2.4.0 (everything else remains identical)
Deployed es-master.yml and es-data.yml using kubectl apply -f

That worked on AKS. I'm still testing the same with GKE and AWS/KOPS, but I'm hopeful that a single kubespec will work across the three engines.
Super thanks to @artushin

@dearjadu It's still work really fine at Azure AKS? What is VM Size are you using?

@alexsandro-xpt Yes, it does. I was able to use the same "sqashed" image in both GKE and AKS. I'm using Standard_DS11_v2 on AKS (and n1-highmem-2 on GKE).

FYI, that image (2.4.0) is super old. I highly recommend the new 6.x versions.


Proxy/config loader refactor

Regarding configuration

Config loading code is cleaner.
Only open allowed channels (do not allow disp if DisplayControl is not allowed in config).

Regarding filters

Make sure that filter_init is exported on filters compilation.
Instead of dropping the connection in filter_demo, just print some info to stdout.

@freerdp-bot can you test ?

side note: VS2010 shows some interesting warnings that may/should be fixed

@hardening changed the definition of PROXY_API and changed the inline function to be a macro, that should fix the msbuild errors
could you test again?

@freerdp-bot can you test ?

@hardening can we merge this one?


ByteData.getUint64() can't able to generate integers greater than (2^63)-1

I am writing code which takes bytes from a buffer and convert them to Uint64... but to my surprise I found out that the method isn't working as expected..
Here is a sample code to reproduce the error..
Code
import 'dart:typed_data';

void main() {
  print(ByteData.view(Uint8List.fromList([0,0,0,0,0,0,0,128]).buffer).getUint64(0, Endian.host));
}

Output
$ dart main.dart
-9223372036854775808

Expected Output:
$ dart main.dart
9223372036854775808

Dart Version: Dart VM version: 2.1.0-dev.9.3.flutter-9c07fb64c4
OS: Windows 10 1803 Build 17134.407

Dart 2 integers are 64-bit signed values (in the VM, they are doubles as ususal in the browser).
So, when you read a Uint64List element, you do get the correct bits, but the value as an int is interpreted as signed.
To represent a positive number larger than 2^63-1, you need to use a BigInt.
You can try something like:
  var uint64Mask = ((BigInt.one << 64) - BigInt.one);
  var big = BigInt.from(uint64List[index]) & uint64Mask;

@lrhn Or
BigInt.from(uint64List[index]).toUnsigned(64);

If that's the case then what's the point of this misleading information present in the API documentation https://api.dartlang.org/stable/2.1.0/dart-typed_data/ByteData/getUint64.html

The misleading information should be fixed. The returned value is the listed value converted to int.
Web compiled programs can't even use Uint64List at all, so we don't have to account for int actually being double on those platforms.

I am writing a flutter application which internally calculates the uint64 value from bytes as given in my first comment.. If the code couldn't generate the correct value then my entire application will be of no use because it entirely depends on the correct parsing of those bytes.. there should be some kind of workaround or a way native to dart should be available since most of other languages I've worked with has that support..

BigInt can represent unsigned numbers of any size.
As @rakudrama pointed out, BigInt.from(int64Value).toUnsigned(64) will get you the unsigned value corresponding to the 64 bits of an integer.
The Dart integer type (on the VM) can only represent signed 64-bit numbers, and Dart only has the one integer type, so you need to use BigInt for any integer which isn't a signed 64-bit.

Apologies for the remark on BigInt.. I have tested the example @rakudrama gave and it indeed worked.. But my main question is still shouldn't the getUint64() method on a ByteData return a uint64 value? Instead of changing the documentation you guys can change its implementation to either return a num or BigInt object.

@rashedmyt int is considerably more efficient than BigInt so we certainly don't want to return BigInt from getUint64.
What kind of math are you doing with the uint64 that you are getting out of the array?

I am implementing keccak crypto in which from a given array of bytes a uint64 number is generated which is xor'ed with the state of hash.. you can see its implementation here which I am trying to port to dart https://github.com/turtlecoin/cs-turtlecoin/blob/master/CantiLib/Blockchain/Crypto/Keccak/Keccak.cs

From a brief look I suspect that you don't actually need uint64 for this crypto algorithm.
If you are only doing +, -, <<, * 8  then int would bitwise behave exactly the same as uint.
The only thing you need to be careful about is shift to the right >>, which behaves differently.
But you can easily work-around those differences by masking upper bits away (or by using >>> which was supposed to be part of Dart 2 - but got by accident omitted and is going to be added in Dart 2.2)

This is the input to my library var res = Keccak.hash(Uint8List.fromList("".codeUnits));
I put a print before the loop to see the temp list
[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 1, 0, 0, 0, 0, 0, 0, 0]

which is exactly the same as produced in any other language.. inside the loop it computes uint64 value from 8 bytes starting from i*8 position in the byte array..
The code in my first comment is failing to generate the required uint64 value.. it should generate 2^63 value but it is generating -(2^63) probably due to out of bounds and wrapping

@rashedmyt yes, the code in your first comment does not print 2^63. However 2^63 and -(2^63) is exactly the same value if you look at it bitwise, which is what most hashing algorithms care about - they look at bits rather the integer value that those bits represent.
Imagine you take uint64 value 2^63 and int64 value -(2^63). Bitwise both of these values are the same thing - 0x80000000. So if your algorithm works with bits there is no difference to it. You just need to be careful about the right shift - all other operations would work the same way. For the right shift you need to do something like this
int logicalShiftRight(int val, int n) => (val >> n) & ~(-1 << (64 - n))

If you post the whole Dart source you have I can take a look and tell you why you are not getting the result you want.

import 'dart:typed_data';
import 'constants.dart';
import 'dart:math';

class Keccak {
  static Uint8List hash(Uint8List input) {
    return keccak(input);
  }

  static void keccakf(Uint64List state, [int rounds = keccakRounds]) {
    int t;

    Uint64List bc = Uint64List(5);

    for (int round = 0; round < rounds; round++) {
      /* Theta */
      for (int i = 0; i < 5; i++) {
        bc[i] = state[i] ^
            state[i + 5] ^
            state[i + 10] ^
            state[i + 15] ^
            state[i + 20];
      }

      for (int i = 0; i < 5; i++) {
        t = bc[(i + 4) % 5] ^ rotl64(bc[(i + 1) % 5], 1);

        for (int j = 0; j < 25; j += 5) {
          state[i + j] ^= t;
        }
      }

      /* Rho Pi */
      t = state[1];

      for (int i = 0; i < 24; i++) {
        int j = keccakfPiln[i];
        bc[0] = state[j];
        state[j] = rotl64(t, (keccakfRotc[i]).toUnsigned(64));
        t = bc[0];
      }

      /* Chi */
      for (int j = 0; j < 25; j += 5) {
        for (int i = 0; i < 5; i++) {
          bc[i] = state[i + j];
        }

        for (int i = 0; i < 5; i++) {
          state[i + j] ^= (~bc[(i + 1) % 5]) & bc[(i + 2) % 5];
        }
      }

      /* Iota */
      state[0] ^= keccakfRndc[round];
    }
  }

  /* Compute a hash of length outputSize from input */
  static Uint8List _keccak(Uint8List input, int outputSize) {
    Uint64List state = Uint64List(25);

    int rsiz = hashDataArea;

    if (outputSize != 200) {
      rsiz = 200 - 2 * outputSize;
    }

    int rsizw = (rsiz ~/ 8).toInt().toSigned(32);

    int inputLength = input.length;

    /* Offset of input array */
    int offset = 0;

    for (; inputLength >= rsiz; inputLength -= rsiz, offset += rsiz) {
      for (int i = 0; i < rsizw; i++) {
        /* Read 8 bytes as a ulong, need to multiply i by 8
                       because we're reading chunks of 8 at once */
        state[i] ^= ByteData.view(input.buffer)
            .getUint64(offset + (i * 8), Endian.little);

      }

      keccakf(state);
    }

    Uint8List tmp = Uint8List(144);

    /* Copy inputLength bytes from input to tmp at an offset of
               offset from input */
    for (int i = 0; i < inputLength; i++) {
      tmp[i] = input[i];
    }

    tmp[inputLength] = 1;
    inputLength++;

    /* Zero (rsiz - inputLength) bytes in tmp, at an offset of
               inputLength */
    for (int i = inputLength; i < rsiz; i++) {
      tmp[i] = 0;
    }

    tmp[rsiz - 1] |= 0x80;
    tmp[rsiz] = 1;

    for (int i = 0; i < rsizw; i++) {
      /* Read 8 bytes as a ulong - need to read at (i * 8) because
                   we're reading chunks of 8 at once, rather than overlapping
                   chunks of 8 */
      state[i] ^= ByteData.view(tmp.buffer,i*8, 8).getUint64(0, Endian.host);
    }

    keccakf(state, keccakRounds);

    Uint8List output = state.buffer.asUint8List(0, outputSize);

    return output;
  }

  static int rotl64(int x, int y) {
    x = x.toUnsigned(64);
    y = y.toUnsigned(64);
    return (x << y | x >> ((64 - y))).toUnsigned(64);
  }

  /* Hashes the given input with keccak, into an output hash of 200
           bytes. */
  static Uint8List keccak1600(Uint8List input) {
    return _keccak(input, 200);
  }

  /* Hashes the given input with keccak, into an output hash of 32 bytes.
           Copies outputLength bytes of the output and returns it. Output
           length cannot be larger than 32. */
  static Uint8List keccak(Uint8List input, [int outputLength = 32]) {
    if (outputLength > 32) {
      throw ArgumentError("Output length must be 32 bytes or less!");
    }

    Uint8List result = _keccak(input, 32);

    Uint8List output = Uint8List(outputLength);

    /* Don't overflow input array */
    for (int i = 0; i < min(outputLength, 32); i++) {
      output[i] = result[i];
    }

    return output;
  }
}

you are right.. the logicalShiftRight worked..

so at the end.. its just documentation misleading then..

@mraleph any work is being done on getting more crypto libraries added here https://github.com/dart-lang/crypto by the Dart Team or will it be just that

@rashedmyt I am not aware of any effort to expand the crypto library. I think people are encouraged to publish their own packages if they would like to implement new algorithms.

ok.. 👍

@mraleph since dart 2.2 is out.. is the >>> operator added to the language? Also I want to submit a PR to fix the documentation.. could you point me to it?


Stack.add_stats: Raises error if on_vars contains only numerical vars and checking_cluster is requested

Using other_source  for delimited_sets does also not work


Update config after install?

I read somewhere in your documentation that you do not edit the configuration of nextcloud after initial install and checked that by changing the config and it didn't get updated.
Can you make your code update the config?
I don't see a reasonable case where someone would want to update the config manually.

Hi
Thank you for your question.
This is a subject I have but some thought in it for quite some time.
The current code is quite old but It is in fact a limitation by design to keep the role indempotent. And do not break the app's configuration.
I would not have issue to use a template and use an yaml data structure if the config.php was only managed by ansible. But it's not the case: Nextcloud and some extra apps can add more entries in it, as the server is living it's life.
The risk of breaking something is big.
There is possibly the use off the occ CLI to manage all the configuration. this whould be a very good alternative to templates but would require a lot of tasks to keep the role idempotent - but I haven't worked enough with it to see what's possible. -
Probably making a custom module for the occ command would be the way to go.
It could allow to rewrite a lot of this role and go to V2 with new ones installation tasks and apps

Sry wrong button.
After all those thoughts, what's blocking me to make the V2 happen is my lack of free time those past months

The biggest issue that needs fixing is that when nextcloud updates it changes something in the config (redis had a tcp port, now it's a socket), or new nginx config, it's time consuming to fix.
occ would be a nice addition, I don't know how powerful it is. That won't work for nginx config though.

Do you mean that currently, the role breaks nginx setup and then Nextcloud config when rerun ?
On what distribution and version ?

It doesn't break them on rerun.
But if the config file has updates when nextcloud updates, the role doesn't take that into account.
For example at some point redis was updated from listening on a tcp port to a socket. The nextcloud documentation reflected that. But since this role doesn't touch the documentation, it didn't fix it.

Hard to standardise this kind of setup for various Nextcloud versions and distributions.
The role shouldn't run the configuration playbook if it detects Nextcloud as installed, unless modified, so it should not break any NC instance.
If the configuration has been updated by a upgrade, it should not be touched and you should not have corrective tasks to do after a run.
If you are in a multi instance installation scenario, you can adjust redis settings with variables. nextcloud_redis_*
By default the last version of the role use a socket to connect to redis. If the socket location has changed or is depending of the distribution please feel free to submit a fix with a PR. 🙂

Just reread the whole issue and I understand what's you meant.
I had misunderstood Nextcloud as the application while you meant the project, the configuration file as the Nextcloud app's config.php while you meant the web server's configuration.
The subject of the issue is that the role doesn't follow closely Nextcloud releases and doesn't update quickly enough configuration changes that are hard coded in templates and tasks.
Well ... This requires continuous integration and testing to actively detect and fix changes. This is kind of overpowered for a single ansible role...
So the role is provided freely as best effort.
Do not hesitate to correct me

I've been using your role since nextcloud 10 or 11, not sure. So my configuration was generated back then. And things change gradually, especially on major releases. And since the configuration is not touched by the role, sometimes something breaks.
What I was imaging as a solution is a heads-up message when something breaking might be released, like "double check redis configuration". And then I'll figure it out.
I guess a good approach is to check the release notes of nextcloud when a major version is released instead of insisting that the role check the configuration for issues that might arise.
This is in no way a critique of the role. I like this role a lot and it works pretty well. So you can close this issue.

Yes the server configuration is an inconvenient for nginx as it doesn't use the .htaccess files edited by the project, all changes have to be reported on the template.
I use nginx too on my setup.
Obviously the role whould requires myself to be more proactive about following Nextcloud RC schedules and update the role in advance...
I'll look on what can be done.
I'm curious to know how you use the role to upgrade your Nextcloud instance

I update via the occ. Can the role update? I haven't tried it.
Why don't you override the config.php file everytime? also the nginx config?

Can the role update? I haven't tried it.

No it purposely install only and skip the install if the application is already here so no upgrade. I made a upgrade role a while ago (nc 9 at the time) but the occ upgrade is much better now, so its kind of obsolete.

Why don't you override the config.php file everytime? Isn't that the ansible way to do things?

Yes IF ansible is the only manager for the file. it's not the case for the config.php: it can be modified by occ  and nc applications
most of the time ansible managed files have a header warning "Ansible managed. do not edit".
It can't be applied here.
I'm now pretty convinced that occ is the way to go to, fully and indempotently, install and configure nextcloud.
I started working on custom module(s) to modify nexcloud configuration throught the occ cli
but i'm a noob in python and full dev so it might take some time.
I aim for a V2 of the role much more robust with  upgrade and configuration as code (yml)

hello @mihailstoynov
after v 2.0 is release future focus will be to develop a system_config module to use occ cli to apply configuration instead of templating the config file. This should allow users to use ansible to apply/update their config and maybe improve the role to run all installation tasks in an true indempotent fashion.
Closing this issue as the proposition is accepted and is now planned to be developed.
I'll open a new task/issue to follow development for this future module after 2.0 is realeased


Adding curve bounds attribute

This PR adds an attribute that allows you to put specific bounds on an AnimationCurve.  This actually changes the curve interface to prevent you from entering a curve that falls outside of the bounds!  A simple variant of the attribute also exists to annotate unit bounds (0-1) for readability.
Also added a few default curve values so that you can have your curves take on a sensible default value other than having no keys at all.
To test:

[ ] Make sure that the attribute functions as expected
[ ] Make sure that the default curves are all formed correctly

This sounds veeerrry nice. Will take a look today.


image versions are not fail-safe for backup restores

At first, i am sorry that i lost the exact log of the problem. So i now try to recapture what i still know from the problem, but it will lack some specific information and snippets. I have no specific idea how to solve this, so i hope to see some nice proposals here^^
Describe the bug
The overall structure of some images (like the webapp) prevent the startup of old ones. The container installs at start-up additional packages from the bundled package manager. The problem is, that some configuration files are not system default, like php-configuration. So when the image is old and the package installation triggers dependencies or updates, which involves to update the not default configuration files, the problem occurs. The startup hangs and the container is not able to boot, because apt-get mentions that configuration files are modified and the new packages are not able to install. This happened to me, when the image had php5 bundled but the installation of the webapp plugins tried to update to php7, mentioning that the php-config was modified.
I know why you do this at start-up, since providing all plugin install combinations as tags for the images would result in a very high number of different images. I vote for an image with all (official) plugins preinstalled, to get a robust behavior for old images, when nothing is done with the package manager at startup.
To Reproduce
Steps to reproduce the behavior:

Try to use an old webapp container, containing php5 and install plugins, which try to update to php7.

Expected behavior
The container starts as expected with the old setup.
Additional context
This is a typicall use-case for us, since we backup our volumes and docker-compose stack configuration (including the .env files) together. In this way we are able to restore the service and its data in their depending specific versions. This assumes, static docker image tags, referencing stable images, which are able to start, even when they are old.

Hi,
since these containers have so far only used Debian 9 there should not have been a container with php5, but at max php7.0.
Normally the upgrade of old packages should be prevented by https://github.com/zokradonh/kopano-docker/blob/master/webapp/start.sh#L22.
For a production deployment I would though recommend to use the bundled makefile to produce your own images with the desired plugins directly backed into the container at startup.

hm.. thinking a bit further about this its probably better to remove the ability to dynamically install packages as startup altogether. Or only install packages that are part of the image already.
The reason to not install any webapp plugins by default is, since every installed package needs to be loaded at login, which slows down the login process. so in general its recommended to only have plugins installed that you plan on using.

Hi @cguentherTUChemnitz,
do you have any further input?

I actually saw this this morning on my testing setup:
Modified configuration file
---------------------------

php.ini: A new version (/usr/lib/php/7.3/php.ini-production) of configuration file /etc/php/7.3/apache2/php.ini is available, but the version installed currently has been locally
modified.

  1. install the package maintainer's version    3. show the differences between the versions            5. start a new shell to examine the situation
  2. keep the local version currently installed  4. show a side-by-side difference between the versions

Which resulted in the container hanging at startup.

I actually saw this this morning on my testing setup:
Modified configuration file
---------------------------

php.ini: A new version (/usr/lib/php/7.3/php.ini-production) of configuration file /etc/php/7.3/apache2/php.ini is available, but the version installed currently has been locally
modified.

  1. install the package maintainer's version    3. show the differences between the versions            5. start a new shell to examine the situation
  2. keep the local version currently installed  4. show a side-by-side difference between the versions

Which resulted in the container hanging at startup.

Wow yeah, that is exactly what i tried to describe from my memory. The non-interactive apt-get should do the trick for the exact problem. Nevertheless, the idea of building the own image with imprinted plugins seems also quite good for me. In my eyes, we can close this ticket.

Ok, lets close it here.
To summarise:

when wanting to include additional packages it is recommended to use the makefile to build own images with the packages already included (the dockerfile uses the same variable name for this).
in the future the current functionality to install packages at startup may/will be removed, if it causes more problems than the added convenience it provides. there is already https://github.com/zokradonh/kopano-docker/issues/265 for this.


Retz should automatically find leader in multi-master setup

At the moment, in a setup with multiple masters who elect a leader via Zookeeper, if the host set in retz.mesos is not currently the leader, then Retz will neither output an error message nor re-connect to the current leader. Retz will output

2017-03-28 11:42:18.771 INFO  [main] io.github.retz.scheduler.Launcher - Connecting to Mesos master mesos1:5050 as nobody

but never arrive at the point where it outputs

2017-03-28 11:52:09.527 INFO  [Thread-2] i.g.retz.scheduler.RetzScheduler - Connected to master mesos3; Framework ID: 6126772b-3a46-478c-b1ef-5b0585e73a16-0001

However, in particular in a high-availability setup, Retz should be able to find the current leader automatically, or at least output a message that the specified host is not the leader.

In my local setup with three Mesos masters, setting retz.mesos = zk://zk1:2181,zk2:2181,zk3:2181/mesos in /opt/retz-server/etc/retz.properties worked for me - Mesos client inside Retz automatically followed Mesos master failover ... with an exception in fetching files in the task sandbox. I'll fix it soon, but without get-file and run subcommand it should work. Please let me know if any other part is not working.

With 0.2.10 this should be fixed. Feel free to reopen if any further issues.


add api for tidevice

针对常用的设备查询和app查询做了简单的api

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.qxz1rba seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


full ajax response from the cache gives undefined

I'm trying to make a smooth background animation transition. So I would like to update the body class / id / style after onReady.
So in onReady callback:
var url = smoothState.href // <-- get the current url
var doc = smoothState.cache[url].doc // <-- full html response
console.log(doc);

Why gives the console an undefined error?

Hi!
I don't know what refer your variable "smoothState", but please try this :
var url = $('#main').data('smoothState').href; // <-- get the current url
var doc = $('#main').data('smoothState').cache[url].doc; // <-- full html response
console.log(doc);

Hi.
I am having same problem. Fetch returns and undefined object.
My code is below.
onReady: { duration: 0, render: function ($container, $newContent) { var smoothState = $('#content').smoothState().data('smoothState'); console.log(smoothState.fetch(smoothState.href)); }},

This appears to be a regression. $('#main').data('smoothState').cache[url] now only contains html, status and title properties. html is just the content to be inserted and not the full document.

@DanSmith70 thanks for the bower notice, I was also using it!


In Step 5 of Getting Started with Snowflake - Zero to Snowflake, I am at the "Load the Data" section, where I am trying to the copy into

In Step 5 of Getting Started with Snowflake - Zero to Snowflake, I am at the "Load the Data" section, where I am trying to run this code:
copy into trips from @citibike_trips
file_format=CSV;
I am receiving this error:
Found character ':' instead of field delimiter ','
File 'citibike-trips/json/2014-04-01/data_01a304b5-0601-4bbe-0045-e8030021523e_005_6_3.json.gz', line 1, character 41
Row 1, column "TRIPS"["STARTTIME":2]
If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.
The error is caused by the JSON Files in the Stage-path, wich was created.
Can this be corrected so I may continue? Thanks.

I'm also having this exact same issue.

Well spotted. The easiest fix is to use the same LOAD statement that we used a few lines above with the Small virtual warehouse, namely:
copy into trips from @citibike_trips file_format=csv PATTERN = '.*csv.*' ;

@tt-ics they've added the files in both csv and json format to the S3 bucket. If you add ON_ERROR='SKIP_FILE' to the line it will only read the csv files


[INT-423] Refactor Ethereum example connector

Where is pb.go?

you added them to .gitignore? anyway I pushed it but there is no gitignore in compound example ass well as makefile etc.

I didn't add them to .gitignore

I didn't add them to .gitignore

I thought you created these folders 😅 now I see it is older


What should we do with unspecced things?

From this comment:

Maybe there should be a "meta-issue" to discuss where the line is drawn for un-specced items, and what the threshold for discovery of undocumented items should be.

Indeed! We need some kind of a consensus.
Also, my thoughts from this comment:

I don't like NOTSPECCED issues floating there like this (especially that amount), so I have the following idea: we should tag stuff as NOTSPECCED, but if nothing can be done now, we can close the issue. Once something is implemented, we can reopen it.
In other words: if there is nothing we can do, then we consider the issue solved at this particular moment in time. This would clear the list of open issues, and the NOTSPECCED label will make it possible to check the list periodically to make sure that nothing falls through the cracks.

Personally, I'd :+1: the second comment approach. For example, we don't have an ability to describe many pragmas because, um, they don't have a description at all, they just reserved. It'd be wise to just mark them in the docs as reserved for a future use or something like this. We need to make our issue list shorter, not longer, to be sure on what things we can focus.
About other things. I believe that we shouldn't(in the perfect world, at least) use notspecced stuff. But some things are already leaked here(correct me if I'm wrong) and we can do nothing with this.
My personal opinion is to have three levels of things(categories if you wish):

Internal rakudo/whatever things -> we shouldn't document it.
Internal rakudo-related things, that are useful and actually can be used outside to write close to compiler stuff, like NQP, some classes, etc(I'm bad with examples of this category). They may be documented with some warnings. It can be debated for sure, but if such things have usage in big language modules, it'd better to have them than trying to cover things under the blanket. On the other hand, it puts more maintenance burden on maintainers and we don't have them too much, to be honest.
Official Perl 6 things -> we must document it.

@Altai-man is there any reason why we cannot mention these things anyway? For example:

Internal rakudo/whatever things → document it by writing “Do not use this!”
Internal rakudo-related things that are useful → document it by writing “Part of rakudo, but not part of the language spec. Use it on your own risk.”

What do you think?

Seen from the standpoint of a language user, the spectest are rather unwieldy. The only official documentation we have for the language and therefor the spectest are the docs. As soon as we add stuff that are rakudo internals, we take responsibility for any fallout that might have. As such any such addition needs to be signed of by a language designer or we may end up painting the language into a corner via proxy.
Also, after Perl 6 has conquered the worlds, there will be many many implementations.

Why reject? Let's document everything!
Imagine this: someone finds an undocumented feature and uses it in his code. Then somebody else finds this code and steals it for his own problem. That user may try to search the docs for this undocumented feature, but won't find anything. “Sure, perhaps the docs are incomplete”, right?
By knowingly ignoring unspecced features we are just making the situation worse. Look, we don't have to explain all of the details about unspecced things, we just have to mark explicitly that it is not recommended for use.
Or is there any other solution to this problem? I don't think that “you have to check every undocumented thing in roast” is something we should recommend. I can't imagine a 6-year-old doing that.

Please run util/list-missing-methods.p6 and look at the sheer number of leaky methods. If we add some, we need to add all. And then remove them when they become private again.

@AlexDaniel the main reason - stable api nonsense in a compiler sense.
Yeah, you can write it with a caps lock on and some red font like "DO NOT USE THIS", but if it's mentioned, some people will use this. And then compiler folks will struggle with a backward compatibility because of us.
As a language, "we" provide just some api to user(i.e. "How to use"-rules in some sense). We use compiler stuff only in some concrete compiler - that's all. Yeah, some things are already leaked as I wrote before. but documenting all private rakudo/whatever stuff is a nonsense. Firstly, because you don't need to know how rakudo internally works to write programs. Secondly, Rakudo can be changed everyday - do you want to reflect every commit? It makes our docs a bad source, because one day it's one, but another day it's something else. We don't have good enough docs in some places for even the standard language, it'll be strange to put one another great bunch of things to do. This parts will be always outdated.
If people want to dig into compiler stuff, they have sources/compiler docs/irc/etc. But we document language, not compilers insides.
The second section is debatable(I don't have anything against documenting things that are rakudo-only, but stable and provided to a user on purpose), but the first part is totally out of question for me.

someone finds an undocumented feature and uses it in his code

If someone finds an undocumented feature or use nqp magic or something else rather obscure, it's his/her own responsibility to do so. How often when you find something awkward about insert your favorite language here you hurry up to use in in production-close code rather than ask about it? It's like 'What if someone finds a gun and shoot himself in the foot, we need to document every gun with a big banner "Don't shoot yourself!"'.

If something is runnable from user level code AND is in roast

document it.
Is runnable from user level code AND is NOT in roast - check with the development team:
it might be accidentally exposed and need to be removed (And therefore not documented)
it might need tests in roast for (and therefore needs to be documented as part of v6.c or a later version)

We should not document everything. It's not up to the docs team to decide what's in the specification, that's up to the development team. There is some overlap, but the final decision rests there.

On Friday, August 12, 2016, Will Coleda<EMAIL_ADDRESS>wrote:
...

We should not document everything. It's not up to the docs team to decide
what's in the specification, that's up to the development team.

Is there a list of the dev team members anywhere?  If not, there should be
IMHO.

Okay @coke, what should we do with a bunch of issues about env vars? I don't see any tests…

@tbrowder  (list of dev team members) - #perl6-dev is a good first approximation. If it's something big, might get deferred to the collective @Larry (another non-explicit list of people, which at least includes Larry and Jonathan). Whatever the scope of the item, the discussion should get some kind of confirmation from the folks on IRC. Maybe add a tag for tickets that require a decision to make it easy to find.
@AlexDaniel - Regarding my recent list of tickets, excellent question - the rakudo/moar based ones can go on the rakudo wiki, and have a link to that from the spec page.

See a5ca28f - I added basically my notes above to the STYLEGUIDE at an attempt at closing this ticket. If there are particular issues that still need to be addressed, we can re-open the ticket and enumerate them.

OK, according to the latest discussion (started here https://irclog.perlgeek.de/perl6/2016-12-02#i_13669580 and conclusion here https://irclog.perlgeek.de/perl6/2016-12-02#i_13669787), we decided to keep notspecced things open. The whole thing needs more clarification, so the STYLEGUIDE will definitely help, but I'll reopen this in case somebody wants to express his opinion on the issue (because the decision was kinda sudden).

Wasn't sudden, it was "no one had anything else to say since Oct 24.".
I'll close this ticket again in a week if we don't have any more actionable feedback.

It's been a week, and no one has commented. Re-closing ticket.


Timing Information

Hi,
First of all, great work! I would like to get exact time information of the previously trained objects rather than the percentage of them. Is it possible if I tweak the current project?

Hey
You could split you record into pieces with some duration(0.5s - 2.5s) and evaluate each piece with known start position.

Brilliant idea!
Couldn't think of that. Thanks for the help.

Hi
I suggest you to split record to pieces multiple of 0.96 Because it's a size of one feature.


Detail of manula renewal operation behavior wanted to be explained.

Hi Team,
I see currently Godaddy can issue TLS/SSL Certificate with maximum valid period 397 days.
https://jp.godaddy.com/help/important-notification-about-ssl-offerings-9322?lc=en-US
But in this doc it's mentioned that manual renewal operation can be executed 60 days in advance from expiration date.
In this case, new certificate's expiration date will be shorter than expected expiration date (original exp date + 1 year) if user do manual renewal 60 days in advance, for example.
I think Doc should give answer to above scenario so could you please include those points?
・ What will be the behavior if user do manual renewal operation more than 30 days in advance? (new expiration date will be shorten compared to original exp date + 1year?)
・ What will be the behavior if user do manual renewal operation less than 30 days in advance? (new expiration date will be longer compared to original exp date + 1year?)
・How will be the behavior of exported certificate?
Thank you!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cc2ee874-df45-1de2-1b30-1fd75c7fd709
Version Independent ID: ee181722-8386-9842-407f-d0549012d2e9
Content: Add and manage TLS/SSL certificates - Azure App Service
Content Source: articles/app-service/configure-ssl-certificate.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

@yutaro113 Thanks for your feedback! We will investigate and update as appropriate.


Merge all other plugins

This adds all the other plugins into src/plugins, each in their own commits.
ping @bartaz

Based on earlier encouragement, merging this and moving forward to next PR (coming soon)


bicep what-if says existing subnet will be updated when importing existing vnet provisioned using ARM

I have an existing vnet with a bunch of subnets that were provisioned and managed using ARM templates.
We have a need to add additional subnets to the vnet. So, we have decided to move towards bicep and have created a template that would match the config of the existing vnet and subnets plus the new ones we want to add.
For example, we have an existing vnet with subnet a, b, c which was provisioned using ARM templates. I have now created a bicep template which matched the existing vnet and the existing subnets and have also added new subnets lets say d and e.
when I do a bicep what-if, i was expecting to see that the the output would say that subnet  d  and e would be added and no changes to either vnet or existing subnets.
However, what I see is, no changes to vnet config but subnet a is getting updated to reflect subnet d config, subnet b is getting updated to reflect subnet e config and subnet a, b, c are getting created as new.

Also, something that I have also noticed is, if i update the template to just include the existing subnets and remove the new subnets then it says no changes.
Please note that this is the first time bicep templates are being run on that resource group and until now we have been using arm templates to manage the resource.


question

"What crypting methods do you recommend?"
let me know thanks

??

to encrypt the rootkit for FUD?

I take that you look for a quick solution to FUD the rootkit? Use the shellcode installer, the documentation explains how to use it and why. You cannot achieve scantime evasion of the executable.

Thanks for answer but what about runtime FUD do you have solution there. Like the stager.exe/powershell
let me know

r77 is already 100% fileless and evades AMSI. However, eventually AV vendors catch up and find some way to detect it, since r77 is popular open source software. It wouldn't make a lot of sense to spend weeks on fixing AV issues only to have them detected again a week later. It's up to the individual and their skills.

oh okey i understand and question i need to crypt this part of the rootkit Several AV and EDR evasion techniques are in use:
AMSI bypass: The PowerShell inline script disables AMSI by patching amsi.dll!AmsiScanBuffer to always return AMSI_RESULT_CLEAN. Polymorphism is used to evade signature detection of the AMSI bypass.      can you give some guidiens?

That's one possibility. You'll need to analyze what gets detected and change accordingly.
I could do this and publish the results, too. But this would take me weeks, after which my evasion will break days later. Anyway, since it's all fileless and so on, it's entirely possible. Can't say this about rootkits that use AppInit_DLL's.