[Q] Can we access the hyperparameters used to run Sweeps?

Hello there, I was wondering if there's any way we can access the hyperparameters that wandb agent runs during Sweeps? I would like to create a python script that calls wandb sweep, runs wandb agent, gets a set of hyperparameters for that run, and uses those hyperparamters for running another downstream task. Is something like this possible with how Sweeps are defined currently?
I also looked into the high-level overview of sweeps in this comment and it seems that in my plan above, I am currently bottlenecked by my understanding of how the Sweep Controller samples hyperparameters (that the agent subsequently runs). Any help on this is greatly appreciated, thank you in advance!

Hi @naga-karthik thanks a lot for your question! The agent isn't currently exposed through our API, therefore it won't be possible to get its attributes externally. It's only feasible if your downstream task is being called by the main function as in this example. This main function can access the wandb.config as there's a run initiated within it. Would this work for you? If not, could you please provide a bit more context on what this task is doing?

Hey @thanos-wandb, thanks for getting back! Sure, here's some more context. Essentially, I am trying to integrate wandb's Sweeps functionality into a package that trains a model by specifying configuration .json files that contain all the necessary hyperparameters. What I had in mind was that if I could retrieve the sampled hyperparamters from wandb agent, then I can create a new configuration .json file for each combination of hyperparameters defined in sweep_configuration and then essentially launch multiple instances of my package.
Here's the idea in terms of pseudo-code:

# define sweep configuration
sweep_configuration = {
    'method': 'random',
    'metric': {...},
    'parameters': {'batch_size': {'values': [16, 32, 64]}}
}

# get sweep id
sweep_id = wandb.sweep(sweep_configuration, project=project_name)

# run agent
wandb.agent(sweep_id, count=5)

# retrieve hyperparameters from the agent
....

# create config file(s) based on retrieved hyperparamters
config_1 = ...

# launch my package with config_1
...

Please let me know if this makes sense! Hope I have not confused you even more!
I also have a follow-up question. When you say:

This main function can access the wandb.config as there's a run initiated within it

Does this mean that the hyperparameters that wandb agent uses is stored in wandb.config for each combination of hyperparameters (which then becomes a run in Sweeps dashboard)? If yes, then if we have access to wandb.config (after running wandb.agent), we also can get the hyperparameters that will be used for sweeps right? Is my understanding correct?

Hey @thanos-wandb, just wondering if you'd have some suggestions whether something could be done here? Thanks!

Hi @naga-karthik,
Have you tried getting the configs from wandb.config.
define sweep configuration
sweep_configuration = {
'method': 'random',
'metric': {...},
'parameters': {'batch_size': {'values': [16, 32, 64]}}
}
get sweep id
sweep_id = wandb.sweep(sweep_configuration, project=project_name)
run agent
wandb.agent(sweep_id, count=5)
retrieve hyperparameters from the agent
create config file(s) based on retrieved hyperparamters
config_1 = wandb.config
launch my package with config_1
...

Hey @scottire, thanks for your suggestion and apologies for the delay in my response! I tried your suggestion using a toy example in wandb's docs and I have one question:
I noticed that after running wandb.agent(sweep_id, count=5), the program first completes the 5 runs using the agent before getting to config_1 = wandb.config. This is rather unintended for my use case since I thought there could be a way to just get the combination of the hyperparamter values from wand.agent, and then do the sweeps? I'm interested in the specific hyperparamter values that are used instead of the final result (of using some combination) on the Sweeps dashboard.

Hi @naga-karthik thanks for the additional context and the pseudo-code, that's really helpful. I wanted to follow up here and provide you with the following ways to access the values, each solution depends on the phase you would want to access the hyperparameters:

after wandb.agent completed, that's feasible using the API:

api = wandb.Api()

sweep = api.sweep("thanos-wandb/my-first-sweep/3o9w3c3q")
sweep_runs = sweep.runs

for sr in sweep_runs:
  print(sr.config)

during wandb.agent the agent calls a main function that will consecutively initiate the experiments. You could access for each run the wandb.config inside the main function, and adjust your code so as to append to a dictionary argument.

before executing wandb.agent, you could mock the sweep runs with the sweeps package, which you can install as pip install sweeps. A minimal code example to get the runs, and their config:

from sweeps import next_run, stop_runs, SweepRun, next_runs

sweep_config = {
        'method': 'grid',
        'parameters': {'a': {'values': [1, 2, 3]}}
}

suggested_run = next_runs(sweep_config, [], n = 3)
display(suggested_run)

for sr in suggested_run:
  print(sr.config)

Please let us know which option would best work for you, and if you have any further questions.

Hey @thanos-wandb, thank you very much for getting back! Your suggestions are really helpful and option 3 is just what I was looking for!
Just a quick follow up question - let's say I go ahead and get the mock configs from the sweeps package and run my downstream task (which only uses wandb.log for tracking my model training) using these configs. Then, unless I am running, wandb.agent I won't be seeing my results on the Sweeps dashboard right?
In other words, does the sweeps package have any link to the Sweeps dashboard such that I don't have use wandb.agent? The thing is, the sweeps dashboard has useful information about the hyperparam combinations so I would like to get that, but, just wondering if there is a way to bypass wandb.agent? Thanks again in advance for your help!

Hi @naga-karthik glad you've found this useful, and that it works for you! The wandb.agent helps to orchestrate all the runs for the certain sweep, and will automatically create a parallel coordinates plot in your project page. However, if you wanted to avoid using wandb.agent and you are running individually wandb.run from your downstream task, you should be still able to Add panel > Parallel coordinates plot from the UI (project level). You can manually then add the config parameters to this chart, would this work for you?

Hi @naga-karthik just checking in here to see if the above would work for you, and if you have any further questions that we could help with? thanks!

Hi @naga-karthik since we haven't heard back from you in a while, I will go ahead and close this ticket for now. If you still have any questions, please let us know and we will be happy to keep investigating!


High Contrast Color Theme - Dark

Adjusted the dark color theme

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Function from ?

How about a function from(x: f64) → Interval?  Its definition would be:

from(NAN) = ∅
from(+∞) = [f64::MAX, f64::INFINITY]
from(-∞) = [f64::NEG_INFINITY, f64::MIN]
otherwise from(x) = [x,x].

This could be useful for producing constants in generic code.  What do you think?

This could be useful for producing constants in generic code.

Would you give me an example?

Do you have any rationale behind these definitions?

from(NAN) = ∅
from(+∞) = [f64::MAX, f64::INFINITY]
from(-∞) = [f64::NEG_INFINITY, f64::MIN]

I don't understand what you mean by this. Could you be more specific?

Suppose I want to write code generic in the type of the interval (in case I need multiprecision intervals in the future).  That code is likely to need numerical constants and it would be good to have a generic way of converting them to intervals.  Example simplified from real code:
struct D2<I> { d11: I, // ∂ₓₓ
               d12: I,
               d22: I }

fn mean_integ<I>(f00: &I, f10: &I, f01: &I, f11: &I, d2fxy: &D2<I>,
                 h1: &I, h2: &I) -> I
where I: AbstractInterval<I> {
    let err11 = *h1 * *h2 * bound_of(d2fxy.d11);
    let err12 = *h1 * *h2 * bound_of(d2fxy.d12);
    let err22 = *h1 * *h2 * bound_of(d2fxy.d22);
    let err = (err11 + err12 + err22) / I::from(12.);
    (*f00 + *f10 + *f01 + *f11) / I::from(4.) + err
}

Do you have any rationale behind these definitions?

I was thinking that from(f(x)) where f: ℝ → [-∞,+∞] should return an enclosure of f(x).  If f(x) is too large and is rounded to +∞, then the appropriate interval is [f64::MAX, f64::INFINITY].  When f(x) is NaN, it means that x ∉ Dom(f), in which case the interval extension should return ∅ (and, so, from).

If f(x) is too large and is rounded to +∞, then the appropriate interval is [f64::MAX, f64::INFINITY].

OK, that makes sense. But this definition can confuse users, since the following does not hold when a == f64::(NEG_)INFINITY:
const_interval!(a, a) == Interval::from(a)
(Note that const_interval!(a, a) results in a compilation error.)
In addition, I hesitate to add non-standardized operations at this moment. cf. https://github.com/unageek/inari/pull/67#issuecomment-1008232465


Update sample templates

I just wanted to be clear as I thought we decided on Quix Code Samples, and perhaps Code Samples?

Not sure what you're referring to here. Is there a specific problem somewhere?

Hopefully all done now @tbedford .
This felt like scope creep on the ticket. Lets create separate tickets for things like this in future.


Feat implement menu pattern

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Widget using the Trix Editor

Is your feature request related to a problem? Please describe.*
Better rich text editing
Describe the solution you'd like
https://github.com/basecamp/trix

This would be great for a custom widget. I will leave here for the one of the community can create. Of course, @jclusso , taking a stab at this by using one of the existing widgets as an example would be an option for you too.
Note: This couldn't take the place of the existing editor, because there needs to be the option for markdown output into the body of the markdown file.


@1aurabrown => Add click tracking for popular artists

Closes https://github.com/artsy/collector-experience/issues/83

cc @anipetrov

Can we change the event name to Click? We are trying to remove the clicked_on_... events since Segment is creating separate tables for each of those and really clogging our system -- for any click that is not of top priority, we would rather have a catch-all Click event that is appropriately propertirized (since we already get things like context_page_path from segment, with proper label and context_module we should be able to differentiate between most unessential clicks easily enough).

Can we change the event name to Click?

Will do- you actually noted that in the original issue!

Awesome, thanks @mzikherman!


Blog: Smart Debug Logging

@jeffgdotorg @indigo423 If you have some free minutes, please review the blog and the wiki article.

/cc @nmsmanjohn


Automatic PR from RestAPI-PR1466

Generated from RestAPI PR: https://github.com/Azure/azure-rest-api-specs/pull/1466

Codecov Report

Merging #1328 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1328   +/-   ##
=======================================
  Coverage   56.04%   56.04%
=======================================
  Files        2691     2691
  Lines       71246    71246
=======================================
  Hits        39932    39932
  Misses      31314    31314

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f477734...4d7ef11. Read the comment docs.

Not relevant


Added composite indexes to subject and causer

Today I noticed a significant decrease in performance for the polymorphic subject and causer relationship. I only have about 4.5 million records, so this surprised me. This PR adds a composite index for the causer and also one for the subject. In my case, this increased the performance quite significantly.
Perhaps this helps others. Also, thank you for the amazing package Spatie :smiley:
EDIT:
The numbers are as follows:
select * from activity_log where subject_id=5749;

This returns 10 rows, duration:
Duration / Fetch
13,362 sec / 12,262 sec

After the migration, I'm executing the exact same query and get these durations:
Duration / Fetch
0,053 sec / 0,055 sec

Thanks for your input.
I will merge this with the next other PR. Because the migration is primary an idea and for existing users this will be not usable directly I don't want to use a release only for this.
Just for completion: could you add some screenshots or copied numbers to prove your performance boost?

@Gummibeer I've got some raw numbers from MySQL Workbench:
Query:
select * from activity_log where subject_id=5749;

This returns 10 rows, duration:
Duration / Fetch
13,362 sec / 12,262 sec

After the migration, I'm executing the exact same query and get these durations:
Duration / Fetch
0,053 sec / 0,055 sec

So the duration is now 252x faster, and the fetching is 223x faster.

Great PR, thanks!

Perhaps the migration can be further optimized to use Laravel's default nullableMorphs method? It creates the correct fields and indices for you. We've used it in our migration a while ago.
$table->nullableMorphs('causer');
$table->nullableMorphs('subject');


PLATUI-2358: Make sbt-test-report a public artefact

Update build.sbt to make build artefact public
Update project version to 0.2.0

Scala can be updated from 2.12.17 to 2.13.8 in build.sbt - See confluence https://confluence.tools.tax.service.gov.uk/x/L4HJHw

Ignoring this as Scala 2.12 is currently required for the plugin to function.


TypeError: Path must be a string. Received undefined

Hi, after updating to 1.3.4, this package stopped working for us. I believe the culrpit is this commit, particularly the removal of this line:
name = _.trimStart(_.trim(name), path.sep + '.') || path.basename(file);

We were sending null as a name param, as we wanted to use the basename of file. This logic is now missing and line if (path.isAbsolute(name)), throws an exception.

@george-cz That's definitely a regression bug. Sorry for the inconvenience, currently pussing out a bugfix release targeting the issue.


unable to fork from github to azure repos

In the first section or this tutorial, Get the code,  the instruction is to fork the sample app in Github or Azure Repos. I need to use Azure Repos. But there's no option in Azure Repos to fork from Github. The only way to get the code from Github into azure repos appears to be to import or clone. I don't think this is the same as "fork". A fork is different than a clone or an import. It maintains a relationships to the original code that was forked. I don't know why forking is necessary for this tutorial, rather than importing or cloning, but I'm just trying to follow the example. Could you please provide an explanation how to fork the Github repository to Azure Repos? or should I just clone instead of fork? Any help would be appreciated. Thanks.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 4cfc7aa5-d9ff-ad64-d26e-d2bd10470025
Version Independent ID: 9b4f02ea-dab9-f613-5345-496435ab2408
Content: Deploy a web app to Azure Web App for Containers - Azure Pipelines
Content Source: docs/pipelines/apps/cd/deploy-docker-webapp.md
Service: azure-devops-pipelines
Sub-service: azure-devops-pipelines-apps
GitHub Login: @juliakm
Microsoft Alias: jukullam

@robidas -- Rob, thank you for your feedback.  Please consider these resources:

Azure DevOps Services
Azure DevOps on Stack Overflow

@juliakm -- Julia, please look into this issue.

@robidas You're right, you can't fork in Azure Repos from GitHub. You'll need to import the repository instead. Here's an overview doc.
I'll update the article to make this clearer.


Regexp exercise 15): solution improvement

First of all, thanks for the great tutorial and the CLI app for exercising!
Sorry, I accidentally pressed Submit before finishing my issue. Here's the rest.
In question 21/88, i.e. exercise 15) of the Regular Expressions section, the proposed solution is this:
https://github.com/learnbyexample/learn_gnuawk/blob/96b8442fc372303f1be838bf01a2210ded152b58/exercises/Exercise_solutions.md?plain=1#L244-L250
Isn't this a shorter solution:
awk '/(.*ar.*){3,}/{print gensub(/ar/, "X", NF-2)}' patterns.txt

Also, is the phrasing "last but second" correct? I was rather confused. Is the intended meaning the same as that of the word "antepenultimate"?

There were some bugs in the original awk code I provided, they occurred because I had to re-write them here in the issue as I couldn't copy-paste them - the CLI app doesn't allow selecting any content. I see this as an issue - should I open a new one here on GitHub? Or is that inherent to that kind of application?

Your solution won't work for input like par:car tar-far;Cart since you are making use of fields. The question is intended as a search and replace problem. I should probably add such an input line or make the question clearer.
Regarding last-but-second:
par car tar far Cart
                 ^^last
             ^^last but one
         ^^last but two

Regarding the CLI app, I don't know if there's a way to select content. I could probably add a button/short-cut to copy the content in the Input box, but other than that I doubt I'll be able to add other features. I'm using a third-party Python module for that app and I'm not really familiar with it.

Building upon your idea, I think awk -F'ar' 'NF>3{print gensub(FS, "X", NF-3)}' will work. Will update if I can think of case where this'd fail.

Thanks for the reply! About an hour after asking I realized that I was making use of fields, but wasn't at the computer to reply. Anyway, I'm still a newcomer to awk, so I'll try forgiving myself this mistake.
Here's an interesting link regarding "last but second".


Question about the content of the  .torch file

Thans for your work！I check the content in .torch file, and the attributes  in 'cameras' has 18 parameters，what does each parameter represent (as the first two items should represent the dimensions of the image)？

You can see how they're used here:
https://github.com/dcharatan/pixelsplat/blob/1947b629da93b7327cb6d7a4901e77500a566231/src/dataset/dataset_re10k.py#L175-L198


TensorFlow 2.0 using pretrained word embeddings as input to sequence model

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information

TensorFlow version (you are using): TF 2.0 Beta
Are you willing to contribute it (Yes/No):
Describe the feature and the current behavior/state.
I've just started exploring TF2.0 and it appears that the word embeddings must be generated as part of the seq2seq model.

Will this change the current api? How?
embedding layer would need to be modified to not train and permit inclusion of pretrained word embeddings
Who will benefit with this feature?
anyone who would like to use pretrained word embeddings**Any
Other info.**
inclusion of pretrained word embeddings would further expedite training so a huge win.

Hi @NeuralGirl,
If you already have pretrained embeddings, you should be able to build a model that takes pre-embedded data as inputs, and add a non-trainable lookup to your embedding matrix to your pipeline so as to conduct the embedding before feeding to the network.
E.g. something in the vein of:
class PretrainedEmbedding(tf.keras.layers.Layer):
    """Non-trainable embedding layer."""

    def __init__(self, embeddings, rate=0.1):
        """"Instantiate the layer using a pre-defined embedding matrix."""
        self.embeddings = tf.constant(embeddings)
        # if you want to add some dropout (or normalization, etc.)
        self.dropout = tf.keras.layers.Dropout(rate=rate)

    def call(self, inputs, training=None):
        """Embed some input tokens and optionally apply dropout."""
        output = tf.nn.embedding_lookup(self.embeddings, inputs)
        return self.dropout(output, training=training)

my_model = tf.keras.models.Sequential([
    tf.keras.Input(input_shape, dtype=tf.int64),
    PretrainedEmbedding(embeddings_array),
    # insert any model architecture here
])

If I am missing something or if you would like to indicate a more precise use case your issue arises from, please let me know :-)

Thanks for the prompt followup! Yes, this is what I am looking to do in TF2.0. Thanks!

You are welcome :-)

Hi, I've tried to use the aforementioned class to load pretrained embeddings, but the an error message is thrown: AttributeError: 'PretrainedEmbedding' object has no attribute '_trainable'

@luiz-resende The code I posted was lacking a super call in its __init__ method; I edited it so that it should be workable. Note that to be rigorous in respecting the keras API you should also write a get_config method and a compute_output_signature one for that class, which I did not provide as this is a simple layer sketch.
Let me know if you run into additional issues!

Note that another take at the initial issue could be to assign the pretrained embeddings to an instantiated and built Embedding layer set to be non-trainable (i.e. whose weights would be frozen).
Something in the vein of:
layer = tf.keras.layers.Embedding(input_dim, output_dim, trainable=False)
layer.build((None, input_dim))  # replace None with a fixed batch size if appropriate
layer.set_weights([pretrained_embeddings_tensor])

Thanks for followup and for the help! I've tried the second option and it worked perfectly and way cleaner in my model (still transitioning from TF 1.x to 2.0). Thanks again!

You are very welcome :)

You can do this through:
embedding_layer = tf.keras.layers.Embedding(...)
embedding_layer.set_weights(pretrained_ embeddings)

I think that simply tf.keras.layers.Embedding(weights=[pre_trained_embeddings]) should work and that way one doesn't need to call build and set_weights manually.


Updated logging to use logrus and set log level and format during runtime

Description
Updated logging in karavi-metrics-powerflex with following changes:

Use logrus for all logging (removed all other log and fmt.Printf statements)
Support the updating of log level and format during runtime without the need to restart the service

Issues
List the issues impacted by this PR:

Issue ID

https://github.com/dell/karavi-observability/issues/43

Checklist:

[x] I have performed a self-review of my own code
[x] I have run 'make check' to ensure my code does not have any formatting, vetting, linting, or security issues
[x] I have run 'make test' to ensure new and existing unit tests pass locally with my changes
[x] I have not allowed coverage numbers to degrade
[x] I have maintained at least 90% code coverage
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

[x] Deployed karavi-metrics-powerflex and updated config map with different log levels and format. Verified the correct log statements were printed.

@tdawe: Thank you for submitting your pull request!
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at<EMAIL_ADDRESS>

run e2e test


Low hashrate on Fedora 35/RX 6800 XT

Hi,
I compiled acominer from the source, without recompiling Mesa nor applying your hrtimer kernel patch, and, using AMDVLK, I get a low hashrate
32.21 MH/s 22.96ms
32.07 MH/s 22.97ms
compared to 61Mh/s I can get with teamredminer. I would like to leave OpenCL... do you have any idea about the reason?
May I get a better performance with AMDGPU PRO or is there any flag/parameter I should set when I run acominer?
This is my launch command:
./acominer stratum://0x76cB31Fdb28Ddf887AF3cA4A0f7786D02C86032a.Provaio@asia1.ethermine.org:4444
Thank you in advance.

Do not use AMDVLK. Use the binaries provided at https://github.com/ishitatsuyuki/acominer/releases/tag/v0.4.0.

./acominer --subgroup-size 128 stratum://0x0Df9589cA10f33A78269EeeB92314CEA9D59E5E7.amdvlk@eu1.ethermine.org:4444
2021-12-23T15:16:45.426476Z  INFO acominer::stratum: Pool set difficulty difficulty=4.30G
2021-12-23T15:16:46.159825Z  INFO acominer: device=AMD Radeon RX 6800 XT type=DiscreteGpu
2021-12-23T15:16:46.163612Z  INFO acominer: subgroup_size=128
2021-12-23T15:16:46.171956Z  INFO acominer: Generating light cache (CPU)...
2021-12-23T15:16:50.412397Z  INFO acominer: Light cache generation done.
2021-12-23T15:16:50.530106Z  INFO acominer: Light cache transferred to GPU.
2021-12-23T15:16:50.531210Z  INFO acominer: Building DAG... epoch=462
███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 1208320/1208320
2021-12-23T15:16:51.864310Z  INFO acominer: DAG built duration=1.333104401s
96.90 MH/s 7.59ms
96.77 MH/s 7.59ms
96.77 MH/s 7.59ms
96.78 MH/s 7.59ms
96.77 MH/s 7.58ms
96.78 MH/s 7.59ms
96.75 MH/s 7.60ms
96.58 MH/s 7.61ms
96.53 MH/s 7.61ms
96.48 MH/s 7.61ms
96.43 MH/s 7.61ms
96.43 MH/s 7.62ms
96.43 MH/s 7.61ms
96.44 MH/s 7.61ms
But just few shares are accepted.
But I'm still with AMDVLK.

You're glitching out the calculation with subgroup-size=128. That's not a valid option and what you are seeing is just fake doubled hashrate.
As I said, the only driver supported right now is the Mesa fork. AMDVLK has its own issue with shader compilation, and only runs at roughly half the theoretical capacity. If you don't want to build it, grab the prebuilt version from releases.

You mean that running run.sh will load your Mesa fork vulkan driver? Which kind of performance uplift should I expect using your miner?

You mean that running run.sh will load your Mesa fork vulkan driver?

Yes, it loads the bundled Mesa fork build through an environment variable override.

Which kind of performance uplift should I expect using your miner?

On RDNA, it should be roughly on par with teamredminer.

Do I need AMDGPU PRO proprietary driver?

You need not. The proprietary AMD Vulkan driver, in fact, performs even worse.

ok nice, using the prebuilt package, it works fine as you mentioned! Thank you and have a nice holiday


⚠ Thumbnails API Endpoint has degraded performance

In 828f25d, Thumbnails API Endpoint (https://thumbnails.roblox.com/v1/assets?assetIds=82738847&format=Png&isCircular=false&size=30x30) experienced degraded performance:

HTTP code: 200
Response time: 1558 ms

Resolved: Thumbnails API Endpoint performance has improved in adf7d5a.


Introduce SIMPLE_SERIAL_SHELL_BUFSIZE to make it configurable

Hi,
There is default limit in 88 bytes as maximum input line length, but it's not enough for our needs.
These changes add possibility to redefine that limit with build flags.  Example for PlatformIO:
build_flags = -D SIMPLE_SERIAL_SHELL_BUFSIZE=256
P.S. Thanks for simple, but quite useful lib :)

@eg321 This change looks promising.  In principle I would not mind merging it in.  I am glad you are  able to adjust the buffer size to meet your needs when you use PlatformIO.  Your change looks clean, if PlatformIO can easily declare project macros.
Remember that at this time the Arduino IDE is my focus for this library.  If there are other platforms where it works that's mostly a happy coincidence -- helped a bit by keeping the SimpleSerialShell very SIMPLE.

Are there other platforms where this library works?
What platforms are we committing to support?
If I make a change, how do I confirm I do not break support for these other platforms?

If we can easily support adjustable buffer size for all platforms that would be fine.  But if adjustable buffer size is only available under PlatformIO, it might be better to fork the library (which you could then customize/tune even more).
Fortunately the SimpleSerialShell library is stable.  I'm not planning on major changes at this point, so it's unlikely you would miss much with a code fork.
That said, if making buffer size configurable still makes sense...
Until now SimpleSerialShell had no need for configuration.  If this library is going to be configurable, there should be some documentation on how to configure it... (and warnings about when it cannot be configured).

Can you add a CONFIGURATION.md description showing how to adjust buffer size, and when it might be necessary?
Can this pull request adjust the buffer size when using this library in the Arduino IDE? How would we make that work?

@philj404, just to be on the same page:

PlatformIO is not another platform. It's an IDE that works with many frameworks including Arduino. It can use most libraries for Arduino without any issues (according to my experience).
Not sure about other platforms. If you mean "IDEs"  here then your lib should be supported by any IDE that supports Arduino framework most probably. For example, Visual Studio Code should work also with your lib by my opinion. But it can be checked separately.
Proposed changes are backward-compatible. They are simple and based on "#define" directive that supported by all C-compilers. Most build environments allow to configure build options and define values for #defined macros. PlatformIO as versatile IDE allows to define such options per project. I'm not very experienced in Arduino IDE, but seems it's allow to configure that per-board or globally (quick googling confirms that).

So, if you're ok with these changes, I'll add some CONFIGURATION.md where describe how to configure buffer size for PlatformIO and Arduino IDE. Most of your users can do nothing and use default value as before, but in some cases (like for us) it can be very useful to increase buffers.

I think I'm still OK with your suggested changes.  I'm looking forward to seeing your CONFIGURATION.md.
Here's some explanation about how the Arduino-IDE affects Arduino-libraries:
While the Arduino-IDE is not exactly the Arduino-library (platform), some of the IDE's limitations have side effects on what its libraries (or platform) can do.
Is there a way for the Arduino-IDE to inject a define to its build environment, as you did within PlatformIO-IDE?
build_flags = -D SIMPLE_SERIAL_SHELL_BUFSIZE=256
I would REALLY like to take advantage of that capability, while still using the Arduino IDE.
A less-simple but more-portable way to allow adjustment of shell buffer size would be to push the shell constructor out into the sketch (instead of having it done automatically by the library).  For example all .ino files could create the shell instance and specify its size.  I specifically tried to avoid this as it's yet another thing that a user could get wrong.  Maybe this flexibility is worth the complication.  Maybe.
Of course requiring that constructor call breaks backward compatibility...

By the way, why do you need more than an 87 character buffer for human text input?  I am curious about your application.

Ok, I've added CONFIGURATION.md. Feel free to correct it as you wish (English is not my native language).

A less-simple but more-portable way to allow adjustment of shell buffer size would be to push the shell constructor out into the sketch (instead of having it done automatically by the library). For example all .ino files could create the shell instance and specify its size. I specifically tried to avoid this as it's yet another thing that a user could get wrong. Maybe this flexibility is worth the complication. Maybe.
Of course requiring that constructor call breaks backward compatibility...

You're right. Your current solution creates buffer statically. It's more robust solution than create it dynamically with needed size.

By the way, why do you need more than an 87 character buffer for human text input? I am curious about your application.
We have several low-level commands that use argument as raw data for packet generation (128 bytes of  copy-pasted HEX-coded string). That command used by some automation also.

So, can you merge it now and do other "magic" to publish new version with these changes? :)

@eg321 I was curious about adjusting buffer size in a more portable way.  I think I have something that could have "worked"... (If you are curious to see what I had in mind look at 2c629a8cd4e4701c0e2cab58e63426e320d6e6a8)
...But my changes were not as SIMPLE and EASY TO UNDERSTAND as your #define change.
Simplicity is very important to me.  Programmers should be able to look at my code and recognize "this does not do what I need", rather than never being quite sure how it works.
I will merge your change.  Thank you for the contribution!

@eg321 OK... this is going faster than I thought it would.
I think the changes are now all in my "next" branch.  I think that it is current as of commit 06f2a0db0412a6fecc0a2a9353a999c29919847e.
But before I do the final merge to main and tag the library for release, would you please confirm that everything you want is correctly in place?
For example,

does "next" work for your application in your build environment?
Does the documentation agree with how you think the library is supposed to work for you?

Thanks again!

@philj404, "next" looks ok for me! Waiting for next version.
Thank you!

@eg321 I created a new release v0.2.1.  It has your changes except the CONFIGURATION.md documentation is not quite what your branch had in it -- it looks like there is a merge conflict.
If what I released is good enough for you, let me know and I will close this PR.
If not -- for example the documentation needs correction -- please provide edits as needed and I will put it into a v0.2.2 release.

@philj404 it looks great for me. One strange thing: there are no fresh versions displayed for your lib - https://platformio.org/lib/show/7399/SimpleSerialShell
0.1.1 displayed as latest.
Some more changes required to complete "release"?

@eg321 Thanks for checking.  I am able to see my FOUR tagged releases for SimpleSerialShell in the Arduino IDE's "manage libraries":

0.1.0
0.1.1
0.2.0 (from November 22, 2020)
0.2.1 new
I do not know why PlatformIO has not detected the newer 0.2.x releases.  Perhaps they have not updated their list recently?

They should crawl repos each day. I've addressed a question to their community. Let's wait for response: https://community.platformio.org/t/latest-versions-of-simpleserialshell-lib-are-not-showed-in-platformio-repos/19387

@philj404 it was caused by renaming "master" into "main". PIO team has updated info at their side.
Now it works ok. See details at url above.
Thanks.

@eg321 Thank you for following up.
I had not realized PlatformIO picked up the library, but has different rules for picking up releases compared to how the Arduino IDE does it.


I find the project very useful, please could you add the ability to control spa temperature from an HA Automation

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Use an action of service and select climate.set_temperature. Then select the spa hvac control and you should be good to go.
The yaml should look something like this …
service: climate.set_temperature
data:
  temperature: 15
target:
  device_id: f4ffc72bf…..5d4bf68b


Browse OTUs Task - Type Specimen History

The history of the status of a type specimen has to be clearly displayed in Browse OTUs. For example, if a syntype was chosen to be a lectotype it has to be displayed as the same specimen and not as different type specimens.
For example, in the OTU Bufonacris claraziana (Saussure, 1884) the syntype specimen is the same specimen that later is chosen as the lectotype of the species. Not a diferent one.

@jlpereira I think we'll just have to nest by collection_oject_id here


Blob search credentials do not work

Hi folks, I ran into an issue because the connection string is not working, the indexer indexes 0 files. I changed the connection string removing all endpoints but the Blob, and now it is working. I'll send a pull request

I'm running into a similar issue in notebook 01-Load-Data-ACogSearch.ipynb when creating the indexer. I've tried both the original connection string and a version with only the Blob endpoint.
'{"error":{"code":"","message":"Error with data source: Credentials provided in the connection string are invalid or have expired.\\r\\nFor more information on troubleshooting connection issues to Azure Storage accounts, please see https://go.microsoft.com/fwlink/?linkid=2049388  Please adjust your data source definition in order to proceed."}}'
Is the demo data storage account still available for use? Is there a recommended way to upload a copy of the demo data into our own storage accounts?

I'm running into a similar issue in notebook 01-Load-Data-ACogSearch.ipynb when creating the indexer. I've tried both the original connection string and a version with only the Blob endpoint. '{"error":{"code":"","message":"Error with data source: Credentials provided in the connection string are invalid or have expired.\\r\\nFor more information on troubleshooting connection issues to Azure Storage accounts, please see https://go.microsoft.com/fwlink/?linkid=2049388 Please adjust your data source definition in order to proceed."}}'
Is the demo data storage account still available for use? Is there a recommended way to upload a copy of the demo data into our own storage accounts?

I am no longer running into this error. I was able to make progress with the latest connection string from the main branch.


Fix inclusion of HunterGate.cmake

#4  was fixing the inclusion of the file, but that was the wrong place to fix it.
In my original code at the top of the file I had a file(DOWNLOAD statement, which I wrongly replaced with include(cmake/HunterGate.
Yesterday I didn't have the time to test #4. Today I've tested my gate fork and it works as expect.
https://github.com/cpp-pm/hunter/issues/56#issuecomment-550152870 had the right suggestion, and I somehow didn't see it.

Can you provide some evidence of testing so that we know this will work as expected? Maybe try running the example from the GitHub issue that flagged this?


Improve or document testability by users

One of the problems I am facing with TTY::Prompt is the lack of formal way of testing interaction with the menu (at least to my knowledge).
Up until now, I was hacking it by capturing STDOUT and STDIN, and injecting keystrokes into it, for example, like can be seen here
This proved to be fragile, as evident in https://github.com/piotrmurach/tty-screen/issues/11.
I would love to have a formal way to test it, which would allow me to "inject" output to the prompt - regardless of what type it is.
Pseudo-code wise, I would love to include TTY::SpecHelpers in my rspecs, and then write something like:
tty_send_input :down, :down, :enter do
  subject.execute ...
end

Or, by stubbing a single, known input handler:
expect(TTY::Prompt.input_handler)
  .to receive(:get_input)
  .and_return [:down, :down, "hello", :enter]

I see, I see... Thank you. Looking forward to it.

Hey @DannyBen, I feel your pain, I'd been in your exact situation just recently and stumbled upon the same things. My case was also worsened by the fact that I had a multi-stage CLI app with multiple consequtive questions/choices.
Basically we need a system where we can feed stdin of an app with any content and expect it to just work.
As Piotr mentioned above I ended up with a way of achieving that without too much black magic. I am now in the process of convincing Piotr that this way of testing is worth being included into ttytoolkit family, please wish me luck :-)

HI @DanielVartanov , happy to hear something is in the works.
I consider myself a heavy TTY::* user, and have quite a few tools using it, and many that use TTY::Prompt in particular.
In my projects, I also pay attention to have efficient tests, so I also usually tend to avoid copy-pasting repetitive code. For this purpose, I developed rspec_approvals, which is an interactive approval testing - meaning, it will show you the output, you can approve/reject it, and it will ask you again if the output changes.
Why am I telling you this? In some of my specs, I simply create a YAML file with a) command to run and b) keyboard input to feed, and let the specs execute them. I then approve the output interactively (using TTY::Prompt of course), and the specs will "let me know" if any output changes from the approved content in the future.

Here is an example of such YAML file
Here is how it is being fed to rspec
Here is the spec_helper that provides the interactive "feeding" of stdin

This approach works, and as you can see (by the spec helper code), it requires only a handful lines of code. Zero black magic of any kind. The problem with it, is since it is not "officially" supported, any change (such as a recent one in TTY::Screen) could potentially break it.
Long story short, if there is any repo or gist where I can take a look and possibly contribute to this effort, let me know.

@DanielVartanov I'm 100% convinced. Apologies if I made you think that I don't approve of this idea or don't want to make it part of TTY toolkit. I thought I was voicing encouragement and supporting your experiments. I really do like the look of where the experiments are going! I juggle many things so for me it's more matter of finding time than anything.
So to bring everyone on the same page, here are my current thoughts:

TTY toolkit needs a dedicated testing solution, therefore I'm keen to pull together any helpers that may be reusable into a single gem.
I'd love this to power the unit and integration testing for the tty gem itself. For some time, I wanted to improve and provide out-of-the-box experience to make testing much simpler.
There are many testing frameworks but I don't believe we need to provide any specific assertions. The implementation could be exposed as a module that can be included in any test class.
No hard dependency on tty gems to keep the interface as flexible as possible. The core principle of tty gems is that they are easy to mix & match with other libraries. However, I envisage a specific module that provides some niceties for testing tty-prompt like sending keystroke names. In the spirit of what Danny has suggested.

I will setup repository soon so we can collaborate.

@piotrmurach that's awesome to hear! And please don't apologize, it's job of an offerer to convince the maintainer, as there's enough people out there who just bully maintainers to do what they want.
Anyway, I too can devote time only once per few days, but I am using soon-to-be tty-testing in a real project so I constantly get experience of what works and what only sounds good. So, please take your time, no need to rush.
@DannyBen awesome lib! Love to community developing artful console tools and apps. Let me share a couple of examples of what I am trying to do here: https://gist.github.com/DanielVartanov/89e08ca436eb73cbaeaae1912f3133cd
Perhaps in the particular case you shared this will make little difference for you, but next time when you do a multi-question CLI app with subprocess (called via tty-command) which must comply with what you feed to stdin too, this approach will help you a lot.

@DannyBen I refactored tests and made TTY::Prompt::Test public to help test prompt interactions. This should help you make your tests more robust and immune against changes. The tty-test gem will make testing much nicer and allow for more complex interactions but I see this as a reasonable intermediate step given that I use it internally to test tty-prompt.

Looks cool, thanks! Will give it a try.

Today I'm ready to present a first proper draft of the testing tool for tty toolkit
https://github.com/DanielVartanov/tty-testing
@DannyBen please check whether it fits your purposes

The tty-test gem will make testing much nicer

Right, I see it should've been named tty-test, let's rename it after I transfer the ownership to you.

Wow, so many TTY gifts in one weekend!
I will take a look soon, thanks for the notification.

Ok. My initial observations for the TTY::Prompt::Test approach.

There seem to be a small mistake in the README. We need to require 'tty/prompt/test' and not tty-prompt/test.

The example provided in the README is helpful, but might confuse novice users. In reality, the prompt instance is not the subject under test . The prompt instance will normally be tucked away inside some subject, assuming the developer considered dependency injection.

I have created a sample gist showing how real life specs might look like ("real life" for anyone NOT testing the prompt itself of course).

Even with this minimal example, it took me a while to get it working due to the fact that providing invalid input (like missing an Enter), causes it to just get stuck waiting for input behind the scenes. I suspect that testing complex interactions with this might not be easy. This problem exists in any other testing method that exists until now, so its not worse, just also not better.

In regards to the new TTY::Testing gem - I am sorry, I really don't get it. Things there seem to be way too complicated, and confusing to me.
The examples are creating an app wrapper, and a prompt. In reality, we already have a CLI app, and the prompt is called from within it. We do not want to stub or mock the entire app. Just its input and output streams.
I am not sure if its just that the examples are confusing, or the entire philosophy of the testing gem is confusing, so forgive me if I missed the point.

@DannyBen humm, I didn't think it might look confusing, thanks for a valuable feedback, I might need to recraft the examples.
So, basically your app is what to be wrapped by the wrapper, your app should accept stdin and stdout as parameters (it is a proper practice anyway) and then it look like this:
let(:app) do
  TTY::Testing.app_wrapper do |stdin, stdout|
     MyCLIApp.new(stdin: stdin, stdout: stdout).run!
  end
end

it 'asks questions' do
  expect(app.output).to eq "What is your name?"
  app.input.puts "DannyBen"
  expect(app.output).to end_with "Hello, DannyBen!"
end

while the prompt is indeed created inside your MyCLIAapp class

Thanks. The above example is clearer, but...

your app should accept stdin and stdout

This requires changing of the application, just for the sake of testing. Not ideal.
Also, I never saw this kind of implementation in any other rspec plugin / helper.
Usually, plugins (such as this one, which is a plugin for TTY testing) does one of these:

Adds matchers - like redirect_to in

expect(subject).to redirect_to(path)

Adds convenience methods - like get in

expect { get '/path' }.to be_successful

Adds stubbing conventions, which is what I would expect more or less from a TTY testing gem.

expect(subject).to receive(prompt).and_return(mock_prompt)
# or
allow(TTY::Testing.input).to receive(:input).with("Hello", :down, :enter)

Adds "test drivers" - like Rails is doing for things like mail deliveries, or browsing the app - this is also a good option for TTY testing in my view - think setting the TTY Prompt "driver" to "test mode" before running tests.

In my view, there is either one or two things that just need stubbing. Everything else is noise. I want to:

Either stub MyApp#prompt and have it return a test prompt, on which I can inject stdin and examine stdout/stderr (which is what the tty/prompt/test approach provides in a sense).
Or, stub the input and output of TTY itself, or in other words, stub the "driver" - as is common in other testing related test plugins (in Rails). Example:

# in spec_helper.rb
TTY::Prompt.driver = TTY::Prompt::Test

# in any test
TTY::Prompt.input = ["Hello", :down, :enter]
MySubject.run_however_i_see_fit
expect(TTY::Prompt.output).to eq "Hello Jax"

For reference, this is similar to expect(ActionMailer::Base.deliveries).not_to be_empty in Rails, when testing mailers.
I guess the three of us here have three different use cases / needs / philosophies... :)

Hi Danny,
Before I reply, let me show something I did in a fork to your gist. No offense or patronising, just a way to demonstrate things:
At first, the app is interacting with the user with input and output, but for tests at jus returns plain value as a string in the method #run, that's a bit odd, console apps are meant to be tested via user input and output. I tool a liberty to update the code so that it gets tested via its inputs and ouputs, please take a look when you have time:
https://gist.github.com/DanielVartanov/19d5cfe8f4cb9215af40c14229ed48ae
I also added an example of how it would be tested via tty-test(ing).

Now, to your points. A huge, fundamental and a principal difference between TTY gems and Rails is that the former is a tool and the latter is a framework. With frameworks you write code where you were told to  -- normally in special methods of descendants of special classes (controllers, models, mailers etc).

This requires changing of the application, just for the sake of testing. Not ideal.

The concept is called Dependency Injection and under most circumstances it is the best  way to organise any app for the sake of testability, maintainability, loose coupling etc. In short, it is a very good and common practice of organising things.
Not surprisingly Rails does exactly that when you want it to expect(subject).to redirect_to(path), in tests it feeds controllers with mock HTTP requests while in development/production environment it feeds it with real HTTP requests received from the network socket. Exactly the same principle is applied here in TTY except for it is visible to the user since, again, TTY gems are tools, not frameworks.

The concept is called Dependency Injection

I know what dependency injection is, thank you.
In rspec, it is done by stubbing methods to return a different object, or by mocking - not by requiring the app under test to have something (stdout/stdin) that it doesnt normally need.
I am sorry, but if this is the direction this gem takes, I feel like testing in any other method - like hijacking stdin/stdout or like using the newly available TTY::Prompt::Test might be better.
I will try to create some better rspec example - and maybe even a "helper" - to demonstrate my point.

Hey Danny,
Sorry, not tried to be patronising or something, I genuinely thought you just didn't know about DI.

In rspec, it is done by stubbing methods to return a different object, or by mocking

RSpec does not really define that, a necessity to expose every substituted entity as a public method and and prohibiting the tested class to use the entity in any other way by via that public method is not what RSpec does. It is definitely a way to test with RSpec, but just one of the many.

I will try to create some better rspec example - and maybe even a "helper" - to demonstrate my point.

Great! :+1: That would be most helpful, thanks in advance.

... and I really meant "thank you for sharing knowledge" when I said thank you, but now when reading it I can see why it sounded like I'm annoyed... :) - I do appreciate knowledge sharing.
In any case - since both the testing gem and the exposure of TTY::Prompt::Test are not yet providing me with the "zen" of testing I expect (https://github.com/piotrmurach/tty-prompt/issues/148), I believe I have something that will probably work well for many TTY users testing in rspec, and it is much simpler than anything we all seen so far.  Hint: It will bypass input streams.
I figured this is analogous to the difference between being clever and being smart - instead of being clever and trying to fix the problem, I want to try and be smart and avoid it altogether.
I will post a proof of concept after I try it in (my-) real world scenarios.

Thank you both for this discussion, it is interesting to see how you wish to approach testing CLIs. This is definitely needed if we are to have an approach that works and is intuitive to use. Minimal and intuitive APIs are something I really care about.
@DannyBen Thank you for giving TTY::Prompt::Test a try and providing feedback. I totally agree with your points and in particular the weak points of this approach. I wanted to take a 'first' step and provide an 'intermediate way' that will provide some form of direction for testing prompt interactions. Possibly a help and improvement over global mocking of standard inputs.
It may be preferable and more flexible to replace the TTY::Prompt::Test with TTY::Prompt and demonstrate how to use :input and :output parameters. At least then nothing will seem like magic and the user will have full knowledge and full control of what is going on. I also want to keep this testing framework agnostic, so I didn't want to use subject and before helpers. We could possibly have rspec and minitest examples.
As for my 'philosophy' of testing. Since Rails has been used as comparison, I will try to draw some parallels. Please bear in mind that my perspective is skewed towards the tty and the teletype executable.
Having said that, for me these are related:
request <=> stdin
response <=> stdout

In tty and teletype the equivalent of controller is a concept of command that when executed takes in some standard input and produces 'potentially' some standard output. There may be some side effects like creation of files etc. But to all intents and purposes, I want to provide a way and a guide for unit testing commands. These commands may use few tty-* components which as Daniel mentioned will be 'wired' using standard in and out. However, at this stage, we don't care whether tty-prompt is used or some other components from other gems.
In my experience, providing mocked input and output tends to provide a robust solution for doing uni testing of CLIs. I often use IO-like object as I'm not interested in interactions but the content that the user inputs and observers afterwards. Stubbing interactions using doubles would focus on implementation details and not the business logic, which to me is the input and output cycle. More concretely:
# => initialize all dependencies including `tty-*` components
cmd = SomeCommand.new
# a more functional approach to mocking interactions similar to the Phoenix framework
# where each action receives all the connection information
cmd.execute(stdin: ..., stdout: ...)

We could also approach it from the perspective of stubbing all the interactions within the command. So, if a command uses, let's says 5 tty components, we would stub all of them and provide canned responses. However, I worry this may lead to brittle tests knowing too much about the command's internals and as such may 'miss' interactions between components instead of 'truly' testing what happens. So in this respect, I'm more 'classicist' rather than 'mockist' because I don't like tying tests to the implementation and prefer to check actual state changes. In my view, the way to achieve this is by injecting standard input and output to the test subject whatever it may be. I'm not dead set on this but this has been the approach that served me well in all tty components and the open-source CLIs, coinpare being one of them. Please see coins_spec. It may well be that we will have some 'hybrid' approach that uses stubs and mocked input and output. I don't know really, but I want to explore the best approach.
I'd like to have more discussion on this as we don't have a solution that matches expectations. Hence I must admit I always 'put off' thinking and creating a library to handle testing tty components. My current goal is to finish updating all the tty components(nearly there) so that I can remove the bundler < 2.0 requirement in the tty gem itself. This is to say that we don't need to settle on any approach yet and I'm keen on exploring how tty-test may/should work so I could use it and suggest for testing teletype generated CLIs.

Thanks for the thoughtful reply @piotrmurach - one comment that I have, based on everything I know and have seen so far in the "tty testing" discussion: I am starting to get the feeling that there is no need for a specific testing gem. I feel that the TTY components that are responsible for providing input and output, should be slightly altered - similar to how you made the Test prompt public - in order to allow for better stubbing and tesatbility.
In my mind, it is similar to how each of us improve our own code when we face hard-to-test areas.
I believe that there could be still a place for a TTY::Test gem, or a require 'tty/rspec helper - but that the core solution lies within the TTY gem suite itself, and these additional gems would just be some syntactic sugar.

@piotrmurach @DanielVartanov - as promised, I am "opening the case wide open" with another approach.
I have created this minimal sample repo - with a readme and everything.
In a nutshell: We never reach a point where we need input/output streams.
Teaser:
it "works" do
  ask "Matz"    # expect(subject.prompt).to receive(:ask).and_return "Matz"
  select "Ruby" # expect(subject.prompt).to receive(:select).and_return "Ruby"

  expect(subject.run).to eq "Matz likes Ruby"
end

@DannyBen If I understand your idea well, your preference would be to add a set of custom assertions that would make it easier to provide 'canned responses'.  Curious to know how would your approach testing in more complex scenarios that Daniel is trying to address? In the latest 'unreleased' as of yet version, I added :quiet option that allows you to skip the standard output altogether. You can use this option to build a 'dynamic' menu with nested options (see menu example). How would you test-drive command that depends on the dynamic input which is not provided at the subject creation or any method call?

If I understand your idea well, your preference would be to add a set of custom assertions

Well, not sure. The one thing I do know, is that I want to work with rspec, not against it.
So - if someone wants to mock the entire "prompt" object, let them. If they want to stub input and output, let them.
However, if we can come up with some helper methods - analogous to get '/path' in rails, that will universally work for everybody, then why not.

how would your approach testing in more complex scenarios

Personally, I never had to deal with CLI apps that have long and complex interactions. But, borrowing from other areas where I had to deal with complex code, and then test it - there is only one solution. Make it shorter and simpler by separating it to different methods.
Regardless of which will be the TTY testing tool - I think that it will not be a good idea to test long interactions as a whole. They must be separated, and then the tests will be easier.
Nevertheless, I understand that Daniel's experience and requirements may differ, and perhaps his use case requires a more elaborate testing framework.

How would you test-drive command that depends on the dynamic input

Not sure I am following. At the end of the day, any app-under-test ends up calling one of TTY::Prompt's methods - so it can be stubbed just like anything.
Does the example menu you provided above demonstrate your dynamic menu? If so, I can take it and "spec it". If not - can you provide some example?

Well - I see the difficulty testing this dynamic example without actually having control over stdin - and BTW, I love this example and its quiet option!
Although I think it may still be possible to test such examples without stdin control, I am not sure it can be done intuitively and naturally enough.
So - perhaps go back one gear to the idea of having a single method somewhere in TTY that can be stubbed so that it will get an array representing the input.
Behind the scenes, I expect it to read like this:
def tty_input(*keys)
  expect(TTY::InputHandler).to receive(:read).and_return(keys)
end

This should be able to handle special named keys - :enter, :down etc.
The test is expected to look something like this:
describe CoolMenu do
  it "works" do
    # Increase coolness to 3 then exit
    tty_input 2, 1, 1, 1, 3
    subject.run
    expect(subject.cool).to eq 3
  end
end

This of course, will cause the menu to print to the screen, interleaving its output with the spec output - which is undesirable. This is why I really think that TTY needs to have a "test driver" which will swallow all the stdout that prompt itself causes (but NOT the other stdout prints).
# spec_helper.rb
TTY::Prompt.driver = :test

For completeness, I converted the dynamic menu example to a more testable class, if anyone wants to try cracking this scenario.

Show the CoolMenu class
require "tty-prompt"

class CoolMenu
  attr_writer :cool

  def prompt
    @prompt ||= TTY::Prompt.new(quiet: true)
  end

  def cool
    @cool ||= 0
  end

  def run
    main_menu
  end

protected

  def main_menu(from = nil)
    name = prompt.select("Cool Menu") do |menu|
      menu.enum "."

      menu.choice "Coolness Status", :status_menu
      menu.choice "Manage Coolness", :manage_menu
      menu.choice "Exit", "exit"
    end

    next_menu(name, :main_menu)
  end

  def next_menu(name, from)
    if name.is_a?(Symbol)
      send(name, from)
    else
      eval(name)
    end
  end

  def status_menu(from)
    name = prompt.select("Coolness is at #{cool}" ) do |menu|
      menu.enum "."

      menu.choice "Back", from
      menu.choice "Exit", "exit"
    end

    next_menu(name, :status_menu)
  end

  def manage_menu(from)
    name = prompt.select("Coolness is at #{cool}.\nManage" ) do |menu|
      menu.enum "."

      menu.choice "Add coolness", :add_cool
      menu.choice "Back", :main_menu
      menu.choice "Exit", "exit"
    end

    next_menu(name, :manage_menu)
  end

  def add_cool(from)
    self.cool += 1
    next_menu(from, from)
  end
end

# Run with:
# CoolMenu.new.run

Alright - posting this as a new message, since the previous one was too long already.
I have updated the example repo to show more or less how stdin control might look like.
This spec: (on github)
require 'spec_helper'

describe App, :tty do
  context "without arguments" do
    it "asks for a name and favorite language" do
      tty_input "Matz", :enter, :down, :enter
      expect(subject.run).to eq "Matz likes Ruby"
    end
  end

  context "with --confirm" do
    it "also asks if I am sure" do
      tty_input "Matz", :enter, :down, :enter, 'y'
      expect(subject.run %w[--confirm]).to eq "Matz likes Ruby"
    end
  end

  context "with --print" do
    it "prints the output" do
      tty_input "Matz", :enter, :down, :enter
      expect { subject.run %w[--print] }
        .to output("==> Matz likes Ruby\n").to_stdout
    end
  end
end

Made possible by this spec_helper

A little side note: it's been a pleasure reading this thoughtful discussion.
As someone who's been deep in CLI testing just recently I've got only two things to amend:

I am starting to get the feeling that there is no need for a specific testing gem

I surprisingly agree :-D
In 90% of cases CLI apps are simple one-step feed-and-read experiences (coinpare mentioned above is a typical example of it), such apps definitely need no additional structures, libraries or tools. The whole thing about tty-test(ing) gem is to deal with multi-stage interactive apps with many many branches of possibilities, it's a heavy duty tool useful for 10% of apps at best.

I'd like to bring your guys attention to a problem of subprocesses which are often used in CLI apps, e.g. when one CLI app runs another CLI app (probably a system one) which's output is used in the main app output. Testing such things would require a proper file descriptor of an IO object passed to TTY::Prompt. Though, it is not an often case either.

I am glad we are all converging to a similar thought pattern.

I'd like to bring your guys attention to a problem of subprocesses which are often used in CLI apps, e.g. when one CLI app runs another CLI app (probably a system one)

In this regard - when the CLI calls something else - I follow what I believe is considered a best practice in testing in general, not only CLI:
There is no need to test the output of that subprocess, and there is not even a need to execute it. We are assuming it works, and that it has its own tests to prove it. All we need to do, is ensure that our CLI tool calls it. So: expect(subject).to receive(:system).with(["--some", "args"])
In other words - testing that subprocess is outside the scope of your own test suite, in my view.

State of the (TTY Testing) Union
I feel we are all more or less close to reaching conclusions, so trying to summarize our situation here in order to materialize this discussion to its desired outcomes.
Tester Types
There will probably be three testing scenarios, for different users and use cases (going from simple to complex):

Testers who design their tests and app to avoid the need to control STDIN.
Testers who want or need STDIN injection control.
Testers who need more than STDIN/STDOUT control.

Implementation
Case 1: Stub TTY::Prompt methods
For case 1, we can easily stub TTY::Prompt methods, perhaps with the help of helpers that will be user-defined in the spec_helper (i.e. no need to add anything to TTY for this case).
Example: expect(subject.prompt).to receive(:select).and_return("Pizza")
Case 2: TTY::Prompt::Test
For case 2, we can stub STDIN. So far, the best approach for this was to have TTY::Prompt::Test available. If this solution can be polished (for example to avoid the StringIO hacks, as I believe was already done) - then this solution is also ready.
I am not mentioning STDOUT specifically, since a) it can be captured natively in rspec with expect { subject.run }.to output("something").to_stdout and b) the TTY::Prompt::Test already provides .output.string, so we can test the prompt output itself if desired.
Case 3: Complex Apps
For case 3, there is probably a need for the separate testing gem that allows for a more involved interaction with the app. Personally, I do not think this gem should be the main recommended testing approach for TTY users, since it will usually be an overkill.
Next Steps
My proposal for next steps:

[ ] Complete polishing of TTY::Prompt::Test or an alternative approach to allow stdin stubbing.
[ ] Create a Markdown document describing and demonstrating the first two test approaches. This can either be put into the README or into a Wiki page, dealers choice. (I can help with this document).
[ ] Complete the TTY::Test(ing) gem, and link to it from the above document.

I'd like to elaborate on subprocesses thingie at least to refer to this convo in future. Several points on the matter, just for the record  (i.e. not trying to argue further, everyone picks their own way of testing):
I.  Besides Kernel#system there are  many other ways to run a subprocess: Process.spawn, fork/exec, Open3, tty-command just to name a few. Tests shall test behaviour, not the implementation.
II. Implementation can switch between running a subprocess and doing the work itself, e.g. between running curl and using libcurl/curb (not the greatest example as curl's output isn't very much valuable, but I hope it illustrates the idea).
III. Stubbing and mocking are used to test a unit in isolation from other units, such tests are a great compliment to functional/acceptance tests, but they are never sufficient (they can pass green when the app does not work and fail when the app actually still works).

Hi Danny,
Great, I love your summary. Perhaps the very fact that we are having this conversation alone states that there are just different preferences in approaches to testing and tty-toolkit should not impose one upon the user.
I actually have an idea on polishing TTY::Prompt::Test, will try to write it up today in the evening, if I imagine everything correctly it will work exactly as you describe it in your examples.

After playing a bit with your ideas of TTY::Prompt.driver = ... and prompt.input = %w[1 2 3] I managed to pull it together, but since this approach is not my cup of tea I can't really assess whether it's any good
TTY::Prompt.test_mode! # in spec_helper.rb

RSpec.describe "console app" do
  let(:prompt) { TTY::Prompt.new } # No TTY::Prompt::Test

  subject do
    name = prompt.ask("Name?")
    if prompt.yes?("Greetings?")
      prompt.say "Hello, #{name}!"
    end
  end

  specify do
    prompt.input = "Danny\nY\n"
    subject
    expect(prompt.output).to end_with("Hello, Danny!\n")
  end
end

Implementation and this ^^^ test being runnable as a single file is in this gist
(sorry implementation is quick-n-dirty, it's just a PoC that such DSL can work)

Thanks Daniel. You gotta love Ruby and the neat tricks it lets you do... :)
I added your example to the demo repo and invited you both.
I used the idea of a "driver" mostly for inspiration, since this is how Rails is doing it and in some cases it makes sense. I believe that in our case - now that TTY::Prompt::Test is shaping up to be (almost?) completely functional and available to end users - we can avoid that use of a driver.
The below code (also available as test2 in the repo) can be achieved with this:
require 'tty-prompt'
require 'tty/prompt/test'

class TTY::Prompt::Test
  def input=(keys)
    keys.each { |key| input << key }
    input.rewind
  end
end

RSpec.describe "console app" do
  let(:prompt) { TTY::Prompt::Test.new }

  subject do
    name = prompt.ask("Name?")
    if prompt.yes?("Greetings?")
      prompt.say "Hello, #{name}!"
    end
  end

  specify do
    prompt.input = ["Danny", "\n", "Y", "\n"]
    subject
    expect(prompt.output.string).to end_with("Hello, Danny!\n")
  end
end

Advantages:

We do not contaminate TTY::Prompt itself
We utilize an already existing TTY::Prompt::Test which is specifically designed for testing.
The line   let(:prompt) { TTY::Prompt::Test.new } expresses intent more clearly than   let(:prompt) { TTY::Prompt.new } - in fact, this is our "setting of a test driver" right here.

In my mind - if we can:

Add this input= method to TTY::Prompt::Test, and
Make it accept symbols for special keys - :enter :down :pgup, then this portion of our experiment can be over, and we can have a nice approach for testing (at least for the 90% apps we discussed).

And just to clarify some things about the "test driver" approach, so that we are all on the same page.
First, I am also not a big fan of this approach. I dislike it in Rails, for two reasons:

It is a little bit "magic" - in Rails you even not necessarily know that it is configured, since it is hidden inside some config/environment file that came like this as a default.
It causes your specs to look less like DSL, since it contains a lot of code calling namespaced modules (e.g. expect(ActionMailer::Base.deliveries).not_to be_empty). Since Rspec invested TONS of resources in making your specs read naturally, this kind of syntax breaks the natural reading pace.

That said, I believe if we have a proper TTY::Prompt::Test - it is a better approach than having a test driver, since it is the test driver in a way, and since TTY::Prompt::Test already exists and 90% operational.
For completeness, I want to describe - in my view - what is the core concept behind a test driver.
To me - when using a test driver - it means that you configure that "tool" to avoid performing its standard operation, and instead, write its artifacts in a single, known location.
In ActionMailer, this location is ActionMailer::Base.deliveries:
something_that_sends_mail.run
expect(ActionMailer::Base.deliveries).not_to be_empty

In ActiveJob the situation is both worse (since it is more complicated behind the scenes) and better (since it is more DSLish to the user) and rspec needs to provide a specific matcher for it:
expect { SomeJob.perform_later }.to have_enqueued_job

Yuck for both.
So if we were to use a driver approach (not saying we should), it should have looked like this:
# spec_helper

# Enable test mode in a way that can be disabled mid-flight if needed
TTY::Prompt.test_mode = true

# spec

# Inject input, to a predetermined static location
it "works" do
  TTY::Test.input = ["Piotr", :enter, "y", :enter, :down, :enter]
  subject.run
  expect(TTY::Test.output).to match "Hello Piotr"
end

# Alternatively - if we borrow from ActiveJob's test approach:
it "works" do
  TTY::Test.input = ["Piotr", :enter, "y", :enter, :down, :enter]
  expect { subject.run }.to have_tty_output("Hello Piotr")
end

I know these are horrible - but this is the "driver approach" in my mind, which we don't need.

Thank you for all your comments! I read all them and hopefully haven't missed anything.
To me, the discussion touches on two main subjects, one of how to test tty-prompt interactions within some context and one a bit wider of testing CLIs(e.i. running external processes that may or may not deal with user input). This is great but as more immediate step I'd probably choose to focus on tty-prompt as suggested at the start of this thread and split the discussion and continue a wider approach to testing of CLIS and tty gems in the tty project itself.
Case 1: Stub TTY::Prompt methods
This is a viable solution, though I'm not convinced about recommending this in docs as an approach for testing classes that use TTY::Prompt to perform their business logic. One of the 'weaknesses' that I see is the fact that the 'canned' responses are predetermined. The choices and options to any prompt methods maybe dynamic and a result of some other computation. Secondly, stubbing 'separates' implementation. It's not about testing tty-prompt itself but ensuring that, for example, input conversion happens and, for instance, returns a boolean answer. Lastly, stubbing adds a lot of 'noise' in a test especially when the user deals with few prompts in a given context of a class. I have therefore reservations about making this as a recommended approach.
Case 2: TTY::Prompt::Test
I use this approach in other tty gems like tty-file. The benefits of using TTY::Prompt::Test are many. You don't need to stub any prompt methods. If a subject under test calls many prompts, the test doesn't need to 'know' about them. This means that the developer is free to use whatever the prompt as long as the end result works. This separates the test from the implementation details and focuses it more on business logic. What binds the different prompts is the fact that standard input is mocked. You control test execution with different inputs. For example, using select prompt may actually create many execution paths through a codebase which depend on different inputs.
To be more precise, in tty-file is use expand prompt which allows a user to choose action when there is a conflict with a file. Each option in the menu triggers different logic and needs to be tested separately. For example, the test for file collision with overwriting logic looks like this:
it "displays collision menu and overwrites" do
  test_prompt = TTY::Prompt::Test.new
  test_prompt.input << "\n"
  test_prompt.input.rewind
  allow(TTY::Prompt).to receive(:new).and_return(test_prompt)

After the setup that tells the prompt to overwrite file, I run the action and observe output:
  file = path_factory.call("README.md")
  TTY::File.create_file(file, "# Title", verbose: false)

  expect {
    TTY::File.create_file(file, "# Header", verbose: true)
  }.to output(/\e\[31mcollision\e\[0m  #{file}/).to_stdout_from_any_process
  expect(test_prompt.output.string).to match(/Overwrite #{file}\?/)

And do some final sanity check:
 expect(File.read(file)).to eq("# Header")

What I like about this test is that I can swap expand prompt for another one and the test wouldn't need to change at all provided that the input triggers the same behaviour.  There are some 'rough' edges in this approach. One, the TTY::Prompt::Test could expose much nicer methods. I'd like to suggest couple of methods:
test_prompt.type_in "input" # an actual input
test_prompt.press :enter  # a generic way to named concepts like :enter, :space, :up

Then common scenarios may be replaced with:
  test_prompt.input << "\n"
  test_prompt.input.rewind
  # instead do this
  test_prompt.press_enter

To me, this is where Daniel's approach using Fibers may actually yield(sorry;) great benefits. If anything, it could be used to power more 'natural' flow to inputs and reverse the logic of the previous test:
  test_prompt = TTY::Prompt::Test.new
  allow(TTY::Prompt).to receive(:new).and_return(test_prompt)

  file = path_factory.call("README.md")
  TTY::File.create_file(file, "# Title", verbose: false)

  expect {
    TTY::File.create_file(file, "# Header", verbose: true)
   test_prompt.press_enter # much nicer!
 } ....

In general, this approach avoids any mysterious configuration options and permits granular usage. You can have a file with many tests and only few of them focus on the prompt interactions.
Case 3: Complex Apps
I think this is more about the level of abstraction and the type of tests you wish to write rather than being 'complex'. Whether you call them unit or integration tests, the aim is to get the most value out of a single test. To me, the more outside-in approach for testing CLIs is more preferable especially in the context of auto-generated tests. In other words, I'm talking about writing capybara like tests to exercise a given scenario. This can be a single command or a few commands working together. For example, in tty gem the new command has a lot of logic and single public execute method and I think it would be tedious and rather difficult to write unit tests that would stub and mock many calls inside. There are way too many inputs/flags and external commands to consider stubbing them all. These commands like Daniel have mentioned, need to be run to affect a real change in the system. In the case of the new command, create an entire CLI structure.
Example of how I think this could work:
cmd = Command.new # subject under test with single `execute` method
cmd_session = TTY::Test::Session.new(cmd) # injects mocked inputs and output and hiding all the 'wiring' details

cmd_session.execute(...) # forwards all calls to the underlying cmd

cmd_session.type_in "input" # input sent to whatever listens on the mocked standard in, could be `gets` method
cmd_session.press_enter # send "\n" to whatever still listens

# check some intermediate results

cmd_session.press :up, :up, :enter

# check some intermediate results

I'd love to see some PRs for the TTY::Prompt::Test that tackle what we have discussed so far and make the API easier to use and more intuitive.
ps @DanielVartanov I released tty-prompt and I will soon release other gems.

Taking a seat, since I need to test a Thor app that uses TTY::Prompt under the hood to display a menu and I don't know how to test it.


Updates needed to move dishwasher error events from op state to a new alarm cluster

Here is the spec PR that moves the error events to a new alarm cluster:  https://github.com/CHIP-Specifications/connectedhomeip-spec/pull/7070
See the referenced PR for details.

@rbultman Is there anything left to do here?

This should be finished.

Completed.


Fire 'zoomlevelschange' when calling setMinZoom & setMaxZoom.

Allows Control.Zoom to enable/disable controls in response to min and max zoom level changes.

Btw, nice catch! :+1:

wow this still hasn't been merged yet!? And for such an obvious fix
@cherniavskii using p as a prefix for previous/old is obvious to me but feel free to update/redo the pr.

@mynameisstephen thanks!
I also added one more check before firing zoomlevelschange, because the only situation it should be fired is when map has the same zoom as new minZoom/maxZoom.
Otherwise, zoomlevelschange can be fired twice, and listener in Zoom Control is quite expensive, since it removes and adds classNames.

I updated it again to make sure zoom control will be enabled after changing minZoom/maxZoom, so I'm just avoiding firing both setZoom and this.fire('zoomlevelschange') at the same time.

Thanks @mynameisstephen and @cherniavskii for keeping in there and making sure this was finally merged! 👍


New Data Science Experience accounts do not include object storage or spark services

The  Create the Project step within Create the notebook needs to add steps for adding a storage instance and spark instance as the current flow when creating a DSX account no longer creates the DSX-ObjectStorage and DSX-Spark services in the user's cloud account.

this is all changed with the new watson studio service, see latest readme


python style formatting not working unless format with python

Environment data

VS Code version: 1.42.1
Extension version (available under the Extensions sidebar):
OS and version: ubuntu 18.04
Python version: anaconda, python 3.7
Type of virtual environment used : conda
Relevant/affected Python packages and their versions: autopep8

Expected behaviour
format selection and format document
Actual behaviour
** not working**
Steps to reproduce:

select python file, right click, format document or format selection
select python file, right click, format document with python or format selection with python

Maybe the IDE don't recognize the python file extension to format python file

Hi @yaochx,
Unfortunately I'm not able to reproduce this behavior. Please provide the logs, and the settings you're using.
Are you using a different language?

@karrtikr @luabud
I am sorry for the trouble aroused by my negligence.
I do not set the default python formater. after setting, both works.
thanks


Update tutorial-configure-multiple-monitors.md

Moved snapshot before the table

#sign-off

@RoseHJM
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Hi @dyadav7, thank you for your suggestion, your contribution is very much appreciated.
Is this just a change in the article structure, moving the screenshot above the table? I don't see any other changes.

Hi @RoseHJM
Yes, this is the only change. Snapshot before the text is much slightly better as user (when doing things quickly) may try to match his screen with the screen shot shown on web page.

@dyadav7 Thanks for sharing your perspective. I'm going to leave the flow as it is for two reasons:

I want the user to slow down, compare the available options available, and choose the option most appropriate to them.
Keeping the screenshot after the table will ensure the article is consistent with other articles in this docset, and across the Microsoft Learn platform.

#please-close


add rle v2 encoding for segment v2

reuse the rle encoding in alpha rowset
modify segment_test because of change of encoding

issue: #2886

If you just move the old code from the AlphaRowset, the question of AlphaRowset still exists.
Like

I/O exemplification of read StreamHead.

Two much virtual functions of read caused by streaming read of ORC.

Not import ORCFile the latest enhancement in recent years.

This are all questions we start to build SegmentV2.


Add -nolauncher steam's launch option by default

Add -nolauncher steam's launch option by default so some games like Shadow of Tomb Raider & Galaxy of the Guardians, etc. will launch directly to game instead of game's launcher

We are not adding game specific launch arguments to default startup procedure, that's very bad idea since it might cause issues with other games where that argument works differently.

But user have an option "Show Steam's game startup selection menu in Desktop/Fullscreen mode" which is line above                 ProcessStarter.StartProcess(steamExe, $"-silent "steam://launch/{Game.GameId}/Dialog"") without -nolauncher default.
The point of Playnite Fullscreen Mode is for controller player Press & Play but it's not the case with game's launcher, we need an option to skip game's launcher by default

No, that /Dialog part what Steam handles, not the game.

No, that /Dialog part is what Steam handles, not the game.

I meant, user have that part to launch normally with game's launcher if they wish to in case of they don't want -nolauncher launch option by default.

I'm sorry but it simply makes no sense to have arguments specific only to handful of games to be part of default startup procedure. You can easily configure additional arguments in Steam itself to be applied to those games on startup.

If I have to come to Steam to manually edit every game there and what's the point of using Playnite? Sorry as it's a game handheld user, it's bad UX.
Yea, as in your view it's bad practice so please have an option to add argument for default Play action.  want to add custom Play action for every gam


3d support

This adds support for the following two layers:

3DConv
3DMaxPool

Tests and README updates are included.

Hello, @s-westphal!
Nice job, I merge the pull request.

Thank you very much!


cached first item size

Problem
We have used snappy list view in hotels list, but while implementing scroll to top functionality I faced an issue like sometimes there is additional top padding after scrolling to first index, which was given by snappy list view widget.
Cause of problem

Each widget's size is stored in internalSizes to calculate padding for alignment.
The getSize function checks sizes in internalSizes.
Some sizes are removed from internalSizes based on scroll position for memory optimization.
When scrolling to the top, if the 0 index widget size is missing inside internalSizes, it miscalculates, adding extra top padding from calculatePadding.

/// Retrieves the size of an item by index, while checking for validity of size
  double getSize(int index) {
    final internalItemIndex =
        internalSizes.indexWhere((element) => element.index == index);
    //initial element alignment correction
    if (internalItemIndex != -1) {
      final itemSize = internalSizes.elementAt(internalItemIndex).size;
      assert(itemSize <= maxScrollDirectionSize,
          "The size of each item is limited to the maximum size of the scroll area.");
      return itemSize;
    } else {
      return 0;
    }
  }

  /// Returns the page padding for the first and last page of the list to simulate
/// a correct (middle) start/end of the pages
EdgeInsets getPagePadding() {
  final firstIndex = widget.reverse ? widget.itemCount - 1 : 0;
  final lastIndex = widget.reverse ? 0 : widget.itemCount - 1;
  final firstSnapItem = assignSnapAlignmentItem(firstIndex);
  final lastSnapItem = assignSnapAlignmentItem(lastIndex);
  final firstAlign = widget.snapAlignment.apply(firstSnapItem);
  final lastAlign = widget.snapAlignment.apply(lastSnapItem);
  final firstItemAlign = widget.snapOnItemAlignment.apply(firstSnapItem);
  final lastItemAlign = widget.snapOnItemAlignment.apply(lastSnapItem);
  final firstPadding = maxScrollDirectionSize * firstAlign -
      firstItemAlign * getSize(firstIndex);
  final lastPadding = maxScrollDirectionSize * (1 - lastAlign) -
      (1 - lastItemAlign) * getSize(lastIndex);
  assert(
      firstPadding >= 0 && lastPadding >= 0,
      "Snap- and SnapOnItemAlignment caused border items to overflow. "
      "Please make sure that snap never pushed items out of the viewport.");
  switch (widget.scrollDirection) {
    case Axis.horizontal:
      return EdgeInsets.only(left: firstPadding, right: lastPadding);
    case Axis.vertical:
      return EdgeInsets.only(top: firstPadding, bottom: lastPadding);
  }
}

Inside sync value we are removing sizes for memory optimisation.
        final listItems = widget.itemPositionsListener.itemPositions.value;
      internalSizes.removeWhere((element) {
        final keepList = listItems.map((e) => e.index).toList() +
            [listItems.first.index - 1, listItems.first.index + 1];
        return keepList.contains(element.index) == false;
      }); //clears sizes (with buffer for border) to prevent high ram usage

Proposed solution
There could be couple of solution either we cache first item size (which I did as of now) or we can avoid removing sizes from internalSizes.

We can store the first item’s size so that, if it’s missing from internalSizes due to memory optimization, we can use the stored size for the 0 index widget.

Screen recording

I tried scrolling 100+ hotels and its working fine.
Note: Ignore more icon I have used for testing.

snappy_scrollview_bug_fix.webm

Lets add the solve for last item also. As when we go to last item it goes to center. Last item should be bottom aligned not center aligned.


Better ladder

Current dependencies on/for this PR:

main

PR #67   👈

This stack of pull requests is managed by Graphite.

Merge activity

Jan 2, 12:25 AM: @kadhirvelm started a stack merge that includes this pull request via Graphite.


The CMake: Select a kit command does not work the first time you click on it.

Brief Issue Summary
Repro steps:

Create a new folder named 'Test' and open it with vscode.
Create a file named CMakeLists.txt with the following content:

cmake_minimum_required(VERSION 3.8)
project(Test)
add_executable(Test "main.cpp")

Create a file named main.cpp with the following content:

#include <iostream>

int main(int argc, char **argv)
{
    std::cout << "Hello World" << std::endl;
    return 0;
}

Run command: “CMake:configure”
Run command: “CMake:Select a kit”

Actual results：

CMake Tools Diagnostics
No response
Debug Log
No response
Additional Information
No response

@gcampbell-msft Thank you very much for your reply, I will keep an eye on this issue when creating future issues :provide CMake Log Diagnostics . I verified this issue on previous versions and since CMake Tools: v1.19.20 (pre-release) the issue hasn't come back so I will close this issue!


Create a Form component and an Array Component

You created the Address inside the Form, but it would be great to create  the one Address component that could be added inside the form.
And it could be even better to create one AddressArray component using the previous Address component.
this way would be very transparent to use one address, or to use the array when it's necessary

Yes, if you plan to reuse an address block throughout your application, building it as a nested component is a great idea!
Since the course was focused on forms, and because the array chapter was already a bit complex  ... I don't plan to make this change to the course.
Have you tackled it yet? If so, I'd be happy to post a link to your github or blog to share your solution with others.
Thanks!

Thanks for the answer.
I don't have a blog, or a github with an example of it, in fact, I didn't make it work, só i'm going to show you my code and how far I got so you can see if it's in the right direction and how close I am to do this.
First I created a base class for a form component I called it BindComponent because it's used for combos and lists, with a way to select an object and bind it to my model property.
`import {Component, ViewChild, ChangeDetectorRef, Input, Output, EventEmitter, OnInit} from '@angular/core';
import {ControlValueAccessor, NG_VALUE_ACCESSOR} from "@angular/forms";
import {Observable}  from 'rxjs/Observable';
const noop = () => { };
export class BindComponent implements OnInit, ControlValueAccessor {
protected values: IBindValue[] = [];
public Source: (obj?:any) => Observable<IBindValue[]>;
@Input() label: string;
@Input() Children: BindComponent[] = [];
@Input() readonly: boolean = false;
private _model: any; // stored value
@Input() AutoLoad: boolean = false;
@Input() idhtml: string;
@Input() property: string = "";
@Output() init: EventEmitter<BindComponent> = new EventEmitter<BindComponent>(false);
//@Output() ModelChange: EventEmitter<any> = new EventEmitter<any>(false); // an event emitter

//Placeholders for the callbacks
private _onTouchedCallback: () => void = noop;

private _onChangeCallback: (_: any) => void = noop;
protected initialized: boolean = false;
public ngOnInit() {

    this.init.emit(this);
    if (this.AutoLoad) {
        this.load();
    }

}

public load(obj?) {
    this.clearChildren();
    if (this.Source) {
        this.Source(obj).subscribe(x => this.values = x, e => { }, () => {
            this.setSelectItem();
            this.finallyLoad();
            this.initialized = true;
        });
    }
}

public get Model(): any {
    return this._model;
};

//set accessor including call the onchange callback
public set Model(v: any) {
    //if (v !== this._model) { //LUIZ: fiz essa alteracao para que campos nulos atualizem os filhos temos que ficar atento para possiveis efeitos colaterais
        this._model = v;
        this._onChangeCallback(v);
        this.onSetModel();
        for (let child of this.Children) {
            if (child) {
                child.load();

            }
        }
    //}
}

protected onSetModel() { }

protected finallyLoad() { }

protected setSelectItem() { }

protected clearChildren() {
    for (let child of this.Children) {
        if (child) {
            child.clear();
            child.clearChildren();
        }
    }
}
protected clear() {
    this.values = new Array<IBindValue>();
}

//Set touched on blur
onTouched() {
    this._onTouchedCallback();
}

//From ControlValueAccessor interface
writeValue(value: any) {
    this._model = value;
    this.onWriteValue();
}

protected onWriteValue() { }

//From ControlValueAccessor interface
registerOnChange(fn: any) {
    this._onChangeCallback = (x) => {
         if (this.initialized) {
             fn(x);
        }
    }
}

//From ControlValueAccessor interface
registerOnTouched(fn: any) {
    this._onTouchedCallback = () => {
         if (this.initialized) {
            fn();
        }
    }
}

public DefaultMap<T>(source: T[], value: (T: T) => string, text: (T: T) => string): IBindValue[] {
    return source.map<IBindValue>((e, i, a) => new BindValue(value(e), text(e), false, e));
}
public DefaultMapValue<T>(source: T[], value: (T: T) => string, text: (T: T) => string, object:(T:T)=> any ): IBindValue[] {
    return source.map<IBindValue>((e, i, a) => new BindValue(value(e), text(e), false, object(e)));
}

}
export interface IBindValue {
value: string;
text: string;
selected: boolean;
model: any;
}
export class BindValue implements IBindValue {
constructor(public value: string, public text: string, public selected: boolean, public model: any) {
}
}`
Then moving on my example I created a liscomponent
`import { Component, Provider, forwardRef, ExistingProvider } from '@angular/core';
import { Observable } from 'rxjs/Observable';
import { BindComponent, IBindValue } from './bind.component'
import { ControlValueAccessor, NG_VALUE_ACCESSOR } from "@angular/forms";
import { List } from '../models/list.model'
const CUSTOM_INPUT_CONTROL_VALUE_ACCESSOR: Provider = {
provide: NG_VALUE_ACCESSOR,
useExisting: forwardRef(() => ListComponent),
multi: true
};
@Component({
selector: 'tst-dual-list',
templateUrl: 'app/common/components/list.component.html',
providers: [CUSTOM_INPUT_CONTROL_VALUE_ACCESSOR]
})
export class ListComponent extends BindComponent {
protected finallyLoad() {
    var selected = this.getSelectedValue();
    if (selected) {
        this.selectItems(selected.map(x => x.value));
    }
}
protected setSelectItem() {
    var modelproperty: any[] = this.Model ? List.getArrayList(this.Model, v => v) : [];
    modelproperty = modelproperty.map(x => x[this.property] ? x[this.property] : -1);
    for (let item of this.values) {
        var itemproperty = item.model && item.model[this.property] ? item.model[this.property] : -2;
        item.selected = modelproperty.indexOf(itemproperty) >= 0;
    }
}

public getSelectedValue(): IBindValue[] {
    var retorno = [];
    for (let v of this.values) {
        if (v.selected) {
            retorno.push(v);
        }
    }
    return retorno;
}

selectItem(selectedValue: string) {
    for (let value of this.values) {
        if (value.value == selectedValue) {
            value.selected = !value.selected;
        }
    }
    this.emitSelected();
}

private selectItems(selectedvalue: string[]) {
    for (let value of this.values) {
        if (selectedvalue.indexOf(value.value) >= 0) {
            value.selected = true;
        }
        else {
            value.selected = false;
        }
    }
    this.emitSelected();
}

protected emitSelected() {
    var selected = this.getSelectedValue();
    var retorno = new List<any>();
    if (selected) {
        retorno.$values = selected.map(x => x.model);
    };
    this.onTouched();
    this.Model = retorno;
}

}with a vey simple html

            {{model.text}}

`
and it's usage is very simple
<tst-dual-list (init)="documentsInit($event)" [readonly]="readOnly" [(ngModel)]="model.Model.Documents" [property]="'Id'" [idhtml]="'documents'" [AutoLoad]="true" formControlName="Documents" [Children]="dependenciesdocumentsBindComponent"></tst-dual-list>
and this event as the init of the component
`    private documentsBindComponent: BindComponent;
private dependenciesdocumentsBindComponent: BindComponent[] = [];
public documentsInit(event: BindComponent) {
    this.documentsBindComponent = event;
    //push to dependencies
    this.documentsBindComponent.Source = () => this._bindService.documentsAutocomplete().map((values) => this.documentsBindComponent.DefaultMap(values, (e) => e.Id.toString(), (e) => e.Description));
}`

what do you think of it? I'm in the right path?
Thanks again, I'm a huge fan of you.

Thank you for the kind words!
I've spent the better part of May at conferences and/or out of the country. I have not (and most likely will not) really have time to go through all of this posted code in detail.
If you do want to set up a github account with something at least runable (even if it does not work), I will find time to go through it.

Thanks for the time spent on this topic
I created another post with the code, it works almost fine, but the form doesn't refresh the model state unless some change is made in the parent form
I zipped the file and uploaded it to the issue.