DynamoDB doesn't work with native AOT

I wrote a lambda function with .net 7 native AOT. Without a database access, it worked properly. I tried to code access to DynamoDB table. Here is my code:
internal class DbRepository : IDbRepository
{
    private static readonly IAmazonDynamoDB _dbClient = new AmazonDynamoDBClient();

    public DbRepository()
    {
        DbTable = Table.LoadTable(_dbClient, Utils.GetRequiredEnvVariable("DDB_TABLE"));
    }

    public Table DbTable { get; }

In the constructor for the statement Table.LoadTable I got following exception:
---> System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property.
---> System.NotSupportedException: 'System.Nullable`1[System.Byte]' is missing native code or metadata. This can happen for code that is not compatible with trimming or AOT. Inspect and fix trimming and AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
at System.Reflection.Runtime.General.TypeUnifier.WithVerifiedTypeHandle(RuntimeConstructedGenericTypeInfo, RuntimeTypeInfo[]) + 0x98
at Amazon.DynamoDBv2.Converter`1.<GetTargetTypes>d__0.MoveNext() + 0xdf
at Amazon.DynamoDBv2.ConverterCache.AddConverter(Converter converter, DynamoDBEntryConversion conversion) + 0x99
at Amazon.DynamoDBv2.DynamoDBEntryConversion.AddConverters(String suffix) + 0x1ca
at Amazon.DynamoDBv2.DynamoDBEntryConversion..ctor(ConversionSchema schema, Boolean isImmutable) + 0xa1
at Amazon.DynamoDBv2.DynamoDBEntryConversion..cctor() + 0x27
at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xb9

I'm using the latest version of AWSSDK.DynamoDBv2 (<IP_ADDRESS>). I added the AWSSDK.DynamoDBv2 in the rd-config file too:
<Directives xmlns="http://schemas.microsoft.com/netfx/2013/01/metadata">
  <Application>
    <Assembly Name="bootstrap" Dynamic="Required All"></Assembly>
	<Assembly Name="AWSSDK.SimpleSystemsManagement" Dynamic="Required All"></Assembly>
	<Assembly Name="AWSSDK.Core" Dynamic="Required All"></Assembly>
	<Assembly Name="AWSSDK.DynamoDBv2" Dynamic="Required All"></Assembly>
  </Application>
</Directives>

I'm using Windows 11 Pro 10.0.22621 with dotnet version 7.0.100. And here is my project file:
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <AWSProjectType>Lambda</AWSProjectType>
    <AssemblyName>bootstrap</AssemblyName>
    <!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <!-- Generate Native AOT image during publishing to improve cold start time. -->
    <PublishAot>true</PublishAot>
    <!-- StripSymbols tells the compiler to strip debugging symbols from the final executable if we're on Linux and put them into their own file.
    This will greatly reduce the final executable's size.-->
    <StripSymbols>true</StripSymbols>
  </PropertyGroup>
  <!--
  When publishing Lambda functions for ARM64 to the provided.al2 runtime a newer version of libicu needs to be included
  in the deployment bundle because .NET requires a newer version of libicu then is preinstalled with Amazon Linux 2.
  -->
  <ItemGroup Condition="'$(RuntimeIdentifier)' == 'linux-arm64'">
    <RuntimeHostConfigurationOption Include="System.Globalization.AppLocalIcu" Value="<IP_ADDRESS>" />
    <PackageReference Include="Microsoft.ICU.ICU4C.Runtime" Version="<IP_ADDRESS>" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.APIGatewayEvents" Version="2.6.0" />
    <PackageReference Include="Amazon.Lambda.RuntimeSupport" Version="1.8.2" />
    <PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.0" />
    <PackageReference Include="AWSSDK.DynamoDBv2" Version="<IP_ADDRESS>" />
    <PackageReference Include="AWSSDK.SimpleSystemsManagement" Version="<IP_ADDRESS>" />
    <PackageReference Include="LazyCache.AspNetCore" Version="2.4.0" />
  </ItemGroup>

  <ItemGroup>
    <!--The runtime directives file allows the compiler to know what types and assemblies to not trim out of the final binary, even if they don't appear to be used.-->
    <RdXmlFile Include="rd.xml" />
  </ItemGroup>
</Project>

This is the big problem we have with AOT.  I don't think it's ready for production use yet as too many of the libraries either contain reflection and/or lack the information for serialization. We backed off from switching to AOT for our Lambdas for now.
To get around your problem I think you'll have to add this directive to your Serializer Context function:
[JsonSerializable(typeof(System.Byte))]
It may be the first of many you'll have to add.

Thank you for the hint. I tried it, but it didn't help. Here is the new code:
[JsonSerializable(typeof(System.Byte))]
internal class DbRepository : IDbRepository
{
    private static readonly IAmazonDynamoDB _dbClient = new AmazonDynamoDBClient();

    public DbRepository()
    {
        DbTable = Table.LoadTable(_dbClient, Utils.GetRequiredEnvVariable("DDB_TABLE"));
    }

    public Table DbTable { get; }

The exception was the same.

Apologies, I might have been a bit unclear.  When you created your Lambda you would have a Main function and a Serializer Context function (these are the boilerplate ones you get when you create in Visual Studio using the AOT template)
private static async Task Main()
{
    Func<string, ILambdaContext, string> handler = FunctionHandler;
    await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
        .Build()
        .RunAsync();
}

[JsonSerializable(typeof(string))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
    // By using this partial class derived from JsonSerializerContext, we can generate reflection free JSON Serializer code at compile time
    // which can deserialize our class and properties. However, we must attribute this class to tell it what types to generate serialization code for.
    // See https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-source-generation
}

You need to add the [JsonSerializable(typeof(System.Byte))] to the Serializer context function like this:
[JsonSerializable(typeof(System.Byte))]
[JsonSerializable(typeof(string))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
    // By using this partial class derived from JsonSerializerContext, we can generate reflection free JSON Serializer code at compile time
    // which can deserialize our class and properties. However, we must attribute this class to tell it what types to generate serialization code for.
    // See https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-source-generation
}

Give that a try.  You might get the same error but with a different type mentioned in this like of the stack trace:
---> System.NotSupportedException: 'System.Nullable`1[System.Byte]' is missing native code or metadata. This can happen for code that is not compatible with trimming or AOT. Inspect and fix trimming and AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
which is another directive to add to that function.  Our feeling is that libraries really need to have these built in before AOT can safely be used in production.

Thank you for the explanation. I added it to my LambdaFunctionJsonSerializerContext:
[JsonSerializable(typeof(System.Byte))]
[JsonSerializable(typeof(APIGatewayHttpApiV2ProxyRequest))]
[JsonSerializable(typeof(APIGatewayHttpApiV2ProxyResponse))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
    // By using this partial class derived from JsonSerializerContext, we can generate reflection free JSON Serializer code at compile time
    // which can deserialize our class and properties. However, we must attribute this class to tell it what types to generate serialization code for.
    // See https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-source-generation
}

but it throws the same exception. I tried it also with
[JsonSerializable(typeof(System.Nullable<System.Byte>))]
But the result/exception was the same.

Thank you @branog68 for contacting the team and thank you @3Scribe for the info.  We'll have a right person look into this issue.

Thanks for opening this issue @branog68 (and thanks for all the helpful details). And thanks for the suggestion @3Scribe. The serialize was a good place to look, but this seems to be related to custom code in the dynamoDb library that is using reflection.
It enters into the library here:
https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Services/DynamoDBv2/Custom/DocumentModel/Table.cs#L427
And eventually throws here (because the type Nullable was trimmed out):
https://github.com/aws/aws-sdk-net/blob/43d13ab087d3938cb9428b251f1c64f02019150a/sdk/src/Services/DynamoDBv2/Custom/Conversion/DynamoDBEntryConversion.cs#L643
Unfortunately, this seems to be an error that can't be fixed by telling the compiler not to trim the DynamoDb library since the thing actually being trimmed out is in a System library. I can see if it would be possible to exclude System from trimming, but you'd probably end up with a gigantic binary that wouldn't be usable. I can also see if using a TrimmerRootAssembly  or other trimming option would work.
I'll also open this issue against the aws-sdk-net repo, so they can track it there.
The real fix will take a bit more effort, which is to actually remove these reflective calls from the DynamoDb library.

I opened the issue here. https://github.com/aws/aws-sdk-net/issues/2574
Let's track it there going forward, so I'll close this issue. But please open another issue or reopen this one if needed.
For what it's worth, I think I was able to put together something that may eventually work, but it's very much a hacky solution. The below code uses the calls to DoNothing to force the compiler to keep those types around. But I only got through 3 types, I assume more and more would just keep throwing the error.
public static string FunctionHandler(string input, ILambdaContext context)
{
        DoNothing<System.Nullable<System.Byte>>();
        DoNothing<System.Nullable<System.SByte>>();
        DoNothing<System.Nullable<System.Int16>>();

        IAmazonDynamoDB _dbClient = new AmazonDynamoDBClient();
        var DbTable = Table.LoadTable(_dbClient, "MyFakeTable");

        return input.ToUpper();
}

public static void DoNothing<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.All)] T>()
{
        if (typeof(T) == null)
        {
            Console.WriteLine("T is null");
        }
}

Possible work-around for now, here: https://github.com/aws/aws-sdk-net/issues/2542#issuecomment-1506539141


Serve as WSGI Application

https://flask.palletsprojects.com/en/3.0.x/deploying/

Fixed in https://github.com/tschaefer/beets-store/commit/a8b98245491100ace096f9fad4bd258bc09b619b


Feat: while registering the alert message come that is not suited on your website.

Describe the feature
Adding a good pop up message while signing in and login of a user not a default chrome alert.
Add ScreenShots

i will add like Login Successfull/Sign up successfull type with bootstrap pop up message.
Record

[X] I agree to follow this project's Code of Conduct
[X] I have starred the repository
[X] I'm a gssoc-24-extd contributor
[X] I want to work on this issue

got it @VinayLodhi1712
add the antd message here


alert count: don't get them on every dropdown click

fixes #4147
In base.html there's jQuery that attaches the get_alerts() function to every element with class dropdown-toggle.
The intention is to get the new alerts whenever somebody clicks on the alerts dropdown. But as a side effect it is also attached to every other dropdown on screen, i.e. opening the filters, opening similar findings, selecting findings for bulk edit, etc.

Later I thought maybe it was on purpose? To get updated alerts every time the user interacts with the UI? In that case we should change it into update_alert_count() because get_alerts doesn't update anything visible in the UI. wdyt @devGregA @mtesauro @Maffooch

Still would seem overkill to me.


Verification of LDAP authentication

Is there anyway to manually verify that LDAP is configured properly in Airflow? All I get is the (nearly) useless message at startup about airflow_login failing to import.

What does your configuration look like?

@planenutz what version of airflow are you using and how did you configure ldap? The error should be logged in the latest version (ie. it is not swallowed any more).

Read in the docs somewhere that LDAP does not work in 1.5.1, which is what we have running. Upgrade is in progress. Probably should close this issue until the upgrade is complete.

Upgraded to 1.6.1. still fails.
Traceback (most recent call last):
File "bin/airflow", line 17, in
args.func(args)
File "/home/vccorp/apps/airflow/lib/python2.7/site-packages/airflow/bin/cli.py", line 323, in webserver
app = cached_app(configuration)
File "/home/vccorp/apps/airflow/lib/python2.7/site-packages/airflow/www/app.py", line 119, in cached_app
app = create_app(config)
File "/home/vccorp/apps/airflow/lib/python2.7/site-packages/airflow/www/app.py", line 28, in create_app
airflow.load_login()
File "/home/vccorp/apps/airflow/lib/python2.7/site-packages/airflow/init.py", line 49, in load_login
raise AirflowException("Failed to import authentication backend")
airflow.utils.AirflowException: Failed to import authentication backend
But the module is in the path
[pads122.sj2.ups.vclk.net]vccorp@pads122:airflow: >!find
find ./ -name "ldap*"
./lib/python2.7/site-packages/airflow/contrib/auth/backends/ldap_auth.pyc
./lib/python2.7/site-packages/airflow/contrib/auth/backends/ldap_auth.py
Configuration. (we use anonymous querying and no cert for internal queries)
[webserver]
...
authenticate = True
auth_backend = airflow.contrib.auth.backends.ldap_auth
...
[ldap]
uri = ldaps://fs1.sj2.vclk.net:636
basedn = "ou=People,dc=vclk,dc=net"

Are the ldap pre-requisites installed? (I'll have a look later at providing more useful error messages).

was missing ldap3, installed that and now gunicorn is complaining. :-(

LDAP working... closing the ticket.


fix: Remove --break-system-packages when installing capellambse

In the current base image the default pip installation seems to be too old to understand that flag and since there is a global venv it should not be needed anymore.

As far as I can see this change is already included in #191

As far as I can see this change is already included in #191

Oh ok just saw a failing pipeline due to that - thanks for pointing that out!


Document NormalModuleReplacementPlugin

Add documentation for NormalModuleReplacementPlugin

As suggested by @TheLarkInn I'm willing to help with this. I'll submit a PR soon.

Added in 7a6540b via #840.


🎁 New: Package into build

Issue: Package structure is juniper-ui/build/... I want to move everything "up" a level. Or somehow allow juniper-ui/style.css to work

Ant Design actually makes you import the css from antd/dist/... so I don't feel bad about this actually.


Add completion configuration for non-Homebrew fish

Not everyone uses Homebrew fish, especially on Linux. Tested on Ubuntu 18.04 LTS.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

set -g -x could be set -gx. Just a suggestion. Otherwise LGTM.

Thanks, @dawidd6. I've made the corresponding edits.

Thanks @gromgit!

I ran into this again installing x86 brew in a Rosetta iTerm session after installing universal fish independently.
Error: Failed to link all completions, docs and manpages:
Permission denied @ rb_file_s_symlink - (../../../Homebrew/completions/fish/brew.fish, /usr/local/share/fish/vendor_completions.d/brew.fish)
I just copied the completions over manually and it works fine. Y'all definitely have bigger fish to fry, but I wanted to mention it.


Remove activities_2024 feature flag

This removes the activities_2024 feature flag.
You'll notice that this removes some of the scoreboard_prizes feature flag content, as that was on the old activities / actions completed page behind the activities_2024 flag. However scoreboard_prizes still feature on the main scoreboard index and individual scoreboard view. Is this ok @ldodds?

I think its OK to remove that. Lets tidy up and re-add if necessary.


refactor: replaced CreateTable with CreateSelect

Notes for Reviewers
Since keyCode is deprecated we should look after all the places where keyCode is getting used. And replace them with the supported keyboardEvents.
This PR fixes #7883
Signed commits

[x] Yes, I signed my commits.

@samyakshah3008 you might want to review.

Sure @leecalcote I'll review this PR 😄💯


ECCV 2020 colab notebook failed in 1st cell .

!curl -L https://raw/.githubusercontent.com/facebookresearch/habitat-sim/main/examples/colab_utils/colab_install.sh | NIGHTLY=true bash -s
!wget -c http://dl/.fbaipublicfiles.com/habitat/mp3d_example.zip && unzip -o mp3d_example.zip -d /content/habitat-sim/data
/scene_datasets/mp3d/
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  2733  100  2733    0     0   8839      0 --:--:-- --:--:-- --:--:--  8844
--2024-01-05 13:34:23--  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
Resolving repo.continuum.io (repo.continuum.io)... <IP_ADDRESS>, <IP_ADDRESS>, 2606:4700::6810:f537, ...
Connecting to repo.continuum.io (repo.continuum.io)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh [following]
--2024-01-05 13:34:23--  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
Resolving repo.anaconda.com (repo.anaconda.com)... <IP_ADDRESS>, <IP_ADDRESS>, 2606:4700::6810:8203, ...
Connecting to repo.anaconda.com (repo.anaconda.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 141613749 (135M) [text/x-sh]
Saving to: ‘Miniconda3-latest-Linux-x86_64.sh’
Miniconda3-latest-L 100%[===================>] 135.05M   162MB/s    in 0.8s
2024-01-05 13:34:24 (162 MB/s) - ‘Miniconda3-latest-Linux-x86_64.sh’ saved [141613749/141613749]
PREFIX=/usr/local
Unpacking payload ...
Installing base environment...
Downloading and Extracting Packages:
Downloading and Extracting Packages:
Preparing transaction: done
Executing transaction: done
installation finished.
WARNING:
You currently have a PYTHONPATH environment variable set. This may cause
unexpected behavior when running the Python interpreter in Miniconda3.
For best results, please verify that your PYTHONPATH only points to
directories of packages that are compatible with the Python interpreter
in Miniconda3: /usr/local
Channels:

aihabitat-nightly
conda-forge
defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:

package habitat-sim-<IP_ADDRESS>4.01.04-py3.9_headless_linux_0ca1511424326a8189615b0447c95b8253d9dfb8 requires python >=3.9,<3.10.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ habitat-sim is installable and it requires
│  ├─ python >=3.9,<3.10.0a0 , which can be installed;
│  └─ python_abi 3.9.* _cp39, which can be installed;
└─ python 3.10*  is not installable because there are no viable options
├─ python [3.10.0|3.10.10|...|3.10.9] conflicts with any installable versions previously reported;
└─ python [3.10.0|3.10.1|...|3.10.9] would require
└─ python_abi 3.10.* *_cp310, which conflicts with any installable versions previously reported.
An error occurred during the installation of Habitat-sim or Habitat-Lab.
--2024-01-05 13:35:46--  http://dl.fbaipublicfiles.com/habitat/mp3d_example.zip
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 67764531 (65M) [application/zip]
Saving to: ‘mp3d_example.zip’
mp3d_example.zip    100%[===================>]  64.62M  19.8MB/s    in 3.3s
2024-01-05 13:35:50 (19.8 MB/s) - ‘mp3d_example.zip’ saved [67764531/67764531]
Archive:  mp3d_example.zip
checkdir:  cannot create extraction directory: /content/habitat-sim/data/scene_datasets/mp3d
No such file or directory

Note we have stopped supporting colab and converted tutorials to local Jupyter notebooks instead.


Content should not flow outside the card

Is your feature request related to a problem? Please describe.
Content should not flow outside the card
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context

What problem is this feature trying to solve?
How do we know when the feature is complete?

Hello @Akshatchaube01 please assign this issue to me under gssoc 24

Hello @Akshatchaube01 i raised pr #379 please check


[Feature] AspectRatio/Resolution control

Is your feature request related to a problem?
Right now, all of the videos are recorded in a 16:9/9:16 ratio.
Describe the solution you'd like
I would like to be able to pass a custom ratio such as 4:5, 4:3, or 1:1
Describe alternatives you've considered
Other than modifying the previewer using AspectRatio() there is nothing to do. No controller other than the video quality is offered.
Additional context
N/A

This is not supported by the underlying camera package.


Certain ADTs cannot be loaded

Might be related to the MCAL and MCSH chunks reporting incorrect sizes.
Known problematic ADTs:

Daggercap Bay (Northrend)

This is covered by improved handling of MCAL chunks in https://github.com/timkurvers/blizzardry/commit/b0a405546667cfbe1d89eb4e220e1219c15537de.


Experimental support for DSC v3 preview

Description of the new feature / enhancement
As the PowerShell team continues to make progress with preview releases it would be helpful if WinGet had experimental support to use DSC v3 with WinGet Configuration.
Proposed technical implementation details
No response

What are the plans going forward regarding WinGet and DSC v3, once v3 stabilises? It seems that both winget configure and DSC v3 are addressing a similar set of problems and taking a similar approach to achieve it. Is the plan to eventually deprecate the winget config file format in favour of DSC v3 config file format? Or will winget config be deprecated altogether in favour of using dsc directly, with WinGet DSC v3 resources provided as part of WinGet? I guess for the latter, #1753 would have to be solved first.


add @guardian/libs

adds @guardian/libs
converts lint-staged config to use config files, so we can custom configs for subfolders

closing this in favour of a more, smaller changes


Don't allow duplicate keys

While it's not technically an issue with our single keys template, it is invalid Miniscript. We shouldn't allow that.

I can prevent it at the installer level. We can forbid keys with same master fingerprint to avoid user to enter two keys derived from the same seed (Or maybe it is too much :stop_sign: ).

No it's too much, the idea is just that no two same keys (like raw, actual, public keys) must be part of the same Miniscript.
(It's because a signature for a public key at a given place is also valid for the same public key at another spot.)
------- Original Message -------
Le jeudi 8 décembre 2022 à 12:28 PM, Édouard @.***> a écrit :

I can prevent it at the installer level. We can forbid keys with same master fingerprint to avoid user to enter two keys derived from the same seed (Or maybe it is too much 🛑 ).
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were assigned.Message ID: @.***>


nejug

Name of the speaker: Brian Benz
Name of the session: Perfecting reliable code delivery for the cloud with MicroProfile and OpenTracing
twitter or/& git account, if available: @bbenz / bbenz
Link of the event or short session abstract: https://nejug.org/events/2019/05/microprofile-and-opentracing.html
Date/ time/location: Monday May 6, 6PM, Burlington, MA (Boston Suburbs)
Speaker must provide "Tweet message" (who better than the speaker to write this..):
Learn about reliable @java code delivery with @microprofileio and @opentracing at the next #nejug meetup, Monday May 6, 6PM at the @MSNewEngland offices (Burlington). https://nejug.org/events/2019/05/microprofile-and-opentracing.html

@bbenz sorry this snuck by me. Maybe best to link me to the ticket next time. Can get tricky to keep track of all the requests.


[Question] Extension C8617 Config. Validate Management

Hello!
We wanted to extended the Config. Validate Management codeunit for our customer and realized that we have no chance to circumvent the rather odd behaviour of the Import Package / Package Configuration functionality.
The culprit seems to be this function:
local procedure CopyRecRefFields(RecRef: RecordRef; SourceRecRef: RecordRef; FieldRefToExclude: FieldRef)
var
    FieldRef: FieldRef;
    SourceFieldRef: FieldRef;
    FieldCount: Integer;
begin
    for FieldCount := 1 to SourceRecRef.FieldCount do begin
        SourceFieldRef := SourceRecRef.FieldIndex(FieldCount);
        if FieldRefToExclude.Name = SourceFieldRef.Name then
            exit;
        FieldRef := RecRef.FieldIndex(FieldCount);
        FieldRef.Value := SourceFieldRef.VALUE();
    end;
end;

Rather than skipping fields that should be excluded, the code copies fields of the recordref UNTIL an exclusion occurs - in this case the whole loop is exited and the fields are only partially copied. We would appreciate it if the code works like that:
local procedure CopyRecRefFields(RecRef: RecordRef; SourceRecRef: RecordRef; FieldRefToExclude: FieldRef)
var
    FieldRef: FieldRef;
    SourceFieldRef: FieldRef;
    FieldCount: Integer;
begin
    for FieldCount := 1 to SourceRecRef.FieldCount do begin
        SourceFieldRef := SourceRecRef.FieldIndex(FieldCount);
        if FieldRefToExclude.Name <> SourceFieldRef.Name then begin
            FieldRef := RecRef.FieldIndex(FieldCount);
            FieldRef.Value := SourceFieldRef.VALUE();
        end;
    end;
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Cloudfront Error

Hello,
when testing cloudfront distribution from CDN tab or trying to purge cloudfront files, I receive the following error:

Error: Distribution for origin "removed_my_domain.net" not found.

I am using latest version of 0.9.4.x branch.
Regards
Thomas

Hi @nolens ...

Are you using Amazon Cloudfront (Origin Pull) or Amazon Cloudfront (Origin Push) ?
If its the Origin Push does this error occur when you click the "Test S3 upload & CloudFront distribution" button" or does it only happen via the Purge button

Thanks

Hi Tom @nolens ... i just did an update to the Amazon Cloudfront files.  For your convenience here are those updated files that you can download directly here:
inc/options/cdn/cf.php
inc/options/cdn/s3.php
lib/S3.php
lib/W3/Cdn/S3.php
lib/W3/Cdn/S3/Cf.php
pub/css/options.css
pub/js/options.js
Give them a try and see if it solves your problem.  And let me know how it goes.
Thanks
Kimberly :octocat:

Hello Kimberly @amiga-500 ,
I have downloaded the whole distribution (not the individual files that you have linked above), activated it on a (different) staging server - and the issues I described above are gone now.
Thank you for this quick reaction, great.


How to modify this for similar captcha image

Hi, can you explain how to modify this captcha breaker for similar captcha image?
I have captcha like this with 6 digit char and the image color is already gray.
This is the example:

Thanks

@opmpro-edwin Hi, this project goal is released model to solve ulozto captcha codes (captcha codes generated by czech website www.ulozto.cz). The captcha codes you need to solve seem to be easier though they have more chars.
You cannot use pretrained model from Releases, you would need to train one yourself which requires significant CPU time. Also you would need dataset of labeled captcha codes.
I am not sure if you knew this so Id like to ask: Are you going to train the model yourself? If you are, I can add some basic documentation and give you instructions.

@JanPalasek Thanks for the reply,
Yeah I'm new in this captcha solving things, usually I use tesseract for break simple captcha.
Yes, I'm willing to train the model myself. And if You can add some basic documentation and instructions like You said, it will very helpful and I really appreciate your help.
Thank you.

@opmpro-edwin I modified code. Now it properly supports different length captcha codes using parameters. I also modified README and it contains basic quickstart how to train your own model.
Regarding your question about not converting from RGB to grayscale:
This is handled in src/train.py file by ConvertToGrayscalePreprocessor. You can just comment line where it is instantiated and added to ImagePreprocessorPipeline class.
Feedback about README or other unclarities is appreciated.

@JanPalasek Thanks for Your effort to modify code and add more info about how to train model. I will definitely try this.

Hi @JanPalasek, sorry to bother you again.
After train finished the dataset created with size more than 20G inside log/train.py-xxx. How to use the dataset as a model?

Hi @opmpro-edwin , no problem, I'm happy to help.
Model is what you train, in our case model is the neural network. Dataset is data that we feed into our model to train it, in our case dataset are the captcha images with labels.
After every epoch (= iteration of learning) finishes model saves neural network weights (parameters) and stores it to logs/train.py-xxx directory. The files containing weights are named something like cp-{EPOCH NUMBER}.h5 (and they, together, are pretty large).
If I understand the situation correctly, you successfully trained the model and checkpoints were stored in the directory. Now if you want to use the model, you could load weights and evaluate it on the test dataset.
For this purpose there is prepared script test.py. Parameter --weights_file specifies file with weights that will be loaded, it should be something like out/train.py-xxx/cp-115.h5. Other parameters are working similarly  as in train.py (e.g. --available_chars), so I suggest to use the same ones you used in train.py). I recommend to also comment out line with converting captcha images to gray (line looks similarly as in train.py).
This script loads test dataset defined by annotations-test.txt, evaluates accuracy and prints it out. It also prints out out/out_test.csv file where it is written where model made mistakes.
Lastly if you wish to use the pretrained model outside of this project (for example in your own one) and you don't want to copy-paste half of the sources, running test.py also stores entire model (entire neural network with architecture + weights) in out/model directory. This is exactly the same directory I store in release files. The section How to use pretrained model in your project in readme.md specifies how to use this directory in your own project.
I'll be happy for any feedback regarding clarity/unclarity of the project/texts.

Thanks for the detail explanation about how to make pretrained model.  The model directory created as expected but still have problem with decoder. When I use captcha image generated inside dataset, the script successfully decode the captcha but when I use my real captcha the script cannot decode it and show error like this:
Traceback (most recent call last):   File "solver.py", line 49, in <module>     loadImage('img/cap-1.png')   File "solver.py", line 25, in loadImage     r, g, b = img[:, :, 0], img[:, :, 1], img[:, :, 2] IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed
My captcha image already have same size (width, height) like captcha image from dataset.
This is my code:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import  os
import sys
from PIL import Image

try:
  import tensorflow
  import  numpy
  os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
  model = tensorflow.keras.models.load_model(os.path.join(sys.path[0], "model"))
  automation  =  True
except:
  automation  =  False

def loadImage(imgPath):
  img = Image.open(imgPath)
  # img.thumbnail((310,80))
  # img.save('img/new.png')
  # exit()
  # img.load()
  img = numpy.asarray( img, dtype="int32" )
  img = (img / 255).astype(numpy.float32)
  r, g, b = img[:, :, 0], img[:, :, 1], img[:, :, 2]
  input = 0.299 * r + 0.587 * g + 0.114 * b
  # input has now shape (70, 175)
  # we modify dimensions to match model's input
  input = numpy.expand_dims(input, 0)
  input = numpy.expand_dims(input, -1)
  # input is now of shape (batch_size, 70, 175, 1)
  # output will have shape (batch_size, 4, 26)
  output = model.predict(input)
  # now get labels
  labels_indices = numpy.argmax(output, axis=2)
  available_chars = "0987654321abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
  def  decode (li):
    result = []
    for char in li:
      result.append(available_chars[char])
    return "".join(result)
    # variable labels will contain read captcha codes
  labels = [decode(x) for x in labels_indices]
  print("Result :")
  print(labels)

if __name__ == "__main__":
  print("Start...\n")
  loadImage('img/cap-1.png')

@opmpro-edwin
Following line converts image from RGB to grayscale:
´´´
r, g, b = img[:, :, 0], img[:, :, 1], img[:, :, 2]
´´´
You need to omit it.

The error gone, but the prediction result always "mmzmmm"

@opmpro-edwin Do I understand correctly that prediction result is always "mmzmmm" even for different images?
Can u enclose training log? What was your validation accuracy during training? What was your test accuracy when u ran test.py?
Can u enclose a few training samples with their annotations and corresponding generated annotations.txt files?

@JanPalasek Yes correct, even for different images the result still same. Enclose training log? are you sure?, what file do you need to view? Because my logs file size 8.28G compressed.
This is my annotations.txt sample:
D:\Workstations\Projects\Python\captcha-breaker\out\data\9qDmsl_c3e8212d-744f-4ff8-8257-7dd19d193891.png 9qdmsl
D:\Workstations\Projects\Python\captcha-breaker\out\data\eL2oi5_092e9fab-2457-4251-89b6-7093d0fe42af.png el2oi5
D:\Workstations\Projects\Python\captcha-breaker\out\data\jb8tE8_0534a101-8ddc-4719-81f3-8dd4fd034ada.png jb8te8
D:\Workstations\Projects\Python\captcha-breaker\out\data\RBDaBn_b2051cec-c9f7-469f-a4c1-d21074be4b20.png rbdabn
D:\Workstations\Projects\Python\captcha-breaker\out\data\Gcc95b_e5141d91-8127-4cf9-803b-d70b054dc5a9.png gcc95b
D:\Workstations\Projects\Python\captcha-breaker\out\data\FhGnio_c69e17c9-6018-4b17-bec5-715ca71db3a6.png fhgnio
D:\Workstations\Projects\Python\captcha-breaker\out\data\J1v0R4_6321d753-68dd-46fd-9fa3-8f4baa1cd2d8.png j1v0r4
D:\Workstations\Projects\Python\captcha-breaker\out\data\tc5Jz8_789fed96-7337-4c2f-8e1c-37621c7cb6ed.png tc5jz8
D:\Workstations\Projects\Python\captcha-breaker\out\data\NvE8uY_9d6d37c7-5e45-42ec-bd3f-b84a53e39b1a.png nve8uy
D:\Workstations\Projects\Python\captcha-breaker\out\data\WMEURc_6fea84f5-79f3-4373-9bab-13ac9863bb20.png wmeurc
D:\Workstations\Projects\Python\captcha-breaker\out\data\qz7PqH_10e7ee6f-1c15-455f-bf6e-074d865fe33c.png qz7pqh
D:\Workstations\Projects\Python\captcha-breaker\out\data\mR6S0u_6db96fa2-6aa7-42a8-b2dd-aaea0bb28d6b.png mr6s0u
D:\Workstations\Projects\Python\captcha-breaker\out\data\oUWtpc_a2e768d7-59b7-463f-9117-f840280d4e22.png ouwtpc
D:\Workstations\Projects\Python\captcha-breaker\out\data\GVYBT4_17c95aaa-21bd-45eb-915a-a3c07737cfb5.png gvybt4
D:\Workstations\Projects\Python\captcha-breaker\out\data\sQtWuU_6abb6c41-f928-42a9-bb51-cd747d52903f.png sqtwuu
D:\Workstations\Projects\Python\captcha-breaker\out\data\gay0Hm_732fc932-ab96-4697-81e1-f813747ac231.png gay0hm
D:\Workstations\Projects\Python\captcha-breaker\out\data\CQXtbd_5ebbb87b-afba-4ddc-881c-597a9ba1cbe2.png cqxtbd
D:\Workstations\Projects\Python\captcha-breaker\out\data\mBeS7B_94e17c88-b37d-4941-aa32-fd19c1e84c4a.png mbes7b
D:\Workstations\Projects\Python\captcha-breaker\out\data\5N1Krm_c3b1927b-c178-4e77-ac3b-9258f943375e.png 5n1krm
D:\Workstations\Projects\Python\captcha-breaker\out\data\dmtTFC_c6a28e4e-002f-4ca4-b43e-85315dfb1922.png dmttfc
D:\Workstations\Projects\Python\captcha-breaker\out\data\hUXytv_9b58e20a-7c8d-4aef-b0a0-4e0b3a4d1f57.png huxytv
D:\Workstations\Projects\Python\captcha-breaker\out\data\1X8340_44f00717-1d75-4901-b309-106e62928cdc.png 1x8340
D:\Workstations\Projects\Python\captcha-breaker\out\data\WpSgoI_30385dd7-4c7d-4313-a6c6-81b0e4729be5.png wpsgoi
D:\Workstations\Projects\Python\captcha-breaker\out\data\nPj7ip_7aa83419-7479-4994-a2fe-9cb4fcec1791.png npj7ip
D:\Workstations\Projects\Python\captcha-breaker\out\data\Ctmfag_f3025d16-1eca-4ae9-a5bf-213d9eca381b.png ctmfag
D:\Workstations\Projects\Python\captcha-breaker\out\data\JBT3TB_23cecf1e-296d-491f-b079-ae53d6e0217c.png jbt3tb

For accuracy I use from tensorboard that have score Smoothed: 0.9999, Value: 1, Step: 1.324k
the filename is: train.py-2020-06-27_171817-ac=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ,bs=32,cl=6,e=1500,fl=0,l=1e-05,od=out,pm=None,rl=False,smp=None,s=42,tih=None,tiw=None,wf=None
And for available chars I use<PHONE_NUMBER>abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ for training and I already corrected my script with that available chars but the result still same.

Did you run test.py with parameter --weights_file (and others from tutorial) with path to correct weights? Also what was your test accuracy running it? Can you try to run it and write what accuracy you have (it prints it as output) and also part of generated out.csv file  (it should be in out directory)? It is possible that you forgot weights_file parameter, prediction tend to behave the way you describe if you use untrained model and unfortunately I forgot to check it in test.py file.
Тhere should be something like following code after loading parameters to args
assert args.weights_file is not None or args.pretrained_model is not None, "Weights file or pretrained model must " \
                                                                               "be passed in order to test " \
                                                                               "it."

If test accuracy printed is 0, there's a problem. Otherwise use outputted model directory for prediction as you did before.
In folder train.py-2020-06-27_171817-ac=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ,bs=32,cl=6,e=1500,fl=0,l=1e-05,od=out,pm=None,rl=False,smp=None,s=42,tih=None,tiw=None,wf=None I'd like to see contents of validation and train directories, they should containt tensorboard logs (if they're not too big).

@JanPalasek  Yes I run the test.py with parameter --weights_file.
I don't know where is the accuracy result, so I put it here the result after execute test.py
$ python src/test.py --weights_file=out/logs/train.py-2020-06-27_171817-ac\=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\,bs\=32\,cl\=6\,e\=1500\,fl\=0\,l\=1e-05\,od\=out\,pm\=None\,rl\=False\,smp\=None\,s\=42\,tih\=None\,tiw\=None\,wf\=None/cp-1324.h5  --available_chars="1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" --captcha_length=6
2020-07-14 01:12:05.495675: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-07-14 01:12:05.495952: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-07-14 01:12:07.892297: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-07-14 01:12:07.905208: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2020-07-14 01:12:07.908230: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: WEBDEV_PCNEW
2020-07-14 01:12:07.908504: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: WEBDEV_PCNEW
2020-07-14 01:12:07.908997: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
Total layers: 75
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 80, 310, 1)] 0
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, 80, 310, 1)   4           input_1[0][0]
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, 40, 155, 32)  1568        batch_normalization[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 40, 155, 32)  128         conv2d[0][0]
__________________________________________________________________________________________________
re_lu (ReLU)                    (None, 40, 155, 32)  0           batch_normalization_1[0][0]
__________________________________________________________________________________________________
max_pooling2d (MaxPooling2D)    (None, 20, 77, 32)   0           re_lu[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 20, 77, 32)   128         max_pooling2d[0][0]
__________________________________________________________________________________________________
re_lu_1 (ReLU)                  (None, 20, 77, 32)   0           batch_normalization_2[0][0]
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 20, 77, 32)   9216        re_lu_1[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 20, 77, 32)   128         conv2d_1[0][0]
__________________________________________________________________________________________________
re_lu_2 (ReLU)                  (None, 20, 77, 32)   0           batch_normalization_3[0][0]
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 20, 77, 32)   9216        re_lu_2[0][0]
__________________________________________________________________________________________________
add (Add)                       (None, 20, 77, 32)   0           max_pooling2d[0][0]
                                                                 conv2d_2[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 20, 77, 32)   128         add[0][0]
__________________________________________________________________________________________________
re_lu_3 (ReLU)                  (None, 20, 77, 32)   0           batch_normalization_4[0][0]
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 20, 77, 32)   9216        re_lu_3[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 20, 77, 32)   128         conv2d_3[0][0]
__________________________________________________________________________________________________
re_lu_4 (ReLU)                  (None, 20, 77, 32)   0           batch_normalization_5[0][0]
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 20, 77, 32)   9216        re_lu_4[0][0]
__________________________________________________________________________________________________
add_1 (Add)                     (None, 20, 77, 32)   0           add[0][0]
                                                                 conv2d_4[0][0]
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 20, 77, 32)   128         add_1[0][0]
__________________________________________________________________________________________________
re_lu_5 (ReLU)                  (None, 20, 77, 32)   0           batch_normalization_6[0][0]
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 10, 39, 64)   18432       re_lu_5[0][0]
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 10, 39, 64)   256         conv2d_5[0][0]
__________________________________________________________________________________________________
re_lu_6 (ReLU)                  (None, 10, 39, 64)   0           batch_normalization_7[0][0]
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 10, 39, 64)   36864       re_lu_6[0][0]
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, 10, 39, 64)   256         conv2d_6[0][0]
__________________________________________________________________________________________________
re_lu_7 (ReLU)                  (None, 10, 39, 64)   0           batch_normalization_8[0][0]
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, 10, 39, 64)   36864       re_lu_7[0][0]
__________________________________________________________________________________________________
add_2 (Add)                     (None, 10, 39, 64)   0           conv2d_5[0][0]
                                                                 conv2d_7[0][0]
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, 10, 39, 64)   256         add_2[0][0]
__________________________________________________________________________________________________
re_lu_8 (ReLU)                  (None, 10, 39, 64)   0           batch_normalization_9[0][0]
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, 10, 39, 64)   36864       re_lu_8[0][0]
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, 10, 39, 64)   256         conv2d_8[0][0]
__________________________________________________________________________________________________
re_lu_9 (ReLU)                  (None, 10, 39, 64)   0           batch_normalization_10[0][0]
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 10, 39, 64)   36864       re_lu_9[0][0]
__________________________________________________________________________________________________
add_3 (Add)                     (None, 10, 39, 64)   0           add_2[0][0]
                                                                 conv2d_9[0][0]
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, 10, 39, 64)   256         add_3[0][0]                      2020-07-14 01:12:11.945028: W tensorflow/python/util/util.cc:319] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
WARNING:tensorflow:From D:\Workstations\Projects\Python\captcha-breaker\venv\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1786: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

__________________________________________________________________________________________________
re_lu_10 (ReLU)                 (None, 10, 39, 64)   0           batch_normalization_11[0][0]
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, 5, 20, 128)   73728       re_lu_10[0][0]
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, 5, 20, 128)   512         conv2d_10[0][0]
__________________________________________________________________________________________________
re_lu_11 (ReLU)                 (None, 5, 20, 128)   0           batch_normalization_12[0][0]
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, 5, 20, 128)   147456      re_lu_11[0][0]
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, 5, 20, 128)   512         conv2d_11[0][0]
__________________________________________________________________________________________________
re_lu_12 (ReLU)                 (None, 5, 20, 128)   0           batch_normalization_13[0][0]
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, 5, 20, 128)   147456      re_lu_12[0][0]
__________________________________________________________________________________________________
add_4 (Add)                     (None, 5, 20, 128)   0           conv2d_10[0][0]
                                                                 conv2d_12[0][0]
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, 5, 20, 128)   512         add_4[0][0]
__________________________________________________________________________________________________
re_lu_13 (ReLU)                 (None, 5, 20, 128)   0           batch_normalization_14[0][0]
__________________________________________________________________________________________________
conv2d_13 (Conv2D)              (None, 5, 20, 128)   147456      re_lu_13[0][0]
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, 5, 20, 128)   512         conv2d_13[0][0]
__________________________________________________________________________________________________
re_lu_14 (ReLU)                 (None, 5, 20, 128)   0           batch_normalization_15[0][0]
__________________________________________________________________________________________________
conv2d_14 (Conv2D)              (None, 5, 20, 128)   147456      re_lu_14[0][0]
__________________________________________________________________________________________________
add_5 (Add)                     (None, 5, 20, 128)   0           add_4[0][0]
                                                                 conv2d_14[0][0]
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, 5, 20, 128)   512         add_5[0][0]
__________________________________________________________________________________________________
re_lu_15 (ReLU)                 (None, 5, 20, 128)   0           batch_normalization_16[0][0]
__________________________________________________________________________________________________
conv2d_15 (Conv2D)              (None, 3, 10, 256)   294912      re_lu_15[0][0]
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, 3, 10, 256)   1024        conv2d_15[0][0]
__________________________________________________________________________________________________
re_lu_16 (ReLU)                 (None, 3, 10, 256)   0           batch_normalization_17[0][0]
__________________________________________________________________________________________________
conv2d_16 (Conv2D)              (None, 3, 10, 256)   589824      re_lu_16[0][0]
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, 3, 10, 256)   1024        conv2d_16[0][0]
__________________________________________________________________________________________________
re_lu_17 (ReLU)                 (None, 3, 10, 256)   0           batch_normalization_18[0][0]
__________________________________________________________________________________________________
conv2d_17 (Conv2D)              (None, 3, 10, 256)   589824      re_lu_17[0][0]
__________________________________________________________________________________________________
add_6 (Add)                     (None, 3, 10, 256)   0           conv2d_15[0][0]
                                                                 conv2d_17[0][0]
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, 3, 10, 256)   1024        add_6[0][0]
__________________________________________________________________________________________________
re_lu_18 (ReLU)                 (None, 3, 10, 256)   0           batch_normalization_19[0][0]
__________________________________________________________________________________________________
conv2d_18 (Conv2D)              (None, 3, 10, 256)   589824      re_lu_18[0][0]
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, 3, 10, 256)   1024        conv2d_18[0][0]
__________________________________________________________________________________________________
re_lu_19 (ReLU)                 (None, 3, 10, 256)   0           batch_normalization_20[0][0]
__________________________________________________________________________________________________
conv2d_19 (Conv2D)              (None, 3, 10, 256)   589824      re_lu_19[0][0]
__________________________________________________________________________________________________
add_7 (Add)                     (None, 3, 10, 256)   0           add_6[0][0]
                                                                 conv2d_19[0][0]
__________________________________________________________________________________________________
global_average_pooling2d (Globa (None, 256)          0           add_7[0][0]
__________________________________________________________________________________________________
dense (Dense)                   (None, 372)          95604       global_average_pooling2d[0][0]
__________________________________________________________________________________________________
reshape (Reshape)               (None, 6, 62)        0           dense[0][0]
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 6, 62)        3906        reshape[0][0]
==================================================================================================
Total params: 3,630,298
Trainable params: 3,625,944
Non-trainable params: 4,354
__________________________________________________________________________________________________
tf.Tensor(0.033, shape=(), dtype=float32)

And this is the validation and train train-validation.zip inside train.py-2020-06-27_171817-ac=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ,bs=32,cl=6,e=1500,fl=0,l=1e-05,od=out,pm=None,rl=False,smp=None,s=42,tih=None,tiw=None,wf=None

@opmpro-edwin
The last number is accuracy: tf.Tensor(0.033, shape=(), dtype=float32) means you have 0.033 accuracy on test set.
Unfortunately for some reason Tensorboard cannot view your logs, we'll have to manage without it.
Question: How large was your dataset? I always used 10k.
Also

Smoothed: 0.9999, Value: 1, Step: 1.324k >
these was validation or training accuracy?

Anyway I'll try to walk through how I use it:

delete data folder in out and generate new samples of size 10k
note: I also tested sample size of only 1k and model did not have good performance on validation data

venv/bin/python src/captcha_generation/simple_captcha_generate.py --available_chars=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ --captcha_length=6 --dataset_size=10000 --width=150 --height=50

Generated image looked like this:

generate annotations files

venv/bin/python src/captcha_generation/captcha_annotate.py

run train.py

venv/bin/python src/train.py --available_chars=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ --captcha_length=6

Output contains something like this
Train on 8000 samples, validate on 1000 samples
Epoch 1/1500
8000/8000 [==============================] - 141s 18ms/sample - loss: 3.7116 - sparse_categorical_accuracy: 0.0377 - all_correct_acc: 0.0000e+00 - val_loss: 3.6358 - val_sparse_categorical_accuracy: 0.0315 - val_all_correct_acc: 0.0000e+00
Epoch 2/1500
8000/8000 [==============================] - 124s 16ms/sample - loss: 3.4871 - sparse_categorical_accuracy: 0.0651 - all_correct_acc: 0.0000e+00 - val_loss: 3.4216 - val_sparse_categorical_accuracy: 0.0720 - val_all_correct_acc: 0.0000e+00
Epoch 3/1500
8000/8000 [==============================] - 126s 16ms/sample - loss: 3.0524 - sparse_categorical_accuracy: 0.1391 - all_correct_acc: 0.0000e+00 - val_loss: 2.8892 - val_sparse_categorical_accuracy: 0.1738 - val_all_correct_acc: 0.0000e+00
Epoch 4/1500
8000/8000 [==============================] - 129s 16ms/sample - loss: 2.3314 - sparse_categorical_accuracy: 0.2850 - all_correct_acc: 5.0000e-04 - val_loss: 2.0704 - val_sparse_categorical_accuracy: 0.3345 - val_all_correct_acc: 9.7656e-04
Epoch 5/1500
8000/8000 [==============================] - 127s 16ms/sample - loss: 1.6041 - sparse_categorical_accuracy: 0.4845 - all_correct_acc: 0.0143 - val_loss: 1.6660 - val_sparse_categorical_accuracy: 0.4882 - val_all_correct_acc: 0.0107
Epoch 6/1500
8000/8000 [==============================] - 131s 16ms/sample - loss: 1.0693 - sparse_categorical_accuracy: 0.6607 - all_correct_acc: 0.0915 - val_loss: 1.1421 - val_sparse_categorical_accuracy: 0.6463 - val_all_correct_acc: 0.0664
Epoch 7/1500
8000/8000 [==============================] - 129s 16ms/sample - loss: 0.7117 - sparse_categorical_accuracy: 0.7802 - all_correct_acc: 0.2315 - val_loss: 1.0008 - val_sparse_categorical_accuracy: 0.7210 - val_all_correct_acc: 0.1484
Epoch 8/1500
8000/8000 [==============================] - 131s 16ms/sample - loss: 0.5335 - sparse_categorical_accuracy: 0.8405 - all_correct_acc: 0.3647 - val_loss: 0.7868 - val_sparse_categorical_accuracy: 0.7833 - val_all_correct_acc: 0.2471
Epoch 9/1500
8000/8000 [==============================] - 131s 16ms/sample - loss: 0.4216 - sparse_categorical_accuracy: 0.8806 - all_correct_acc: 0.4749 - val_loss: 0.8812 - val_sparse_categorical_accuracy: 0.7775 - val_all_correct_acc: 0.2363
Epoch 10/1500
8000/8000 [==============================] - 127s 16ms/sample - loss: 0.3430 - sparse_categorical_accuracy: 0.9082 - all_correct_acc: 0.5679 - val_loss: 0.7386 - val_sparse_categorical_accuracy: 0.8177 - val_all_correct_acc: 0.3057
Epoch 11/1500
8000/8000 [==============================] - 127s 16ms/sample - loss: 0.3012 - sparse_categorical_accuracy: 0.9231 - all_correct_acc: 0.6211 - val_loss: 0.6407 - val_sparse_categorical_accuracy: 0.8467 - val_all_correct_acc: 0.3867
Epoch 12/1500
8000/8000 [==============================] - 138s 17ms/sample - loss: 0.2901 - sparse_categorical_accuracy: 0.9281 - all_correct_acc: 0.6430 - val_loss: 0.6893 - val_sparse_categorical_accuracy: 0.8387 - val_all_correct_acc: 0.3457
Epoch 13/1500
8000/8000 [==============================] - 134s 17ms/sample - loss: 0.2660 - sparse_categorical_accuracy: 0.9364 - all_correct_acc: 0.6770 - val_loss: 0.6218 - val_sparse_categorical_accuracy: 0.8578 - val_all_correct_acc: 0.4062
Epoch 14/1500
8000/8000 [==============================] - 128s 16ms/sample - loss: 0.2378 - sparse_categorical_accuracy: 0.9479 - all_correct_acc: 0.7280 - val_loss: 0.8136 - val_sparse_categorical_accuracy: 0.8265 - val_all_correct_acc: 0.3193
Epoch 15/1500
8000/8000 [==============================] - 127s 16ms/sample - loss: 0.2571 - sparse_categorical_accuracy: 0.9431 - all_correct_acc: 0.7097 - val_loss: 0.6875 - val_sparse_categorical_accuracy: 0.8527 - val_all_correct_acc: 0.3848

find best epoch by val_all_correct_acc,  (in my case it's epoch 13 with val_all_correct_acc of 0.4062) and use corresponding checkpoint for weights_file and run test.py

venv/bin/python src/test.py --available_chars=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ --captcha_length=6 --weights_file=out/logs/train.py-2020-07-14_074256-ac=1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ,bs=32,cl=6,e=1500,fl=0,l=1e-05,od=out,pm=None,rl=False,smp=None,s=42,tih=None,tiw=None,wf=None/cp-13.h5

Last line of output is something like
tf.Tensor(0.46, shape=(), dtype=float32)

which indicates 46% accuracy.

now you can use your model directory for prediction and you can use it for predictions as in your code

@JanPalasek
I use 10k dataset too.
So from your explanation to find best epoch by vall_all_correct_acc we no need to wait till finish (Epoch 1500/1500), just find the highest vall_all_correct_acc right?
Okay will try again as you suggest.
Thanks

So from your explanation to find best epoch by vall_all_correct_acc we no need to wait till finish (Epoch 1500/1500), just find the highest vall_all_correct_acc right?

@opmpro-edwin yes.
Good luck and let me know how it went.

@JanPalasek
Which epoch with better result from this?
Epoch 6/1500
8000/8000 [==============================] - 183s 23ms/sample - loss: 1.0334 - sparse_categorical_accuracy: 0.6707 - all_correct_acc: 0.0921 - val_loss: 7.4869 - val_sparse_categorical_accuracy: 0.3425 - val_all_correct_acc: 9.7656e-04
Epoch 137/1500
8000/8000 [==============================] - 180s 22ms/sample - loss: 0.1153 - sparse_categorical_accuracy: 0.9902 - all_correct_acc: 0.9436 - val_loss: 12.0510 - val_sparse_categorical_accuracy: 0.1527 - val_all_correct_acc: 9.7656e-04
Epoch 285/1500
8000/8000 [==============================] - 179s 22ms/sample - loss: 0.0725 - sparse_categorical_accuracy: 0.9967 - all_correct_acc: 0.9810 - val_loss: 11.8792 - val_sparse_categorical_accuracy: 0.1595 - val_all_correct_acc: 9.7656e-04
Epoch 478/1500
8000/8000 [==============================] - 181s 23ms/sample - loss: 0.0713 - sparse_categorical_accuracy: 0.9936 - all_correct_acc: 0.9629 - val_loss: 10.2297 - val_sparse_categorical_accuracy: 0.2755 - val_all_correct_acc: 9.7656e-04
Epoch 797/1500
8000/8000 [==============================] - 181s 23ms/sample - loss: 0.0510 - sparse_categorical_accuracy: 0.9969 - all_correct_acc: 0.9818 - val_loss: 10.9829 - val_sparse_categorical_accuracy: 0.2090 - val_all_correct_acc: 9.7656e-04

@opmpro-edwin
Every one of them is unusable (too low val_all_correct_acc). The model seems to be overfitting on the training data and does not generalize well.

@JanPalasek
Ah is too low, I think 9.7x is high because the others value is more like 0.02x.
My captcha resolution is set to 310x80 pixels

@opmpro-edwin
9.7656e-04 is something like 0.0009.
I generated 10k captcha images of size 310x80 and ran training for a few episodes. Generated images look something like this  .
Log:
Epoch 1/1500
8000/8000 [==============================] - 351s 44ms/sample - loss: 3.7132 - sparse_categorical_accuracy: 0.0334 - all_correct_acc: 0.0000e+00 - val_loss: 3.6217 - val_sparse_categorical_accuracy: 0.0310 - val_all_correct_acc: 0.0000e+00
Epoch 2/1500
8000/8000 [==============================] - 349s 44ms/sample - loss: 3.5953 - sparse_categorical_accuracy: 0.0402 - all_correct_acc: 0.0000e+00 - val_loss: 3.6308 - val_sparse_categorical_accuracy: 0.0312 - val_all_correct_acc: 0.0000e+00
Epoch 3/1500
8000/8000 [==============================] - 343s 43ms/sample - loss: 3.4618 - sparse_categorical_accuracy: 0.0709 - all_correct_acc: 0.0000e+00 - val_loss: 3.7899 - val_sparse_categorical_accuracy: 0.0633 - val_all_correct_acc: 0.0000e+00
Epoch 4/1500
8000/8000 [==============================] - 345s 43ms/sample - loss: 2.9916 - sparse_categorical_accuracy: 0.1455 - all_correct_acc: 0.0000e+00 - val_loss: 2.8040 - val_sparse_categorical_accuracy: 0.1740 - val_all_correct_acc: 0.0000e+00
Epoch 5/1500
8000/8000 [==============================] - 341s 43ms/sample - loss: 2.1425 - sparse_categorical_accuracy: 0.3182 - all_correct_acc: 0.0015 - val_loss: 2.3429 - val_sparse_categorical_accuracy: 0.3392 - val_all_correct_acc: 0.0020
Epoch 6/1500
8000/8000 [==============================] - 345s 43ms/sample - loss: 1.3195 - sparse_categorical_accuracy: 0.5623 - all_correct_acc: 0.0331 - val_loss: 1.3928 - val_sparse_categorical_accuracy: 0.5617 - val_all_correct_acc: 0.0312
Epoch 7/1500
8000/8000 [==============================] - 388s 48ms/sample - loss: 0.7459 - sparse_categorical_accuracy: 0.7654 - all_correct_acc: 0.2097 - val_loss: 0.7959 - val_sparse_categorical_accuracy: 0.7545 - val_all_correct_acc: 0.1816
Epoch 8/1500
8000/8000 [==============================] - 397s 50ms/sample - loss: 0.4639 - sparse_categorical_accuracy: 0.8688 - all_correct_acc: 0.4324 - val_loss: 0.7158 - val_sparse_categorical_accuracy: 0.7940 - val_all_correct_acc: 0.2666
Epoch 9/1500
8000/8000 [==============================] - 370s 46ms/sample - loss: 0.3255 - sparse_categorical_accuracy: 0.9146 - all_correct_acc: 0.5865 - val_loss: 0.4183 - val_sparse_categorical_accuracy: 0.8902 - val_all_correct_acc: 0.5137

It seems to do much more than in your case, though your captcha images seem to be a bit harder.
You can try to

Obtain more training data.
Resize to have more or less shape like 175x70. Training script has parameters transformed_img_width and transformed_img_height parameter for this. Be careful though, this could damage quality of the image and make learning even harder (check a few images with opencv imwrite).
Standard ML stuff, like regularization, change model a bit (start from smaller one, increase size, watch if performance increases, mb rectangular convolutions, because your image is rectangle), maybe some preprocessing. If you don't know about this and you'd like to learn it, I recommend Deep learning course on coursera from Andrew Ng., some parts can be found on youtube.

Otherwise I don't have any more ideas what could be wrong, this model was designed to work on specific ulozto captchas, even though it seems to work in general to some extent too.

@JanPalasek
Yeah maybe I force Your design to fit in my needs, but I will try Your recommendation anyway and if it still fail I will take Your advice to learn Deep learning course from Andrew Ng.
Anyway, thank you for taking the time and effort to help me, it really meant a lot.


ElementTriN1 on different length-scales

The ElementTriN1()*ElementTriP1() works fine for me as long as the lengthscale of the system is ~1, if I change it to the order of ~ 1e-6 the results become unprecise. This seems especially for solutions which have non-zero values in both subspaces.
I've had a look on the assembly:
from skfem import *

@BilinearForm
def test_form(u_n, u_p, v_n, v_p, w):
    print(u_n.max(), u_p.max())
    return u_p * v_p

mesh = MeshTri().refined(2)
basis = Basis(mesh, ElementTriN1()*ElementTriP1())
test_form.assemble(basis)

print('-' * 80)

mesh = MeshTri().refined(2).scaled((1e-6,1e-6))
basis = Basis(mesh, ElementTriN1()*ElementTriP1())
test_form.assemble(basis)

The first assembly has values like
0.0 0.816847572980459
3.6336951459609206 0.0

while the second one is like
0.0 0.816847572980459
3633695.1459609214 0.0

Could this be the problem, that there are orders of magnitude between the values and eigs cannot solve the system precisely?

Perhaps.
Quick comment: Sometimes in solid mechanics you are able to use different length units to solve issues such as this.

Thanks!
At the moment I scale all length units (i.e. the size of the mesh and the wavelength to be in the oder of 1), effectively I just leave out the μ in μm for all values.
While this works, it would somehow simplify the whole description for more complex problems if I could stick to SI-base-units for everything, as I otherwise have to scale all physical parameters in the system, too.
This is of course doable, but it would be somehow nice to understand why the system, which should in principle invariant of the scale changes that dramatically.
Might there be some error a minor error in the implementation of ElementTriN1? Is it expected that the values for u_n change a lot while the values for u_p keep constant? And do you maybe know of any literature which addresses such a problem?
Somehow it seems wrong to me, that the precision changes when I scale the whole system :/

Perhaps not 'error', but there are usually several 'equivalent' options to implementing a finite element. What I mean that there are different sets of basis functions that span the same polynomial space. But since we use floats and not reals they are not equivalent in practice.
A simple example is one dimensional P2 element. We can use (1-x, x, (1-x)x) or we can use the ones listed here:
https://github.com/kinnala/scikit-fem/blob/master/skfem/element/element_line/element_line_p2.py
Former type of basis is used when the polynomial degree changes from element-to-element and a high degree is needed in some parts of the domain.
What I'm saying is that there are most likely different 'equivalent' (in the sense of reals but not floats) ways to write down the basis for N1 which could be more robust against such small elements.
Literature is usually rather quiet on such topics so it might be worthwhile checking other FEM codes and how they write the basis there.

BTW did you test with ElementTriN2, does it lead to the same issue?

What I'd like to see here is some sort of minimal example of the issue.

I'll try to now study your first example if I can see anything amiss.

All those elements ElementTriRT1, ElementTriBDM1, ElementTriN1 exhibit same behaviour. I checked and double checked all the formulations and they look correct. I cannot find anything that'd be wrong.
While looking for some hints on the subject I found this reference which points out an interesting problem related to these elements which I hadn't heard about: https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=2867&context=all_theses
Nevertheless, that is not related to these 'large values' you are describing. I think these are simply caused by scaling 1/det(J) in the Piola transformation. Jacobian determinant gets smaller for smaller elements and it's normal for the quantity to get larger.

Thanks!  Sad to hear that literature is so quiet on this stuff, seems to be quite relevant for actually doing the simulations (?)
And thanks for all the checking, I'll have a look on the reference.
I've modified ex64 to have a minimal example I think:
"""Waveguide cutoff analysis."""

import numpy as np
from skfem import *
from skfem.helpers import *

# three different mesh and element types
mesh_elem = [
    (
        MeshTri.init_tensor(np.linspace(0, 1, 40),
                            np.linspace(0, .5, 20)),
        ElementTriN1() * ElementTriP1(),
    ),
    (
        MeshQuad.init_tensor(np.linspace(0, 1, 40) ** 0.9,
                             np.linspace(0, .5, 20)),
        ElementQuadN1() * ElementQuad1(),
    ),
    (
        MeshTri.init_tensor(np.linspace(0, 1, 20),
                            np.linspace(0, .5, 10)),
        ElementTriN2() * ElementTriP2(),
    ),
]

mesh, elem = mesh_elem[0]
failing = True

errors = []
scale_factors = np.linspace(-1,8)

for scale_factor in scale_factors:
    scale = 10**(-scale_factor)
    mesh_scaled = mesh.scaled([scale]*2)
    basis = Basis(mesh_scaled, elem)

    epsilon = lambda x: 1. + 0. * x[0]
    # epsilon = lambda x: 3 * (x[1] < 0.25) + 1
    one_over_u_r = 1

    @BilinearForm
    def aform(E, lam, v, mu, w):
        return one_over_u_r * curl(E) * curl(v) * (1 if failing else scale**2)

    @BilinearForm
    def gauge(E, lam, v, mu, w):
        # set div E = 0 using a Lagrange multiplier
        return dot(grad(lam), v) + dot(E, grad(mu))

    @BilinearForm
    def bform(E, lam, v, mu, w):
        return epsilon(w.x) * dot(E, v) / (scale**2 if failing else 1)

    A = aform.assemble(basis)
    B = bform.assemble(basis)
    C = gauge.assemble(basis)

    lams, xs = solve(*condense(A + C, B, D=basis.get_dofs()),
                     solver=solver_eigen_scipy_sym(k=6))

    # compare against analytical eigenvalues
    err1 = np.abs(lams[0] - np.pi ** 2)
    err2 = np.abs(lams[1] - 4. * np.pi ** 2)
    err3 = np.abs(lams[2] - 4. * np.pi ** 2)

    if __name__ == "__main__":
        print('TE10 error: {}'.format(err1))
        print('TE01 error: {}'.format(err2))
        print('TE20 error: {}'.format(err3))

    errors.append((err1, err2, err3))

if __name__ == "__main__":
    import matplotlib.pyplot as plt

    errors = np.array(errors)
    plt.xlabel('scale factor')
    plt.ylabel('error')
    plt.plot(scale_factors, errors)
    plt.show()

    fig, axs = plt.subplots(4, 1)
    for itr in range(4):
        (E, Ebasis), _ = basis.split(xs[:, itr])
        Ei = Ebasis.interpolate(E)
        plotbasis = Ebasis.with_element(ElementDG(ElementTriP2()))
        Emag = plotbasis.project(np.sqrt(dot(Ei, Ei)))
        plotbasis.plot(Emag,
                       colorbar=True,
                       ax=axs[itr],
                       shading='gouraud',
                       nrefs=2)
    plt.show()

It only fails for failing=True, as I'd guess in that case we equally have different orders of magnitude for the matrix entries (?)
It also fails for all different elements, so I'd guess another implementation won't help?
For this case it's nice, that we can scale both subsystems differently as the Lagrange multipliers anyway go to zero, in the general case this wouldn't be possible, right?

Okay, I just realize, that I can just scale both subsystems independently. I just multiply every v_z and u_z by that scale and done.
It seems to me, that I should aim for each term being (without considering u/v) in the order of 1 and and considering a derivative as something like 1/scale_factor (?)


[Docs] TRE Service Integrator can add workspace template to TRE workspace

Description
As a TRE Service Integrator
I want to be able to make workspaces templates available to be deployed into a TRE instance by:

Registering and existing workspace template with the TRE
Creating a new workspace template and registering it with the TRE
So that Azure TRE users (Researcher etc) and have workspace available to them in the Azure TRE instance.

Acceptance criteria

TRE service integrator is able to register and deploy a workspace, based on existing workspace template, to a TRE instance.
TRE service integrator is able to create, register, and deploy a new workspace template  (not yet defined), to a TRE instance.

Technical notes:
You can refer to the entire TRE documentation, but recommended to focus on
https://github.com/microsoft/AzureTRE/blob/develop/docs/registering-workspace-templates.md
https://github.com/microsoft/AzureTRE/blob/develop/docs/authoring-workspace-templates.md

To be revisited and validated once #584, #718 and #903 are closed.

Registering templates
Authoring workspace templates


Plate confusion

Which plate files should I use for corne classic with cherry MX? Thank you!

https://github.com/foostan/crkbd/tree/master/case/acrylic_plate/cherry

thank you!


List as inputs in sweep yaml

Python version: 3.6.9
wandb version 0.12.5
OS: Linux
Summary: In my code run.py I use argparse for specifying the model's hyperparameters. One of those parameter is the number of hidden layers and their size, which is specified by a list i.e.
parser.add_argument("--layers_size_prior", default=[500, 500, 500], nargs='+', help="No of units in then hidden layers", type=int)
Then, I can specify the size of --layers_size_prior simply as python run.py --layers_size_prior 100 100 for example.
I want to use sweep for parameter optimization for --layers_size_prior  and I create the following yaml file
project: my_project
program: ./run.py
method: grid
metric:
  name: auc
  goal: maximize
parameters:
  layers_size_prior:
     values: [500 500 500, 1000 1000 1000]

but I get the following error message
ValueError: invalid literal for int() with base 10: ' '

Which makes me think that there is no nargs='+' functionality in sweep.
As a workaround, I did something similar as in using tuple in sweeps

hi @rogelioamancisidor, i think the issue here is that the values 500 500 500 and 1000 1000 1000 do not get parsed as lists in yaml. if you want them passed to your --layers_size_prior arg as a list, try specifying them as a list in yaml:
project: my_project
program: ./run.py
method: grid
metric:
  name: auc
  goal: maximize
parameters:
  layers_size_prior:
     values: [[500, 500, 500], [1000, 1000, 1000]]

It passes a string, i.e. ['1000 1000 1000'], so it crashes.

hi @rogelioamancisidor, as a workaround, you can do:
sweep.yaml:
project: my_project
program: ./run.py
method: grid
metric:
  name: auc
  goal: maximize
parameters:
  layers_size_prior:
     values: ["500 500 500", "1000 1000 1000"]

run.py:
#!/usr/bin/env python

import argparse
class ParseAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        print('%r %r %r' % (namespace, values, option_string))
        values = list(map(int, values.split()))
        setattr(namespace, self.dest, values)

parser = argparse.ArgumentParser()
parser.add_argument("--layers_size_prior", default=[500, 500, 500], help="No of units in then hidden layers", action=ParseAction)
args = parser.parse_args()

import wandb
wandb.init()
print(wandb.config)

i verified locally that this works. can you let me know if this fixes things for you?

reference: https://docs.python.org/3/library/argparse.html#action

@dannygoldstein Thanks for the suggestion. I'll check it out soon and let you know.

Are you planning on creating a fix for nargs input?

+1

+1

+1

We are closing this issue due to inactivity. If you feel that it merits additional review, please provide a reproduction and add a comment explaining why. Thank you.

When using something like
parameters:
  kernel-sizes:
    values:
      - "3 3 2"
      - "2 2 1"
      - "4 3 2"

this is how it gets input to the python script:
About to run command: /usr/bin/env python grid.py "--kernel-sizes=3 3 2"

argparse won't recognize this as a list of numbers because they are quoted, so it will see them as the string "3 3 2" rather than the sequence of strings "3" "3" "2". With parser.add_argument("--kernel-sizes", type=int, nargs="+" (or nargs="*") it would not be able to parse the former but would be able to parse the latter.


Fix appearing extra labels when 'keras_generator' function is called

Hi @mdbloice, I found Augmentator very useful tool, but I faced with distracting bug with keras_generator function, so my pull request is gonna fix it. The bug is following:
I have a training set which contains images of two classes, so there's two subfolders in the train_128_path, and i'm doing this
p = Augmentor.Pipeline(train_128_path)
...
generator = p.keras_generator(batch_size, image_data_format="channels_first") # It's ok
...
# Second call
generator = p.keras_generator(batch_size, image_data_format="channels_first")
# It's not ok, it behaves like there's three labels
# and generator returns label vectors of size three

After the first call it creates directory output relative to train_128_path, and on the second call  keras_generator treats output like new image class.
I found problem with the function 'Pipeline._populate', it calls scan and passes abs_output_directory which actually is not absolute, but scan expects an absolute path (problem with the line 195)
abs_output_directory = os.path.join(source_directory, output_directory)
...
self.augmentor_images, self.class_labels = scan(source_directory, abs_output_directory)

So I've fixed it in the scan function, by the way removed extra spaces.

Hi @RuslanIsrafilov I've been trying to reproduce the error you are speaking about, and I can't seem to be able to do it. Could you post a minimal working example?
P.S. are you by chance creating another Pipeline object after you have created the first one (in the same directory)?

For instance, I have directory structure like this:
├── data
│   ├── 1
│   ├── 2
├── train.py

Content of train.py
import Augmentor

p = Augmentor.Pipeline('data')
p.crop_random(probability=1, percentage_area=0.5)

g = p.keras_generator(1)
print('\nlabel = {}'.format(next(g)[1]))

After the first run it will print
python3 ./train.py
Initialised with 2 image(s) found.
Output directory set to data/output.
label = [[0 1]]

And it will create new directory data/output. No doubt, I can run the same one more time
python3 ./train.py
Initialised with 2 image(s) found.
Output directory set to data/output.
label = [[1 0 0]] # <- three labels???

Ok yes indeed, I see now. That's a good find @RuslanIsrafilov , thanks for putting the effort in.
Actually all this time I didn't realise that in _populate() I've actually been sending the relative path all along. I'll fix that as well now that you've pointed it out.
Thanks again!

@mdbloice, from my point of view, unconditional creation of output directory is not the best design decision. Perhaps it should be created somewhere in the sample() method. Calling the constructor of Pipelinedoent's mean what I'm really going to write images to disk.

Yeah, it's an artefact of back when the only option was to write images to disk. I'll keep it in mind and change it to write that folder only when sample() is called. Thanks for the feedback.


Code fixes

This fixes most Prettier and ESLint errors. One ESLint error remains, maybe you can look into it @KilDesu:
$ npx eslint .

quaff/src/lib/components/list/QItemSection.svelte
  10:6  error  'ctx' is defined but never used  @typescript-eslint/no-unused-vars

Notes from the commit message:

removed unused "getAllProperties" from parse-interface
refactored QCodeBlock to support svelte and include styles
QDrawer: lots of currently unused props were removed
QFooter: wrapped inside an if statement to actually use "value"
removed many (apparently) unused styles
props should never have the same variable name as their type, so I had
to change QComponentDocs to prop name componentDocs

I'll likely add a few more changes to this branch.

@KilDesu: I've also added automatic type check and lint for PRs now. After you've fixed the last ESLint error in this branch, the checks should pass.

@KilDesu: I've also added automatic type check and lint for PRs now. After you've fixed the last ESLint error in this branch, the checks should pass.

Just checked the error, it is unfounded because ctx is used: it's set just few lines below. I think eslint is expecting it to be used at least once in "read" but as this is a writable store, it's only used in "write" so the parent can use it in "read". This error shouldn't be here.


Can we use the latest openssl and libssh2 libraries version with existing pods ?

Can you please update the libraries to latest.
openssl: 1.1.1o
libssh2: 1.10.0
Can we use the latest library version with existing pods or is there any  plan to update those ?

@iiimartin  Did you manage to find a work around for this issue? Thanks

Same issue here because the server require an algorithm that's only implemented in libssh 1.10.0, just wondering if you guys manage to use the latest library with NMSSH? Thank you!

any update on this?


fix nougat

closes #376
http://stackoverflow.com/a/32710030/1410905 solution based on that answer.
@marcshilling I hope you can point cases which I need to test.

What is the difference between this and what #464 is trying to fix?

First of all, It's OLD solution.
In my PR I use a new way.

Closing in favor of #464


[FEATURE] Add KF_SPECTATORS variable

Is your feature request related to a problem? Please describe.
Without this, the server won't allow any spectators! The default is 0 right now.
Describe the solution you'd like
Simply create a variable that can set this up like the other ones. Make its default value as 6.
Describe alternatives you've considered
Manually editing the config.

Added in v3.8 -- closing.


First Batch of Tests is Complete and All Pass

I have written unit tests for each and every operator (~40 for each of the ~25 operators).  The interpreter seems to match the output of the standard Python shell on each test. Through this process, I found and fixed a few bugs.
We can add more unit tests later if I missed any cases. I will also add integration tests in the future that combine multiple operators into one expression to tests things like associativity and precedence. But, I think it's good to get this set of tests on main for now.

No merge conflicts


How to Update with a multi column primary key?

I am trying to update with:
// update
err = repo.Update(ctx, entity.Driver{
	SessionID: 1241414,
	CarIdx:    11,
	CarNumber: "33",
	CreatedAt: time.Now(),
	UpdatedAt: time.Now(),
})

.
// Update saves the changes to an driver in the database.
func (r repository) Update(ctx context.Context, driver entity.Driver) error {
	return r.db.With(ctx).Model(&driver).Update()
}

The SessionID and CardIdx are the primary key fields.
CREATE TABLE driver
(
    session_id INTEGER NOT NULL,
    car_idx INTEGER NOT NULL,
    PRIMARY KEY (session_id, car_idx),
    ....

I don't understand all your magic how the model is build, but the model of Driver does not have any Primary Keys.
// Driver represents an driver record.
type Driver struct {
	SessionID               int
	CarIdx                  int       `yaml:"CarIdx"`
	UserName                string    `yaml:"UserName"`
	AbbrevName              string    `yaml:"AbbrevName"`
	Initials                string    `yaml:"Initials"`

Found the solution.
https://github.com/go-ozzo/ozzo-dbx#crud-operations
If the struct has a field named ID or Id, by default the field will be treated as the primary key field. If you want to use a different field as the primary key, tag it with db:"pk". You may tag multiple fields for composite primary keys. Note that if you also want to explicitly specify the column name for a primary key field, you should use the tag format db:"pk,col_name".


Harvesters Usage

Hello all,
I intend to use harvesters with a Cheetah Pregius camera. This is all new to me so if someone can validate the steps below, it would be great.
I would have to connect the camera to my local network.
Then I will have to install harversers and a GenTL Producer.
Ideally, once this is done, my camera should be recognized by harvesters and I should be able to issue commands to my camera.
Do let me know of I am missing something. Thanks so much in advance!

Hi, I would suggest you following the instruction in the README.rst (or the top page of this GitHub Harvester web). It should not be difficult as you concern. Good luck! Regards, Kazunari. PS: Please do not forget that we do not talk anything about the camera-specific issues. If you have a question about the camera itself, I will highly recommend you to ask a question to your local distributor of the camera because that is their job.


Search not working on Russian language  first letter is uppercase (v 2.0.0-beta.12)

Search not working in v. 2.0.0-beta.12
<multiselect
                                    :value="orderStore.tehcenter"
                                    @select="onSelectTehcenter"
                                    :options="orderStore.tehcenters"
                                    :searchable="true"
                                    :close-on-select="true"
                                    :show-labels="false"
                                    :allow-empty="false"
                                    :internal-search="true"
                                    placeholder="Выберите техцентр"
                                    label="name"
                                    track-by="id"
                                    group-values="tehcenters"
                                    group-label="city"
                            >
                            </multiselect>

I want to search on label="name", name it is сyrillic string on Russian language and If the first letter is uppercase, It does not work :(
Example:
name - Москва - ООО Рога и Копыта ул. Ленина 15 А
Search string - Москва - working
Search string - Москва - not working
Search string - МОСКВА - not working
How to search not case insensitive?

@bigperson I believe this issue belongs to different repository :)
Please fill it again in http://github.com/monterail/multiselect

That's cute :D
https://github.com/monterail/vue-multiselect actually xD


Coin Hunter

Fiverr Solana (SPL) token creator
Example token for customer to experience (SPL) token
https://www.fiverr.com/coin_hunter
I agree to not ping anybody on Discord/Twitter/email about this pull request. Instead I will inquire by posting a new comment in the pull request if needed.

PRs are reviewed in bulk and and can take up to two weeks to be merged.
Please provide the following information for your token.
Please include change to the src/tokens/solana.tokenlist.json file in the PR.
DON'T modify any other token on the list.
At minimum each entry should have

Token Address: 77G8qexRPteENifYefxkRXdNEE8M5cG5XAAeLjfeANqp
Token Name: Coin Hunter
Token Symbol: CH
Logo<EMAIL_ADDRESS>Link to the official homepage of token: https://www.fiverr.com/coin_hunter

please follow the instructions

please follow the instructions

followed can u mention issue

Hello,
I'm freelancer on fiverr my job is submitting SPL token details to git hub can i know about what the issue of this submitting.
Thank You.


Test benchmarks in CI

Objective

This reverts #16833, and completely goes against #16803.
Turns out running cargo test --benches runs each benchmark once, without timing it, just to ensure nothing panics. This is actually desired because we can use it to verify benchmarks are working correctly without all the time constraints of actual benchmarks.

Solution

Add the --benches flag to the CI test command.

Testing

cargo run -p ci -- test

Something happened with the bench
thread 'main' panicked at benches/benches/bevy_tasks/iter.rs:80:25:

Approving since the failure should be fixed in a separate PR

Approving since the failure should be fixed in a separate PR

Surprisingly not! The panic was an integer overflow that was never caught in our benchmarks because they are usually compiled in release mode. I fixed it in aa2e600f3aab8427821a6a21b73fc82895a3cc77, though, to explicitly wrap.


how to run locally (better docs)

I am a newbie regarding k8, so please bear with me, I have being trying to install the operator locally but can't follow, understand this from the README:
Running locally
You can run the Operator locally against a remote namespace using the operator-sdk:

Prerequisites:

operator-sdk installed
kubectl pointing to the local context. minikube automatically sets the context to the local VM. If not you can use kubectl config use <context> or (if using the OpenShift CLI) oc login -u <user> <url>
make sure to deploy the custom resource definition using the command kubectl create -f deploy/crds
$ operator-sdk up local --namespace=<namespace> --operator-flags="<flags to pass>"

I have minikube, already cloned the repo (https://github.com/integr8ly/grafana-operator) and ran:  kubectl create -f deploy/crds:
$ kubectl create -f deploy/crds
customresourcedefinition.apiextensions.k8s.io/grafanas.integreatly.org created
customresourcedefinition.apiextensions.k8s.io/grafanadashboards.integreatly.org created
customresourcedefinition.apiextensions.k8s.io/grafanadatasources.integreatly.org created

I installed opensdk using brew install operator-sdk but then I am stuck with:
 operator-sdk up local --namespace=<namespace> --operator-flags="<flags to pass>"

I created my namespace:
  kubectl create namespace grafana

But have no idea how to proceed, could the docs please be updated or any idea about how to then install the operator?
thanks in advance.

@nbari Here are some steps you can follow

Deploy the CRDS
Run the operator-sdk up local ( this may change to run --local --watch-namespace with sdk versions)
You should see the operator-sdk logs on your local, without any error such as grafana CRD missing.
After you see this running something like this

adheip@adheip:~/data/rill/rill-druid-operator$ operator-sdk run --local --watch-namespace operator
INFO[0000] Running the operator locally in namespace operator.
{"level":"info","ts":1591183927.953777,"logger":"cmd","msg":"Go Version: go1.14"}
{"level":"info","ts":1591183927.953795,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1591183927.953803,"logger":"cmd","msg":"Version of operator-sdk: v0.17.0"}
{"level":"info","ts":1591183927.954635,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1591183927.954646,"logger":"leader","msg":"Skipping leader election; not running in a cluster."}
{"level":"info","ts":1591183931.5419123,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"<IP_ADDRESS>:8383"}
{"level":"info","ts":1591183931.5422516,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1591183931.542636,"logger":"cmd","msg":"Could not generate and serve custom resource metrics","error":"operator run mode forced to local"}
{"level":"info","ts":1591183934.534026,"logger":"cmd","msg":"Could not create metrics Service","error":"failed to initialize service object for metrics: OPERATOR_NAME must be set"}
{"level":"info","ts":1591183937.5209918,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1591183937.5210438,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1591183937.5210743,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1591183937.521505,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1591183937.5215435,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"grafana-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1591183937.9222095,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"grafana-controller"}
{"level":"info","ts":1591183937.9223292,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"grafana-controller","worker count":1}

This sdk will look for the kubeconfig in your .kube folder by default
You can now deploy the CR which grafana, datasource and dashboard. You should see the grafana pods coming up, kubectl get grafana -n {{namesspace}}...........

Hope this helps.

Hi @AdheipSingh many thanks for the help I was available to run it using:
kubectl create namespace operator

I cloned the repo:
$ git clone https://github.com/integr8ly/grafana-operator
cd grafana-operator

I deployed the cdr:
kubectl create -f deploy/crds

And then using the operator-sdk:
operator-sdk run local --watch-namespace operator

This is the version I have:  operator-sdk version operator-sdk version: "v0.18.0", commit: "921c33982687d6e9219fe0a1a326515fb37ae040", kubernetes version: "v1.18.2", go version: "go1.14.3 darwin/amd64"
After that seems to be running and I can also get the Prometheus metrics from http://localhost:8080/metrics , from logs I see:
{"level":"info","ts":1591200913.1568131,"logger":"controller_grafanadashboard","msg":"running periodic dashboard resync"}
{"level":"info","ts":1591200913.156875,"logger":"controller_grafanadashboard","msg":"no grafana instance available"}
{"level":"info","ts":1591200922.543613,"logger":"controller_grafanadatasource","msg":"running periodic datasource resync"}
{"level":"info","ts":1591200922.5437078,"logger":"controller_grafanadatasource","msg":"no datasources configmap found"}
{"level":"info","ts":1591200923.155421,"logger":"controller_grafanadashboard","msg":"running periodic dashboard resync"}
{"level":"info","ts":1591200923.155477,"logger":"controller_grafanadashboard","msg":"no grafana instance available"}
{"level":"info","ts":1591200933.153769,"logger":"controller_grafanadashboard","msg":"running periodic dashboard resync"}
{"level":"info","ts":1591200933.153811,"logger":"controller_grafanadashboard","msg":"no grafana instance available"}

But now from step 6 If I am understanding properly, I could install  CR  (custom resources) in the namespace operator but could you please give me an example or hint about how to do this?  I found https://github.com/integr8ly/grafana-operator/tree/master/deploy/examples but not sure.

@nbari yes, you can use one of those examples. To deploy Grafana you could create for example this: https://github.com/integr8ly/grafana-operator/blob/master/deploy/examples/Grafana.yaml
The command would be: kubectl create -f deploy/Grafana.yaml -n <your namespace>. This will trigger the operator to install Grafana in that namespace.