GSoC2024: Dask Starter Problem

Dask Starter Problem
This is a toy problem meant as a pre-application exercise for the GSOC project Enable Dask execution of NASA time-domain analysis.
Estimated completion time: a few hours
Please complete all required tasks and whichever optional task(s) allows you to convey your thought process most easily. The goal is to see what your starting points are, not what you could do with several days of research and optimization.
Overview
The gsoc/2024/dask-toy-problem branch contains a directory called gsoc-dask-toy-problem/ with one file, archives.py. The file contains four public functions, get_*_lightcurves. These functions vary in their resource usage and runtimes. They take at least one argument, num_sample, which may vary between about 5 and 500,000.
The task is to parallelize these functions using Dask so that they run efficiently for large sample sizes. A basic script to run the functions serially looks like this:
import archives  # be sure to import this **first**
import pandas as pd

num_sample = 100  # may vary between about 5 and 500,000

gaia_df = archives.get_gaia_lightcurves(num_sample)
heasarc_df = archives.get_heasarc_lightcurves(num_sample)
wise_df = archives.get_wise_lightcurves(num_sample)
ztf_df = archives.get_ztf_lightcurves(num_sample)

lightcurves_df = pd.concat([gaia_df, heasarc_df, wise_df, ztf_df])

Instructions

Clone this repo and check out the branch gsoc/2024/dask-toy-problem.
Write code.

Required: Start a Dask cluster and stop it when finished. A local cluster is sufficient.
Optional: Execute the four archives functions and concatenate the results into a single Pandas DataFrame. Use any Dask method(s) you want. Your code should complete at least as fast as the serial script above. You can look at the code in archives.py, but do not alter it.

Optional: Write text. 300 words max, included as a '.md' file.

For any code that you did not write but would if you had more time, write down what you would do.
Write down any questions you have that, if answered, might help guide your design choices.
Test your code to determine how it scales as num_sample increases. Write down the results, your interpretation of the results, and/or what you would try next to see if it improves your code.

Required: Open a PR with your code and (optional) writeup to merge to gsoc/2024/dask-toy-problem branch. Have GSoC2024 in the title of your PR.

If you have a question, please ask in a comment on this issue.

Clarifying the anticipated tasks for this GSoC project
Please reread the project description. Development of the entire Science Console is in its early stages, so we don't have much more information to share right now. I will add the few details here that I can. Implementation of a Dask cluster within the Console is being worked on in parallel by a different team. Major tasks for this GSoC project are anticipated to be:

Learn how Dask is expected to be implemented within the Science Console.
Determine an efficient method for running the light curve collection code at scale using the Console's planned Dask implementation.
Write code implementing the solution from step 2.
Test the code.
Iterate steps 2-4, if needed.
Begin work on a different use case, time permitting. (Steps 2-5 on a different code than light curve collection.)

My guess is that the existing light curve collection code will not need to be altered, but that's a guess based on incomplete information. The most relevant detail of the collection code that was not included in the toy problem is this:
One of the functions takes much longer than the others (for large sample sizes). When running the code using Dask, this will have to be addressed in some way.
Here is what we have done to address this so far:
To speed up that function, we have added parallelization internally (to that function only) using python’s multiprocessing . This is separate from the parallelization of the full code mentioned in the project description, and two cannot be used together, which limits the effectiveness of the method demonstrated in the notebook. I have recently written a bash script that executes the same underlying python code and outperforms the notebook because it allows that long-running function to use its internal parallelization.


NoClassDefFoundError crash if WAILA is no present

Hi
Minecraft will crash during loading screen if WAILA is not present.
Tested with:
RFDrills-1.7.10-1.7.1
Log:
http://pastebin.com/tnyh20XD
Adding WAILA fixes this issue.

Oh.
That's quite a derp. I'll fix it today though.

Tested the latest version RFDrills-1.7.10-1.7.2. No longer crashes if WAILA is not present.

Sorry for not responding earlier. I released a fix to Curseforge yesterday but forgot to push it to Github.


Android IInAppBillingVerifyPurchase

Curious why productId and transactionId are not set to  IInAppBillingVerifyPurchase.VerifyPurchase in the Android implementation?

They aren't needed really at all as you are just verifying the signed data and signature.
Do you need them for something?

they come in handy for server side verification and processing of GCP notifications

It is also rather confusing if you want to access this information directly from the methods parameters only to realize that you have to parse it from the JSON in signedData yourself.


Winampjs request 0.js file from build directory

I wanted to test new filepicker API but the version in npm seems to be out of date. So I've clone the repo build be project as library but when I included bundle.js or bundle.min.js it reqest 0.js file that also in root directory but I'm not opening from build.
What is this file for and how can I make it load from <ROOT>/apps/winamp/winamp2-js/build instead of from <ROOT>.
My code look like this:
jQuery(function($) {
    swift.then(function(swift) {
        swift.register_app('winamp', {
            label: 'winamp',
            run: function() {
                if (typeof winamp2js === 'undefined') {
                    $.getScript('apps/winamp/winamp2-js/built/bundle.js').then(winamp);
                } else {
                    winamp();
                }
                function winamp() {
                    const winamp = new winamp2js({
                        initialTracks: [{
                            url: "apps/winamp/built/mp3/llama-2.91-5b9fc63f9d440bc6078cbdd21ab2a9d9.mp3"
                        }],
                        initialSkin: {
                            url: "apps/winamp/built/skins/XMMS-Turquoise-46ab165b574d97e2e51b22cb47a7661c.wsz"
                        },
                        filePickers: [
                            {
                                contextMenuName: "server file...",
                                filePicker: async () => {
                                    // just to test new API
                                    var url = 'apps/winamp/built/mp3/llama-2.91-5b9fc63f9d440bc6078cbdd21ab2a9d9.mp3';
                                    const files = await fetch(url).then(function(response) {
                                        return response.blob();
                                    }).then(blob => {
                                        return [{
                                            link: URL.createObjectURL(blob),
                                            name: 'llama.mp3'
                                        }];
                                    });
                                    return files.map(file => ({
                                        url: file.link,
                                        defaultName: file.name
                                    }));
                                },
                                requiresNetwork: true
                            }
                        ],
                        enableHotkeys: false // Enable hotkeys
                    });
                    // Render after the skin has loaded.
                    winamp.renderWhenReady($('<div/>').appendTo('body')[0]);
                }
            }
        });
    });
});

Ah, sorry about this. In https://github.com/captbaritone/winamp2-js/commit/ea9bc832b304567c6b935936d694da97b876bf9c I moved jsmediatags (which we use to parse id3 tags) to a lazy dependency since we don't need it until someone gives us an mp3 file that we haven't seen yet.
I need to figure out how to handle this in the npm module before I ship a new version. I've created an issue to track here: https://github.com/captbaritone/winamp2-js/issues/523
As for your example, is the conversion from URL to blob and back to URL just to test things out? Is there a reason you're not just passing the URL to Winamp2-js directly?

Could you try version 0.0.5?

This just for test but my final code will fetch the file using proxy script so I can open files that are on the server but not accessed from url.


Visual C++ 2003 support

What steps will reproduce the problem?
1.building in Visual C++ 2003 with RAPIDJSON_SSE2
2.
3.

What is the expected output? What do you see instead?
error C3861: '_BitScanForward': identifier not found, event with
argument-dependent lookup

What version of the product are you using? On what operating system?

Please provide any additional information below.

My solution is below:

#if _MSC_VER < 1400
extern "C" unsigned char _BitScanForward(unsigned long* Index, unsigned long
Mask);
#pragma intrinsic(_BitScanForward)
#endif

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 8 Jan 2014 at 2:49

Original comment by<EMAIL_ADDRESS>on 20 Jun 2014 at 11:24

Changed state: Accepted

https://github.com/miloyip/rapidjson/commit/e63125f1a1c03abbaabb489a00d88fc9224f
b095

Original comment by<EMAIL_ADDRESS>on 13 Jul 2014 at 4:17

Changed state: Fixed


QFlow Fails Due to ABC String Length Assertion

Running a design through QFlow fails during synthesis due to ABC stage due to the following assertion fail:
ABC: Scl_Libertyosys-abc: src/map/scl/sclLiberty.c:344: Scl_LibertyReadString: Assertion `Pair.End-Pair.Beg < ABC_MAX_LIB_STR_LEN' failed.
I worked around this by just increasing the ABC_MAX_LIB_STR_LEN in the ABC source files and recompiled Yosys but I suspect there's a more elegant way to get around this issue.

This is not specifically a qflow issue;  you would need to take this issue up with Claire Wolf (yosys) or Alan Mischenko (abc).  I've never seen this issue before. . .  Just how long are the strings that are causing this?

Given that this issue is not open_pdks-specific, and not even qflow-specific, I am going to close this issue now.  If you have any objections to that, feel free to re-open it.


Improve Fuzzer Job logging

Logging to stderr massively slows down the actual fuzzer jobs when a verbose log level is used but we still want the information. This PR adds logging to file (ERROR and above will still show on stderr) which will be uploaded as part of the repro artifacts.

Thanks @assignUser


Adding support for indexing Date data types

Description
Adding support for indexing Datetime data type in qbeast-spark.
In reference to issue.
Type of change
It is an enhancement that would allow for Datetime data type to be indexed without the need of a workaround like casting the value to a Long type or similar.
Spark has two date data types, we could add support for both.
More elaborated explanation and guidelines regarding the addition of new types here
Checklist:

[x] New feature / bug fix has been committed following the Contribution guide.
[x] Add comments to the code (make it easier for the community!).
[ ] Change the documentation.
[x] Add tests.
[ ] Your branch is updated to the main branch (dependent changes have been merged).

How Has This Been Tested?
Please describe the tests that you ran to verify your changes.

TransformerTest: I have added a test to check the correct behavior of LinearTransformation with Timestamps
TransformerIndexingTest: I have added a test to ensure timestamps read and write correctly with spark

Test Configuration:

Spark Version: 3.1.1
Hadoop Version: 3.2
Running Locally on M1

Well done! Just a few more comments/questions:

We could put something more user-friendly in the title of the PR. For example: Support for indexing Date columns.  Then everyone can understand what this is about. Because Transformer and QDataTypes are something more dependent on implementation rather than functionality. :man_technologist:
I guess for the implementation we are only adding Timestamp. But you have intentions to add a Date as well, right?

Looks good!


mysql socket not found

In a current test, I found that I always got 'database error: file or directory not found'
This was fixed by adding to the dockerfile a command to link the mysql socket to /var/mysql/mysql.sock (from /tmp/mysql.sock)
And can submit a pull request if you like.

It feels strange to put something related to db into this container.
Maybe you should try something with mysql container or with docker-compose service.

Same error here. What do you suggest to proceed?

This image is too old for new docker versions, i'll keep some time later to update it.
Any of you changed anything in compose file?

No. I was trying to use yours.


Wrong circuits and wrong measurement results are produced by specific backend

Environment

Qiskit version: 1.1.1
Python version: Python 3.12.4
Operating system: Ubuntu 22.04.3 LTS

What is happening?
When running program1, two wrong behaviors were found:
(1) Some fake backends declare duplicated ancilla qubits. The number of duplicated ancilla qubits varies between different fake backends.
(2) The Statevector simulator treats ancilla qubits as common qubits and measures all of them. This issue can also be reproduced by running program2.
These issues appear to persist across multiple versions of qiskit (e.g., 1.0.2), not just the latest ones. Do you have any suggestions or workarounds for these behaviors?
How can we reproduce the issue?
program1:
from qiskit import QuantumCircuit, transpile
from qiskit.quantum_info import Statevector
from qiskit_ibm_runtime.fake_provider import FakeYorktownV2, FakeSherbrooke

qc = QuantumCircuit(2)
qc.cp(0.0, 1, 0)
qc.measure_all()

backend = FakeYorktownV2()
qc1 = transpile(qc, backend)
print(qc1) # wrong circuit

result = backend.run(qc1).result().get_counts()
print(result)

qc1.remove_final_measurements()
statevector = Statevector(qc1)
print(statevector.probabilities_dict()) # wrong measurement result

output1:
                ░ ┌─┐
      q_0 -> 0 ─░─┤M├───
                ░ └╥┘┌─┐
      q_1 -> 1 ─░──╫─┤M├
                ░  ║ └╥┘
ancilla_0 -> 2 ────╫──╫─
                   ║  ║
ancilla_1 -> 3 ────╫──╫─
                   ║  ║
ancilla_2 -> 4 ────╫──╫─
                   ║  ║
       meas: 2/════╩══╩═
                   0  1
{'01': 62, '10': 24, '00': 938}
{'00000': 1.0} # where should be {'00': 1.0}

The openqasm code:
OPENQASM 3.0;
include "stdgates.inc";
bit[1] meas;
barrier $0, $1;
meas[0] = measure $0;

program2:
from qiskit.qasm3 import dumps, load
from qiskit.quantum_info import Statevector
from qiskit_aer import AerSimulator, QasmSimulator

qc = load("demo.qasm")
backend = AerSimulator()
result = backend.run(qc).result().get_counts()
print(result)
qc.remove_final_measurements()
statevector =Statevector(qc)
print(statevector.probabilities_dict())

output2:
{'0': 1024}
{'00': 1.0} # where should be {'0': 1.0}

What should happen?
(1) The fake backends should not declare duplicated ancilla qubits.
(2) Statevector should identify the correct number of declared qubits.
Any suggestions?
No response

These are both expected behaviours - when you transpile your circuit to hardware, it becomes a physical circuit, which means it must account for all the qubits on the device. These are the extra ancillas you're seeing.
The call Statevector(qc1) gets the statevector of all qubits in the circuit. After ancilla expansion, it's correct that it includes all hardware qubits; there are various swap-mapping and gate syntheses passes that could cause mid-circuit ancilla use, or even the ancillas to end in a dirty state at the end of the circuit. If you want to reduce it to the qubits you care about at the end, you'll need to use something like qiskit.quantum_info.partial_trace, and you can find the qubits arguments by inspecting qc.layout.final_index_layout.

Thank you for your quick response. I also found that using qc1.draw(idle_wires=False) is helpful for observing the key parts of the circuit.


Installed npm modules are incomplete/missing files after packaging

[x] I have read the contribution documentation for this project.
[x] I agree to follow the code of conduct that this project follows, as appropriate.
[x] I have searched the issue tracker for an issue that matches the one I want to file, without success.

Please describe your issue:
When running in development mode npm run start everything works perfectly.
After packaging the app and running it an error is given (in the browser) saying that the module "acorn" is missing.
The module is found in the resources folder though, but it's "main" file is missing. (which is found in the module's "dist" folder, which is missing).
This somehow points to a difference how the modules are being installed in dev and when the app is packaged.
When packaging the app the following commands are run to install the npm modules: npm install --production flag and npm prune --production.
So I tried installing my modules like that in my dev environment to see if the acorn would also be missing it's "dist" folder but that does not seem to be the case.
Is there something I'm missing in the difference how the modules are installed when packaging?

Console output when you run electron-packager with the environment variable DEBUG=electron-packager. Please include the stack trace if one exists.
  electron-packager Electron Packager 8.7.0 +0ms
  electron-packager Node v7.6.0 +3ms
  electron-packager Host Operating system: linux (x64) +1ms
  electron-packager Packager Options: {"_":["..","electron-packager-missing-modules"],"all":false,"deref-symlinks":true,"download":{"strictSSL":true},"overwrite":true,"prune":true,"quiet":false,"ignore":"(dist)","dir":"..","name":"electron-packager-missing-modules","protocols":[]} +0ms
  electron-packager Target Platforms: linux +0ms
  electron-packager Target Architectures: x64 +0ms
  electron-packager Inferring appVersion from version in /home/resurge/repos/electron-packager-missing-modules/package.json +16ms
  electron-packager Inferring target Electron version from electron in /home/resurge/repos/electron-packager-missing-modules/package.json +9ms
  electron-packager Application name: electron-packager-missing-modules +1ms
  electron-packager Target Electron version: 1.6.10 +0ms
  electron-packager Ignored path regular expressions: [ '(dist)',
  '/node_modules/electron($|/)',
  '/node_modules/electron-prebuilt(-compile)?($|/)',
  '/node_modules/electron-packager($|/)',
  '/\\.git($|/)',
  '/node_modules/\\.bin($|/)',
  '\\.o(bj)?$',
  '/tmp/electron-packager' ] +0ms
  electron-packager Downloading Electron with options {"strictSSL":true,"platform":"linux","arch":"x64","version":"1.6.10"} +5ms
Packaging app for platform linux x64 using electron v1.6.10
  electron-packager Creating /tmp/electron-packager/linux-x64-template +259ms
  electron-packager Extracting /home/resurge/.electron/electron-v1.6.10-linux-x64.zip to /tmp/electron-packager/linux-x64-template +1ms
  electron-packager Initializing app in /tmp/electron-packager/linux-x64/electron-packager-missing-modules-linux-x64 from /tmp/electron-packager/linux-x64-template template +1s
  electron-packager Ignored paths based on the out param: [ '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-darwin-ia32',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-darwin-x64',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-darwin-armv7l',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-linux-ia32',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-linux-x64',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-linux-armv7l',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-mas-ia32',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-mas-x64',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-mas-armv7l',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-win32-ia32',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-win32-x64',
  '/home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-win32-armv7l' ] +2ms
  electron-packager Pruning modules via: npm prune --production +480ms
  electron-packager Renaming electron to electron-packager-missing-modules in /tmp/electron-packager/linux-x64/electron-packager-missing-modules-linux-x64 +2s
  electron-packager Moving /tmp/electron-packager/linux-x64/electron-packager-missing-modules-linux-x64 to /home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-linux-x64 +1ms
Wrote new app to /home/resurge/repos/electron-packager-missing-modules/dist/electron-packager-missing-modules-linux-x64

What command line arguments are you passing? Alternatively, if you are using the API, what
parameters are you passing to the packager() function?
(Ran from in the "dist" folder)
DEBUG=electron-packager electron-packager .. electron-packager-missing-modules --overwrite --ignore="(dist)"

Please provide either a failing minimal testcase (with a link to the code) or detailed steps to
reproduce your problem. Using electron-quick-start
is a good starting point.
https://github.com/resurge/electron-packager-missing-modules/commit/3dbf8902eebe784820ec8fde371433958684cda6
This project just renders a pug template after clicking a button.
Internally one of pug's dependencies relies on acorn, which seems to have issues being packaged.
(Although this is not just an issue with acorn, if I manually copy the module so that it's complete an other module starts complaining about missing files)

I'm sorry, it seems I didn't read well enough.
I had found this very similar issue: https://github.com/electron-userland/electron-packager/issues/527
But it mentioned the node_modules folder being entirely missing from the packaging output, but this was not the case with me so I assumed this was a different issue.
I then tried building my app not in a separate folder, but straight in the project root and then everything seems to work.
I guess this one can be closed right away.
Sorry for the hassle.


Add support for subschemas

This PR introduces support for anyOf and allOf subschemas.

It breaks with this testcase:
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$defs": {
        "foo": {
            "type": "object",
            "properties": {
                "foo": {
                    "type": "string"
                }
            },
            "required": [
                "foo"
            ]
        }
    },
    "id": "https://example.com/objectAllOf",
    "type": "object",
    "properties": {
        "flags": {
            "anyOf": [
                {
                    "type": "string"
                },
                {
                    "type": "boolean"
                }
            ]
        },
        "configurations": {
            "type": "array",
            "items": {
                "anyOf": [
                    {
                        "type": "object",
                        "properties": {
                            "bar": {
                                "type": "number"
                            }
                        },
                        "required": [
                            "bar"
                        ]
                    },
                    {
                        "type": "object",
                        "properties": {
                            "baz": {
                                "type": "boolean"
                            }
                        }
                    },
                    {
                        "$ref": "#/$defs/foo"
                    }
                ]
            }
        }
    }
}

Any updates on the status of this PR?  The fixes in this branch would be helpful to a project I am working on as well.  Thanks

Hi @omissis any updates on when you think this is ready for use? I am using allOf and anyOf quiet extensively in one of my project and I would really like to use this tooling. Is there anything I might do to help get it ready?

Seems to break when there is an allOf or anyOf at root level:
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "id": "https://example.com/objectAllOf",
    "allOf": [
      {
        "type": "object",
        "properties": {
          "foo": {
            "type": "string"
          }
        },
        "required": ["foo"]
      },
      {
        "type": "object",
        "properties": {
          "bar": {
            "type": "number"
          }
        },
        "required": ["bar"]
      },
      {
        "type": "object",
        "properties": {
          "configurations": {
            "type": "array",
            "items": {}
          }
        }
      }
    ]
  }

A quick update for all those who are waiting: I have just finished bringing this one up to date with all the things I have merged on main, and fixed all the broken tests code that I could find. This puts me in a position where I can look at all the feedback above and try to fix the issues. This is now the top priority for this repo to merge, and save for very small and trivial fixes and improvements, I will not merge anything else until this is done. Thank you for your patience.

Thank you, @omissis! If you are looking for more inspiration for testing, it might be worth looking into what other json schema tools use in their repos:

Marwes/schemafy
openapi-json-schema-tools/openapi-json-schema-generator
corvus-dotnet/Corvus.JsonSchema
YousefED/typescript-json-schema
GREsau/schemars
victools/jsonschema-generator
JamesNK/Newtonsoft.Json.Schema
invopop/jsonschema

If I find time in the coming weeks, I could also take a look to see if there are any interesting test cases we could add.

update: I have added support for the cases provided by @joepjoosten and @iiacoban42, now I am looking into the case @afrittoli highlighted. Once that is done, I still have a (private) set of schemas I want to test the library onto, and then if no further objections are raised, I will merge this PR and release a new version. Hopefully it'll see the light before end of the year :)

Hey, I was using this from a branch and I have noticed the issue that wasn't present in published version - occasionally it would generate duplicates now  that were not there before.
I am running it like this:
go-jsonschema -p mcp https://raw.githubusercontent.com/modelcontextprotocol/specification/refs/heads/main/schema/schema.json -o ./schema.go

Go vet tells me "schema.go:2634:6: ImageContent redeclared in this block".
I typically see the tool generating something like this:
type BlobResourceContents = BlobResourceContents

This creates an alias type, that has same name as original type, which creates conflict as golang does not like that.
I saw the connected issue about duplicate types, but it does not quite seem related, as I am not getting any numbers in those names, they are just full duplicates.

Hey @strowk thanks for your comment. yeah, I have noticed that, too, using my own private battery of tests where I also have that case. I think it might have to do with the fact I am trying to do deep merges of the underlying data structures. Unfortunately this is a blocking issue for merging this PR. I will try find a solution and report.

@strowk can you check again using the latest commit from this branch please?


Fix Android breadcrumbs on web

Fixes https://github.com/getsentry/sentry-dart/issues/979

Sorry, I mistakenly added a bad initial commit with an extra line.


How to change the code block font size?

Is looks like TextStyle::Monospace doesn't affect that part.

I don't know why does the code size is always smaller then the normal text.

Interesting. By default egui_extras is used for syntax highlighting and it appears that it has hardcoded the monospace font to 10!
It should work correctly with the better_syntax_highlighting feature enabled.
Side note: I recommend to use the default font as a fallback font, so that the copy button in the code block will be displayed with the correct icon instead of the ?. I think the proportional font is used for that.

Looks like there is an issue for this already. https://github.com/emilk/egui/issues/3549

Interesting. By default egui_extras is used for syntax highlighting and it appears that it has hardcoded the monospace font to 10!
It should work correctly with the better_syntax_highlighting feature enabled.
Side note: I recommend to use the default font as a fallback font, so that the copy button in the code block will be displayed with the correct icon instead of the ?. I think the proportional font is used for that.

I don't think this is an issue with this crate. By the way, how do I set the fallback font?

Should be fixed in the next egui version https://github.com/emilk/egui/pull/5090


Test drivers or framework of plugins

We want to write tests of plugins. For example, fluentd provides test framework (set of mocks) so that we can write tests easily for plugins.

I've written tests for embulk-output-oracle plugin.
I hope they can be of some help.
https://github.com/hito4t/embulk-output-jdbc/tree/oracle-test/embulk-output-oracle/src/test/java/org/embulk/output/oracle

embulk-junit4 (f.k.a. embulk-test) is there, and we plan to have a new embulk-junit5 sooner. #1597


Broken images with uninitialized constant OpenSearch::Transport::Transport::HTTP::Faraday::VERSION

Hey,
I'm currently having problems with the images complaining about uninitialized constant OpenSearch::Transport::Transport::HTTP::Faraday::VERSION which is an error from https://github.com/opensearch-project/opensearch-ruby/issues/205 and https://github.com/fluent/fluent-plugin-opensearch/pull/116.
I tried v1.15-ruby3-build.99, v1.15-ruby3-build.100 and v1.15-ruby3 as the operators default for 4.2.2.
@pepov or someone else, would it be possible for you to bump the versions accordingly and build new images (recognized that the v1.15 directory is already gone).
Thanks in advance

There is no release yet...
https://github.com/fluent/fluent-plugin-opensearch/issues/118

fluentd:
  image:
    tag: v1.16-full-build.118

seams to work

Hey @sebastiangaiser,
As you already mentioned the v1.16 is the latest release and it will be the default version for a later logging operator release, if everything goes well then 4.5 will be shipped with it by default.
If you have further questions please reopen.


Sema: Warn when Swift classes attempt to implement ObjC +initialize.

Swift's language model doesn't guarantee that type metadata will ever really be used, which makes overriding initialize() error-prone and not really any better than manually invoking an initialization function. Warn about this for Swift 3 compatibility and reject attempts to override +initialize in Swift 4.

@swift-ci Please smoke test

@jrose-apple Does this look reasonable for 3.1?


Mill integration failed to resolve build dependency

This happened while Scala Steward tried to extract dependencies from https://github.com/ToToTec/CmdOption:
'mill -i -p /home/frank/scala-steward/workspace/repos/ToToTec/CmdOption/scala-steward.sc show org.scalasteward.mill.plugin.StewardPlugin/extractDeps' exited with code 1
Compiling /home/frank/scala-steward/workspace/repos/ToToTec/CmdOption/scala-steward.sc
Downloading https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloaded https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloading https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Downloaded https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Downloading https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloaded https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloading https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Downloaded https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Failed to resolve ivy dependencies:Error downloading de.tototec:de.tobiasroeser.mill.osgi_2.13:0.2.0
  not found: /home/frank/.ivy2/local/de.tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
  not found: https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom

Mill couldn't resolve the de.tobiasroeser.mill.osgi plugin which is imported in the build of ToToTec/CmdOption.
@hamnis @lefou Any idea how to fix this?

@fthomas as I mentioned on gitter. this repo seems to be using mill 0.6 and the plugin is using mill 0.7 ... I never found a way to make both work satisfactory.
Mill 0.7 is Scala 2.13 and that is probably the reason.
@lefou might be able to shed some more light on this.

@fthomas how is the mill script installed ?
It needs to respect the .mill-version file, or env variable MILL_VERSION.
I finally found a way to integrate 0.6 and PR is coming shortly.

It seems that #1501 wasn't sufficient to fix this problem:
2020-07-04 13:21:24,081 ERROR Check cache of ToToTec/CmdOption failed
java.io.IOException: 'mill -i -p /home/frank/scala-steward/workspace/repos/ToToTec/CmdOption/scala-steward.sc show org.scalasteward.mill.plugin.StewardPlugin/extractDeps' exited with code 1
Compiling /home/frank/scala-steward/workspace/repos/ToToTec/CmdOption/scala-steward.sc
Downloading https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloaded https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloading https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Downloaded https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Downloading https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloaded https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
Downloading https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Downloaded https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom.sha1
Failed to resolve ivy dependencies:Error downloading de.tototec:de.tobiasroeser.mill.osgi_2.13:0.2.0
  not found: /home/frank/.ivy2/local/de.tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom
  not found: https://oss.sonatype.org/content/repositories/snapshots/de/tototec/de.tobiasroeser.mill.osgi_2.13/0.2.0/de.tobiasroeser.mill.osgi_2.13-0.2.0.pom

Could it be the case that the mill binary in the PATH also need to be a 0.6 version? Scala Steward is currently using Mill 0.7.3:
$ mill --version
Mill Build Tool version 0.7.3

Yeah, that's that the point of using an older version. It needs to use the older runtime. Specifically, mill changed it's Scala version beween 0.6.x and 0.7.x, so a build for 0.6.x will most probably always fail if it is using additional external plugins, which is the case here.

Oh sorry, I somehow missread your message. I think mill does not invoke any mill binary it found on the PATH. So, it should make no difference what's their binary version.

To summary: If you try to run a build.sc with is intended for mill 0.6.x than that means it uses mill's 0.6 API and Scala 2.12 and  also requires to load mill plugins for this environment. if you now do that with a mill 0.7.x runtime it will most probably fail. In this case, the used mill plugin mill-osgi in version 0.2.0 was never released for mill 0.7.x, hence there is no Scala 2.13 version of it.

I think mill does not invoke any mill binary it found on the PATH

But Scala Steward invokes the mill binary on the PATH to load the build and determine the dependencies.

if you now do that with a mill 0.7.x runtime it will most probably fail.

Ok, so is the mill binary installed by Coursier (which is a 0.7.x version) a mill 0.7.x runtime? I guess Scala Steward then needs to invoke different commands depending on the used mill version?

I'm sorry for not being familiar how the coursier installment of mill works. The idea how mill builds should works is: respect whatever version is given via .mill-version file or MILL_VERSION environment variable, and use a mill runtime with exactly that version. That's also how the mill script bundled with mill behaves. I think the metals developers use a version of lefou/millw to ensure exactly the same behavior.

I think Coursier bundles the latest Mill JAR to an executable (ref) and that does not work for Scala Steward since it only supports one Mill version. I'm now trying out millw as mill executable in the PATH. :crossed_fingers:

It worked with millw: https://github.com/ToToTec/CmdOption/pull/25 :tada:
Thanks @lefou for guiding me through this and thanks again @hamnis for #1501.

Thank you for your great engagement!


fix(deps): update terraform github.com/terraform-ibm-modules/terraform-ibm-landing-zone-vsi to v1.0.3

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/terraform-ibm-landing-zone-vsi
module
patch
v1.0.2 -> v1.0.3

Release Notes

terraform-ibm-modules/terraform-ibm-landing-zone-vsi
v1.0.3
Compare Source
Bug Fixes

add tags to resources (#35) (29e2b18)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.2.2 :tada:
The release is available on:

GitHub release
v1.2.2

Your semantic-release bot :package::rocket:


Warning: No files were found with the provided path for tfplan.binary when using Terragrunt

tfaction version
v1.11.1
Overview
We are encountering an issue while using Terragrunt in combination with OpenTofu.

Terragrunt version: v0.69.3
OpenTofu version: v1.8.7
Both are the latest versions at the time of writing.

In the tfaction workflow, after generating a plan file, actions/upload-artifact is executed to upload the plan file. However, the following warning is displayed during the upload step:
Warning: No files were found with the provided path: ***/tfplan.binary. No artifacts will be uploaded.

How to reproduce
tfaction-root.yaml
・
・
・
  - working_directory: ***
    target: ***
    aws_region: ap-northeast-1
    s3_bucket_name_plan_file: ***
    s3_bucket_name_tfmigrate_history: ***
    terraform_plan_config:
      aws_assume_role_arn: arn:aws:iam::***:role/***-tfaction_terraform_plan
    tfmigrate_plan_config:
      aws_assume_role_arn: arn:aws:iam::***:role/***-tfaction_tfmigrate_plan
    terraform_apply_config:
      aws_assume_role_arn: arn:aws:iam::***:role/***-tfaction_terraform_apply
    tfmigrate_apply_config:
      aws_assume_role_arn: arn:aws:iam::***:role/***-tfaction_tfmigrate_apply
    terraform_command: terragrunt
    env:
      TERRAGRUNT_TFPATH: "tofu"
      TERRAGRUNT_LOG_DISABLE: "true"
・
・
・

tfaction.yaml
{}

GitHub Actions Workflow
・
・
・
      - uses: suzuki-shunsuke/tfaction/plan@50688bc5e4bf266c7db4b841d348eaff199bff95 # v1.11.1
        with:
          github_token: ${{steps.generate_token.outputs.token}}
        env:
          GITHUB_TOKEN: ${{steps.generate_token.outputs.token}} # For GitHub Provider and tfcmt and github-comment
          TF_CLI_ARGS_plan: "-parallelism=30"
          TF_VAR_github_run_id: ${{ github.run_id }}

Expected behaviour
When using the tfaction workflow with Terragrunt, the Terraform plan file (tfplan.binary) should be created in a predictable and accessible directory, allowing the actions/upload-artifact step to successfully locate and upload the file.
Actual behaviour
In the current setup:

The plan file (tfplan.binary) is generated in the .terragrunt-cache directory by default, which is Terragrunt's working directory.
As a result, the actions/upload-artifact step fails with the following warning:

Warning: No files were found with the provided path: ***/tfplan.binary. No artifacts will be uploaded.

Note
Likely Cause

The plan file is generated using the -out=tfplan.binary option, as seen in the following script:

https://github.com/suzuki-shunsuke/tfaction/blob/main/terraform-plan/main.sh#L16

When using Terragrunt, the default behavior is to create the tfplan.binary file under the .terragrunt-cache directory, which is the working directory for Terragrunt.
This results in the actions/upload-artifact step failing to locate the plan file because it is not in the expected directory.

Proposed Solution

As a workaround, the plan file is moved to the Terragrunt directory after execution using an after_hook.
For a more fundamental fix, the plan file could be generated with an explicit path by using the -out=$(pwd)/tfplan.binary option. This ensures the plan file is created in the current working directory instead of the default .terragrunt-cache.

If needed, I would be happy to create a Pull Request to address this issue.

Thank you for reporting the issue.

For a more fundamental fix, the plan file could be generated with an explicit path by using the -out=$(pwd)/tfplan.binary option. This ensures the plan file is created in the current working directory instead of the default .terragrunt-cache.

It looks good to me.
https://github.com/suzuki-shunsuke/tfaction/blob/5cff56e2324da7b44c196876cf428aaaf9f07002/terraform-plan/main.sh#L16

If needed, I would be happy to create a Pull Request to address this issue.

Thank you. Could you create a pull request?

Hmm. I can't reproduce the issue on my laptop (M3 Pro).
$ tofu --version
OpenTofu v1.8.7
on darwin_arm64
+ provider registry.opentofu.org/hashicorp/null v3.2.3

$ terragrunt --version
terragrunt version v0.69.3

main.tf
resource "null_resource" "foo" {}

terragrunt.hcl: empty
$ TERRAGRUNT_LOG_DISABLE=true TERRAGRUNT_TFPATH=tofu terragrunt plan -no-color -detailed-exitcode -out tfplan.binary -input=false

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # null_resource.foo will be created
  + resource "null_resource" "foo" {
      + id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan.binary

To perform exactly these actions, run the following command to apply:
    tofu apply "tfplan.binary"

tfplan.binary was created at the current directory.
$ ls
aqua.yaml  main.tf  terragrunt.hcl  tfaction.yaml  tfplan.binary

I'm not familiar with Terragrunt, so I'm looking into it.

https://terragrunt.gruntwork.io/docs/reference/terragrunt-cache/
https://github.com/gruntwork-io/terragrunt/issues/1601

@suzuki-shunsuke Thank you for looking into this issue. Based on my understanding, here’s a possible way to reproduce the behavior.
It seems that the issue might not be reproducible in your environment because you're not using a module in your setup.
In most use cases with Terragrunt, modules are typically specified in the terraform block's source attribute within the terragrunt.hcl file. When a module is used, Terragrunt creates the .terragrunt-cache directory to store the downloaded module and related files.
Could you try setting up the following structure and configuration to reproduce the issue?
tree

.
├── module
│   └── main.tf
└── terragrunt.hcl

terragrunt.hcl
terraform {
  source = "module"
}

module/main.tf
resource "null_resource" "foo" {}

With this setup, the module directory is used as the source for Terraform. When running Terragrunt with this configuration, you should see that the .terragrunt-cache directory is generated. This is where the tfplan.binary file might be saved by default, which could help reproduce the issue.

Oh, thank you!
I could reproduce the issue on my laptop and CI.
https://github.com/suzuki-shunsuke/test-tfaction/actions/runs/12453570698?pr=1

plan
No files were found with the provided path: terragrunt/tfplan.binary. No artifacts will be uploaded.

And I've confirmed $PWD/tfplan.binary resolves the issue.
TERRAGRUNT_LOG_DISABLE=true TERRAGRUNT_TFPATH=tofu terragrunt plan -no-color -detailed-exitcode -out $PWD/tfplan.binary -input=false

$ ls
aqua.yaml  main.tf  module  terragrunt.hcl  tfaction.yaml  tfplan.binary

Looks good. Thank you.

I'm planning to release v1.13.2 tomorrow morning.

🎉 https://github.com/suzuki-shunsuke/tfaction/releases/tag/v1.13.2

Thank you for the release! I’ve already tried it in my project, and the plan file was successfully uploaded without any issues.


WarningException on System.DateTime when using .NET Native Compilation

Working with Ceras 4.1.7 in an UWP project (targetting build 16299 of Win10).
With .NET Native tool chain compilation disabled, everything works fine.
However, with that option enabled (and I need it for release builds), I'm getting the following WarningException from Ceras:
Ceras.Exceptions.WarningException: Warning: The type 'System.DateTime' is a struct and by default Ceras serializes structs only through their fields. This struct has only auto-properties, for which the compiler generates hidden 'backing-fields' that are all marked as 'CompilerGenerated'.
You can do multiple things to fix this:
(1) Change all properties to fields.
(2) Explicitly include the properties either with the [Include] attribute, or [MemberConfig].
(3) Disable this warning-exception in 'config.Warnings'.
at Ceras.TypeConfig.Seal() + 0x2b8
at Ceras.CerasSerializer.CreatePrimarySchema(Type, Boolean) + 0x90
at Ceras.CerasSerializer.CreateMetaData(Type, Boolean) + 0x184
at Ceras.CerasSerializer.GetReferenceFormatter(Type) + 0x27
at Ceras.Formatters.DynamicFormatter1.GenerateSerializer(CerasSerializer, Schema, Boolean, Boolean) + 0x491    at Ceras.Formatters.DynamicFormatter1.Initialize() + 0x4e
at Ceras.CerasSerializer.PrepareFormatter(IFormatter) + 0x28
at Ceras.CerasSerializer.GetSpecificFormatter(Type, TypeMetaData) + 0x15a
at Ceras.CerasSerializer.GetSpecificFormatter(Type) + 0x2d
at Ceras.Formatters.ReferenceFormatter1.GetOrCreateEntry(Type) + 0xc7    at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x2b7
at Ceras.Formatters.CollectionFormatter2.Deserialize(Byte[], Int32&, TCollection&) + 0x24a    at System.Action3.Invoke(T1, T2, T3) + 0x24
at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372    at _$ILCT$.$ILT$ReflectionDynamicInvoke$.InvokeRetVIRR[T0, T1, T2](Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x74    at System.InvokeUtils.CalliIntrinsics.Call(IntPtr, IntPtr, Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x2f    at System.InvokeUtils.CallDynamicInvokeMethod(Object, IntPtr, Object, IntPtr, IntPtr, Object, Object[], BinderBundle, Boolean, Boolean, Boolean) + 0x10c --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c    at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException) + 0x14    at System.Linq.Expressions.Interpreter.ByRefMethodInfoCallInstruction.Run(InterpretedFrame) + 0x203    at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame) + 0x25    at System.Linq.Expressions.Interpreter.LightLambda.RunVoid(Object[]) + 0x96    at Ceras.Formatters.DeserializeDelegate1.InvokeObjectArrayThunk(Byte[], Int32&, T&) + 0x86
at Ceras.Formatters.DynamicFormatter1.Deserialize(Byte[], Int32&, T&) + 0x24    at System.Action3.Invoke(T1, T2, T3) + 0x24
at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372    at _$ILCT$.$ILT$ReflectionDynamicInvoke$.InvokeRetVIRR[T0, T1, T2](Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x74    at System.InvokeUtils.CalliIntrinsics.Call(IntPtr, IntPtr, Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x2f    at System.InvokeUtils.CallDynamicInvokeMethod(Object, IntPtr, Object, IntPtr, IntPtr, Object, Object[], BinderBundle, Boolean, Boolean, Boolean) + 0x10c --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c    at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException) + 0x14    at System.Linq.Expressions.Interpreter.ByRefMethodInfoCallInstruction.Run(InterpretedFrame) + 0x203    at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame) + 0x25    at System.Linq.Expressions.Interpreter.LightLambda.RunVoid(Object[]) + 0x96    at Ceras.Formatters.DeserializeDelegate1.InvokeObjectArrayThunk(Byte[], Int32&, T&) + 0x86
at Ceras.Formatters.DynamicFormatter1.Deserialize(Byte[], Int32&, T&) + 0x24    at System.Action3.Invoke(T1, T2, T3) + 0x24
at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372    at Ceras.Formatters.CollectionFormatter2.Deserialize(Byte[], Int32&, TCollection&) + 0x24a
at System.Action3.Invoke(T1, T2, T3) + 0x24    at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372
at _$ILCT$.$ILT$ReflectionDynamicInvoke$.InvokeRetVIRR[T0, T1, T2](Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x74
at System.InvokeUtils.CalliIntrinsics.Call(IntPtr, IntPtr, Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x2f
at System.InvokeUtils.CallDynamicInvokeMethod(Object, IntPtr, Object, IntPtr, IntPtr, Object, Object[], BinderBundle, Boolean, Boolean, Boolean) + 0x10c
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException) + 0x14
at System.Linq.Expressions.Interpreter.ByRefMethodInfoCallInstruction.Run(InterpretedFrame) + 0x203
at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame) + 0x25
at System.Linq.Expressions.Interpreter.LightLambda.RunVoid(Object[]) + 0x96
at Ceras.Formatters.DeserializeDelegate1.InvokeObjectArrayThunk(Byte[], Int32&, T&) + 0x86    at Ceras.Formatters.DynamicFormatter1.Deserialize(Byte[], Int32&, T&) + 0x24
at System.Action3.Invoke(T1, T2, T3) + 0x24    at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372
at _$ILCT$.$ILT$ReflectionDynamicInvoke$.InvokeRetVIRR[T0, T1, T2](Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x74
at System.InvokeUtils.CalliIntrinsics.Call(IntPtr, IntPtr, Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x2f
at System.InvokeUtils.CallDynamicInvokeMethod(Object, IntPtr, Object, IntPtr, IntPtr, Object, Object[], BinderBundle, Boolean, Boolean, Boolean) + 0x10c
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException) + 0x14
at System.Linq.Expressions.Interpreter.ByRefMethodInfoCallInstruction.Run(InterpretedFrame) + 0x203
at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame) + 0x25
at System.Linq.Expressions.Interpreter.LightLambda.RunVoid(Object[]) + 0x96
at Ceras.Formatters.DeserializeDelegate1.InvokeObjectArrayThunk(Byte[], Int32&, T&) + 0x86    at Ceras.Formatters.DynamicFormatter1.Deserialize(Byte[], Int32&, T&) + 0x24
at System.Action3.Invoke(T1, T2, T3) + 0x24    at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372
at _$ILCT$.$ILT$ReflectionDynamicInvoke$.InvokeRetVIRR[T0, T1, T2](Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x74
at System.InvokeUtils.CalliIntrinsics.Call(IntPtr, IntPtr, Object, IntPtr, InvokeUtils.ArgSetupState&, Boolean) + 0x2f
at System.InvokeUtils.CallDynamicInvokeMethod(Object, IntPtr, Object, IntPtr, IntPtr, Object, Object[], BinderBundle, Boolean, Boolean, Boolean) + 0x10c
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
at System.Linq.Expressions.Interpreter.ExceptionHelpers.UnwrapAndRethrow(TargetInvocationException) + 0x14
at System.Linq.Expressions.Interpreter.ByRefMethodInfoCallInstruction.Run(InterpretedFrame) + 0x203
at System.Linq.Expressions.Interpreter.Interpreter.Run(InterpretedFrame) + 0x25
at System.Linq.Expressions.Interpreter.LightLambda.RunVoid(Object[]) + 0x96
at Ceras.Formatters.DeserializeDelegate1.InvokeObjectArrayThunk(Byte[], Int32&, T&) + 0x86    at Ceras.Formatters.DynamicFormatter1.Deserialize(Byte[], Int32&, T&) + 0x24
at System.Action3.Invoke(T1, T2, T3) + 0x24    at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[], Int32&, T&) + 0x372
at Ceras.CerasSerializer.Deserialize[T](T&, Byte[], Int32&, Int32) + 0x131
at Ceras.CerasSerializer.DeserializeT + 0x35
at Calidos.Maat.ClientData.ServiceClient.WebApiCerasBroker.DeSerializeT + 0x5d
All my classes are marked with [MemberConfig(TargetMember.AllProperties)]
How can I resolve this?
Thx,
Tom

I was able to work around this issue by marking the DateTime-properies with Exclude, and adding a string-property that wraps the original DateTime prop.
Something like:

[Include()]
public String ValidFromAsString
{
get
{
return this.ValidFrom.ToString("yyyy-MM-dd");
}
set
{
this.ValidFrom = DateTime.ParseExact(value, "yyyy-MM-dd", CultureInfo.InvariantCulture);
}
}

DateTime can't be serialized when using .NET Native?
I have little experience with .NET Native, so I don't know what it is doing to DateTime behind the scenes.
We could bypass the problem by creating a custom formatter for this type, but the same issue will likely pop up with other types.
For now, you can try just including the two relevant props:

DateTimeKind Kind
long Ticks

Like this:
config.ConfigType<DateTime>()
    .ConfigMember(p => p.Kind).Include()
    .ConfigMember(p => p.Ticks).Include();

Definitely let me know if that helps, and if you get a similar problem with other types.
Maybe we can identify a common pattern and automatically apply this sort of inclusion.

I can confirm this works when marking the Kind & Ticks properties with Include as you've suggested.
For now, this has been the only type I've seen this error on.
(We were using JSON (and XML) serialization before Ceras, never had any issues with DateTimes.)
This issue isn't a priority for us anymore, as I'm using the "AsString" variant I described above. This allows me to solve another issue regarding differences in timezones when passing dates between my server and client.

I can confirm this works when marking the Kind & Ticks properties with Include as you've suggested.

Thanks for testing that! 😄 ❤
Could you also try adding the following setting (together with that .Include() workaround)
config.ConfigType<DateTime>()
      .CustomResolver = (c, t) => c.Advanced
          .GetFormatterResolver<DynamicObjectFormatterResolver>()
          .GetFormatter(t);

Does Ceras still round-trip a DateTime value correctly when running that in .net native?
No exceptions? No corrupted or lost data?

We were using JSON (and XML) serialization before Ceras, never had any issues with DateTimes.

So I've looked into all of this some more...
Since the definition for DateTime consists of a single field  https://github.com/microsoft/referencesource/blob/17b97365645da62cf8a49444d979f94a59bbb155/mscorlib/system/datetime.cs#L131-L140
that warning usually doesn't get triggered.
But it seems the UWP libraries are written in a slightly different syntax, so that there's no explicit/manual field definition! So when Ceras "discoveres" the type and creates the TypeConfig for it, that check gets triggered.
Like other serializers, Ceras also uses a Formatter made specifically for DateTime:
https://github.com/rikimaru0345/Ceras/blob/cc24d604c701ee17cc3ff89aba14bbfc7b30197d/src/Ceras/Resolvers/StandardFormatterResolver.cs#L199-L212
I'm hesitant about going with the straight-forward approach (just disabling the check for types that Ceras can handle with a built-in formatter), because the user can always override the formatter.
But then again maybe that's a rare / advanced scenario and someone who changes the formatter for a built-in type is probably aware of the potential issues.
Maybe the easiest solution would be to just make an exception for DateTime, treating it as a special case? :/

Hmm,
To test it this time, I've written a small program. But now the previous workaround (where you mark the Kind and Ticks properties to be included) no longer works!
Ceras.Serialize() is throwing the following exception there:
property must be readable and writable
It is referring to the property "Kind" here. But I suspect it will also throw this error on the property Ticks. I saw in the DateTime class that both only have a getter, and no setter.
No idea why this was working earlier today! (I didn't use a test-project, and just putted the config in our main solution)
If I try your new config, it throws the original WarningException:
Warning: The type 'System.DateTime' is a struct and by default Ceras serializes structs only through their fields. This struct has only auto-properties, for which the compiler generates hidden 'backing-fields' that are all marked as 'CompilerGenerated'. You can do multiple things to fix this:  (1) Change all properties to fields. (2) Explicitly include the properties either with the [Include] attribute, or [MemberConfig]. (3) Disable this warning-exception in 'config.Warnings'.
Trying a config that tries to include the field dateData fails as ConfigField() returns null.
I've published my sample app here: https://github.com/RCTycooner/CerasTryouts
If you run it in Debug/x86, it will use .NET Native compilation. The various scenario's we've tried can be found in the file https://github.com/RCTycooner/CerasTryouts/blob/master/CerasTryouts/CerasTryouts/CerasTryer.cs
Let me know if I can help with anything regarding this. As stated before, our production code will be using the wrapped "AsString" property.

But now the previous workaround (where you mark the Kind and Ticks properties to be included) no longer works!

The resolver part was missing from there.
When you set those two fields to be included that only changed the TypeConfig.
That's why the warning didn't appear again (it is triggered from inside the TypeConfig while it validates itself).
That means it was still using the built-in DateTimeFormatter, which doesn't respect the TypeConfig as you can see from its code (its extremely simple).
Most settings in TypeConfig are only relevant to DynamicObjectFormatter<>. (well, user formatters could use those settings as well in theory)

Ceras.Serialize() is throwing the following exception there:

I see, seems like those two members really are "getter-only properties".

Trying a config that tries to include the field dateData fails as ConfigField() returns null.

In that case a field with that name doesn't exist.
I looked it up:

dateData for net framework
_dateData for .net core
and in .net native it seems to be written as a getter only property.

In any case, the built-in formatter that is used by default should work perfectly on any platform.

Let me know if I can help with anything regarding this.

The only thing left to do is deciding on how to handle this in the future.
Should I word the warning-exception differently?
Or add that little if() special case / exception for DateTime that I mentioned before?
What do you think?

The ideal situation is ofcourse that basic types such as a DateTime work out of the box. So I would add a special condition for DateTimes.


Rendering of generics  as response

Hi!
Using Springfox 2.0.0
Having this rest operation
 @RequestMapping(
            value = "/pop/{referenceId}",
            method = RequestMethod.DELETE,
            produces = ContentTypes.LIST
    )
    @ApiOperation(value = "Pop operation",
    notes = "Pops it")
    public DeferredResult<ResponseEntity<StateList>> pop_1_0(
            @PathVariable(value="referenceId") final String referenceId, @ApiIgnore final HttpEntity<String> request) {
        ...
    }}

Here is part of my config
/**
     * Swagger Spring MVC configuration
     */
    @Bean
    public Docket swaggerSpringMvcPlugin() {
        return new Docket(DocumentationType.SWAGGER_2)
                .genericModelSubstitutes(ResponseEntity.class, DeferredResult.class)
                .select()
                        .paths(paths()) // and by paths
                                .build()
                                .apiInfo(apiInfo());
    }

How would I configure springfox to render StateList as the response type correctly?
Now it renders this to ResponseEntity in swagger-ui and nothing more is shown.
Do I need to add add a alternaterule and if that is the case how would that rule be implemented.
Thanks in advance.

@cwstenberg You would need to add an alternate rule. This answer should point you in the right direction.
While the types are not really the same, the basic idea is that we need to explicitly tell spring fox to replace DeferredResult<ResponseEntity<T>> with T. Where T can be Wildcard (from springfox) meaning replace any message with that shape or StateList if you want that to be narrow.
If you have any questions let us know

Ok thanks!
Resolved it like this.

    @Autowired
    TypeResolver typeResolver;

/**
     * Swagger Spring MVC configuration
     */
    @Bean
    public Docket swaggerSpringMvcPlugin() {
        return new Docket(DocumentationType.SWAGGER_2)
                .alternateTypeRules(newRule(typeResolver.resolve(DeferredResult.class, typeResolver.resolve(ResponseEntity.class, WildcardType.class)),
                        typeResolver.resolve(WildcardType.class)))
                .select()
                        .paths(paths()) // and by paths
                                .build()
                                .apiInfo(apiInfo());
    }

:+1:

is @dilipkrish possible to make the alternateTypeRules reverse from question mentioned by @cwstenberg? ex.
 @Autowired
    TypeResolver typeResolver;

/**
     * Swagger Spring MVC configuration
     */
    @Bean
    public Docket swaggerSpringMvcPlugin() {
        return new Docket(DocumentationType.SWAGGER_2)
                .alternateTypeRules(
                 newRule(typeResolver.resolve(WildcardType.class),
                         typeResolver.resolve(GenericMode.class, WildcardType.class)))
                .select()
                        .paths(paths()) // and by paths
                                .build()
                                .apiInfo(apiInfo());
    }

@emanuelbatista Without really testing I would have to say no.