Add support for Jupyterlab 3

I think we also need this part in the __init__.py file:
https://github.com/jupyterlab/extension-cookiecutter-ts/blob/d8c08687f8661545a65d65ab8eb9433267a45b35/{{cookiecutter.python_name}}/{{cookiecutter.python_name}}/__init__.py#L9-L16

@jtpio Oh ok, what is this for? Isn't JupyterLab able to automatically find the extensions installed under /share/jupyter/labextensions/?

@jtpio let's get this one in first #1223

This seems to work perfectly locally. Let's wait for a JupyterLab 3 before merging this and updating the CI.

Hmm, all of the sudden the directory layout is different (js files are in a static subdir):
(jlab3f) mbp ~/src/bqplot (jlab3 ✗)$ ls share/jupyter/labextensions/bqplot/
package.json static

Yes this has changed in one of the release candidates. The setup.py must be changed:
('share/jupyter/labextensions/bqplot', 'share/jupyter/labextensions/bqplot/', '**'),

ah, that's prettier! let me change that.

$ python setup.py bdist_wheel
copying share/jupyter/labextensions/bqplot/static/536.9b1aaa73e0be13c207d2.js.LICENSE.txt -> build/bdist.macosx-10.9-x86_64/wheel/bqplot-0.12.19.data/data/share/jupyter/labextensions/bqplot/tatic

tatic???? 🤯

I think this PR is orange, we get a:
ImportError: Error importing plugin "pytest_tornasync": No module named 'pytest_tornasync'

Which if I'm not mistaking is https://github.com/jupyter-server/jupyter_server/issues/337
this should already be released (in 1.0.6), while we install 1.0.7 🤷

Ok, GitHub UI shows https://github.com/jupyter-server/jupyter_server/commits/master in a weird order (by some date), that makes it look like https://github.com/jupyter-server/jupyter_server/issues/337 was merged before the 1.0.6 release, but it is not :( It is unreleased

Closes #1245

Closes #1245

@jasongrout How should we proceed concerning the processPhophorMessage method?

@jasongrout How should we proceed concerning the processPhophorMessage method?

@jasongrout How should we proceed concerning the processPhophorMessage method?

Leave it. We left the phosphor->Lumino bridges in jlab 3. The hope is that we release ipywidgets 8 early this year, and that will be the migration to Lumino.

@jasongrout How should we proceed concerning the processPhophorMessage method?

Leave it. We left the phosphor->Lumino bridges in jlab 3. The hope is that we release ipywidgets 8 early this year, and that will be the migration to Lumino.

For some reason, the CSS is not included anymore

For some reason, the CSS is not included anymore

Should the jstargets also include an entry pointing to the prebuilt assets for the lab extension? Similar to:
https://github.com/jupyterlab/extension-cookiecutter-ts/blob/4d0b82636eb72da1f7260536944823c3a3f075d0/{{cookiecutter.python_name}}/setup.py#L25-L27

Should the jstargets also include an entry pointing to the prebuilt assets for the lab extension? Similar to:
https://github.com/jupyterlab/extension-cookiecutter-ts/blob/4d0b82636eb72da1f7260536944823c3a3f075d0/{{cookiecutter.python_name}}/setup.py#L25-L27

This should be good. Trying one last time locally.

This should be good. Trying one last time locally.


emails: Rewrite the day 1 followup email.

The old one was a list of features that by now are pretty standard, so the new one is more of a personal appeal/advertisement.

Automated message from Dropbox CLA bot
@jrowan, it looks like you've already signed the Dropbox CLA. Thanks!

@zulipbot label "area: emails"

@timabbott this is now at https://github.com/rishig/zulip/tree/review-5634
Added in the "Zulip should always be open". In a few more weeks I think we can change this to point directly at the mobile/desktop apps.

Why not make "desktop and mobile apps" a link now?  I think that's strictly better; while we may want to tweak the framing later, it seems like there's no benefit in mentioning them without a link..
One more thing: is it necessary that the plain-text version has super long lines?  I think we probably don't need to do that, though my recollection is that to make line-wrapping work correctly in that case with plain-text emails, you need to put a trailing space at the end of each line, which might require some linter work...

The benefit to mentioning them without a link is that they'll know the apps exist, but still have their early experience with Zulip be via the desktop. I think our current apps make sense once you've used Zulip on the desktop, and know how Zulip is supposed to work. But as a first experience, e.g. not being able to compose to a new topic gives you a completely different perception of what the product is.

long lines

Not in particular. But I expect ~0 people are getting the plain-text emails, so we should do whatever is easiest. Changed to wrap to 76 characters.

Compose to new topic is in the iOS app release happening this week, so that issue is irrelevant for future users.

okay, it's possible it's ready to recommend, then. The other thing is I think unread counts, maybe on Android? I.e. the lack of unread counts means you basically only use Home, but then in a new organization that can mean no stream/topic discipline is ever developed.

Unread counts is in progress, but will take some time.  Regardless, the apps are definitely useful even if imperfect (a lot of people do use them, as is!), and it's just annoying for users to mention the apps without a link.

Updated (and rewritten) to https://github.com/rishig/zulip/tree/review-5634.

We can make keyboard notifications and other features their own email later. (And possibly even an email about zulip being open source / the zulip community.)

Merged, thanks @jrowan and @rishig!
I like the idea of a day3 email on keyboard shortcuts.  The other features feel like things better handled through another part of the tutorial (perhaps duplicated in a longer "getting started with Zulip" article that we probably should finish and link to briefly from this email).


chrome autofill interaction

Chrome autofill interacts poorly with the autocomplete widget. To see this behavior, you need to have autofill enabled in your desktop chrome browser, and using a field-name that chrome interprets as "fillable" like a street address or zipcode. And, you need to have something in your autofill settings. The consequence of this problem is a dropdown that appears above the autocomplete dropdown and prevents you from clicking on the first item amongst the autocomplete options.
I am using Chrome 40.0.2214.111. ngTagsInput v2.0.1. Angular 1.2.16.
A simple solution is to specify
<input autocomplete="off"'

in the tags-input.html template.

Thanks for reporting this! I'll add it to the backlog.


[PE-D][Tester C] Extraneous parameters not detected

edit 1 n/Ale p/92929292 o/notes

Might be good to check for extra parameters, at the least in its current iteration, user can tell where the error might have started!
Also the command text history is pretty interesting :) and helpful for testing

Labels: severity.Low type.FunctionalityBug
original: ghos7ie/ped#14

Just quality of life improvement


Tests for business logic

How?

using puppeteer or something like you mean? 🤔

Unit testing only

duplicate #24


Feature request: convert all fields of the Location object, not just latitude and longitude

Hi,
As can be seen in the log fragment below, LocationUpdatesService receives a full Location object with many fields, but only 'lat' and 'lon' are passed to the Flutter package.
I/flutter (28390): locationUpdate: {lat: 51.508281, lon: -0.1397356, logging_enabled: true}
I/flutter (28390): Location Update: Lat: 51.508281 Lon: -0.1397356
D/BackgroundManager(28390): Got success, destroy engine!
D/LocationUpdatesService(28390): New location: Location[fused 51.508503,-0.139252 hAcc=5 et=+1d9h18m55s853ms alt=72.1 vel=11.020249 bear=39.803856 vAcc=1 sAcc=1 bAcc=30 {Bundle[mParcelledData.dataSize=52]}]

The conversion seems to be happening in the file below and is hard-coded for only lat and lon. Would it be possible to modify it to pass all available Location fields to the Flutter package?
android\src\main\kotlin\com\icapps\background_location_tracker\flutter\FlutterBackgroundManager.kt
private fun handleInitialized(call: MethodCall, result: MethodChannel.Result, ctx: Context, channel: MethodChannel, location: Location, engine: FlutterEngine) {
        val data = mutableMapOf<String, Any>()
        data["lat"] = location.latitude
        data["lon"] = location.longitude
        data["logging_enabled"] = SharedPrefsUtil.isLoggingEnabled(ctx)
        channel.invokeMethod("onLocationUpdate", data, object : MethodChannel.Result {
            override fun success(result: Any?) {
                Logger.debug("BackgroundManager", "Got success, destroy engine!")
                engine.destroy()
            }

            override fun error(errorCode: String?, errorMessage: String?, errorDetails: Any?) {
                Logger.debug("BackgroundManager", "Got error, destroy engine! $errorCode - $errorMessage : $errorDetails")
                engine.destroy()
            }

            override fun notImplemented() {
                Logger.debug("BackgroundManager", "Got not implemented, destroy engine!")
                engine.destroy()
            }
        })
    }

Thank you!

Hi, good question.
Since we try to keep everything cross platform, we are looking into how it effects the IOS side.
I will let you know when/how we decide to implement this.

@slycos thanks for creating this issue. For us this is currently not a high prio. We are accepting pull requests, so if you want to add this for android. and find a good way so iOS is not left out, or it is clear that iOS does not have these values. we will review this pul request.


Reduce the amount of drinks per robust softdrinks machine

About the PR
Screenshots

Changelog

:cl:

tweak: Reduced the number of each drink for Robust Softdrink machines from 10 to 2.

Instant merge lmao


Task2

Seems nice! Please attach a screenshot of the ui to the or :)

Next time attach the screenshot to the PR :)


Estimate/assess the impact of an intervention

This is a core project goal and may not be solely a 2.0 feature, but BTD have indicated that being able to estimate how a proposed intervention might affect a segment's risk would be a valuable feature.
Having historical intervention data available would obviously help - if a user wanted to know what impact installing a stop sign would be, we could search for similarly profiled segments that were treated with the same intervention, and use that segment's change in risk as a guide (controlling for other variables as necessary).
In the absence of historical interventions, I wonder if we could have a "second best" form of estimation, whereby we find a similar segment to the one in question that already has the feature being considered (but date of installation unknown) and compare the risk between the two. This feels a bit blunt though, any other ways we could approach this prior to having a history of interventions?

I think a possibility here is using Public Works data.  This is available for (Boston)[https://data.boston.gov/dataset/public-works-active-work-zones] and likely other cities.

Answering @terryf82 's question: This is where we got to with the intervention effect exploration: d68f919c8408be72784416dba2b7e9b4495eadff.  We're going to pause on this for now because it did not seem particularly compelling after talking to BTD.


Merge261into262

See detailed description in the commit message.

Should the base branch be CB-2.62.0?

you are right, thanks


Fails on Linux/Wayland

./sharpview: symbol lookup error: /lib/x86_64-linux-gnu/libnvidia-vulkan-producer.so: undefined symbol: wlEglCreateSurfaceExport
Ubuntu 22.04.2 LTS, latest update. First time trying Wayland. Works with Xorg.
Will try other drivers.

Seems to be an outdated library problem: https://github.com/NVIDIA/egl-wayland/issues/80

Upgraded NVidia driver from 425 to 430. Segmentation fault running "cube".
Reproduced under gdb.
0x0000555556305eab in wgpu_hal::vulkan::Device::create_swapchain (self=0x5555577c4a50, surface=0x555557164f00, config=0x7fffffffa890, provided_old_swapchain=...)
at src/vulkan/device.rs:606
Unclear whether WGPU or what is at fault.
Backtrace:
cubecrash.txt

I would guess it's a driver problem as it's crashing deep in the driver trying to dl_close something, might be worth trying vkcube?

That's what I thought, too. It's some problem involving Linux, Wayland, Vulkan, and too many NVidia driver versions. Wayland vs. Nvidia is known to be ... difficult. But it's good to have it logged as a Rend3 issue, so that when it blows up for someone else, it's known.


Remove asset caching

Issue: AP-3979
What Changed
Removes the asset caching that was breaking multi-page flows because the cached responses being used were missing critical headers, like content-type. I don't think we're gaining much by caching so let's not.
How to test

Change Type

[ ] maintenance
[ ] documentation
[x] patch
[ ] minor
[ ] major

📦 Published PR as canary version: 0.0.45--canary.43.3d53d71.0

:sparkles: Test out this PR locally via:
npm install<EMAIL_ADDRESS># or
yarn add<EMAIL_ADDRESS>

:rocket: PR was released in v0.0.45 :rocket:


Allow gitmojis

Allow using gitmojis in the commit message description.
Looks like they are detected as part of the scope.
For example, the following commit message generated in VSCode using Conventional Commits extension:
chore: :hammer: Add commitlint as git commit-msg hook

Also add **Conventional Commits** VSCode extension

gives:
commitlint

→ input: "chore: :hammer: Add co..."

Errors:
  ❌ parser: scope must be followed by ': '

Total 1 errors, 0 warnings, 0 other severities

Found it! The bug is not in commitlint itself, but in conventionalcommit/parser
And it is slightly different than I thought.
The parser doesn't work with regex but uses a lexer instead.
When the parser reads a header like type(scope): description and moves from type(scope) to description it uses a greedy take(": ") to read the delimiter.
This take will read any number (and order) of    and : into the current token. and then the parser compares the result to :  .
This fails when the description starts with a : as the take will have read : : already.
I've fixed it in my fork and I'll open a PR.


HDF5 Limit on attribute creation order tracking

[This discussion was taking place over email, but I think I should put it here for easier searching and tracking]
Original Issue:

It looks like I am hitting a limit of 65,536 for creating attributes with attribute creation order tracking enabled.  Is that the limit in HDF5?  If so, is it possible to increase this number by changing a #define, or is it an inherent limit in the data format?
This is happening in a NetCDF-4 format file, so the HDF5 is being created via the NetCDF API:
HDF5-DIAG: Error detected in HDF5 (1.10.7) thread 0:
#000: /scratch/gdsjaar/seacas/TPL/hdf5/hdf5-1.10.7/src/H5A.c line 285 in H5Acreate2(): unable to create attribute
major: Attribute
minor: Unable to initialize object
#001: /scratch/gdsjaar/seacas/TPL/hdf5/hdf5-1.10.7/src/H5Aint.c line 275 in H5A__create(): unable to create attribute in object header
major: Attribute
minor: Unable to insert object
#002: /scratch/gdsjaar/seacas/TPL/hdf5/hdf5-1.10.7/src/H5Oattribute.c line 296 in H5O__attr_create(): attribute creation index can't be incremented
major: Attribute
minor: Unable to increment reference count

HDF5 Response:
According to the section IV.A.2.v. "The Attribute Info Message" in the File Format Spec maximum creation index is 2 bytes, so it is a file format issue. I think we do have an issue since there is no limit now on the number of attributes. We will need to introduce changes to the file format. We need to have a conversation on HDF5 limitations on sizes and how much work it will be.

The issue is that NetCDF-4 uses "attribute creation order" tracking as described in docs/file_format_specification.md
\subsection creation_order Creation Order

The netCDF API maintains the creation order of objects that are
created in the file. The same is not true in HDF5, which maintains the
objects in alphabetical order. Starting in version 1.8 of HDF5, the
ability to maintain creation order was added. This must be explicitly
turned on in the HDF5 data file in several ways.

Each group must have link and attribute creation order set. The
following code (from libsrc4/nc4hdf.c) shows how the netCDF-4 library
sets these when creating a group.

\code
           /* Create group, with link_creation_order set in the group
            * creation property list. */
           if ((gcpl_id = H5Pcreate(H5P_GROUP_CREATE)) < 0)
              return NC_EHDFERR;
           if (H5Pset_link_creation_order(gcpl_id, H5P_CRT_ORDER_TRACKED|H5P_CRT_ORDER_INDEXED) < 0)
              BAIL(NC_EHDFERR);
           if (H5Pset_attr_creation_order(gcpl_id, H5P_CRT_ORDER_TRACKED|H5P_CRT_ORDER_INDEXED) < 0)
              BAIL(NC_EHDFERR);
           if ((grp->hdf_grpid = H5Gcreate2(grp->parent->hdf_grpid, grp->name,
                                            H5P_DEFAULT, gcpl_id, H5P_DEFAULT)) < 0)
              BAIL(NC_EHDFERR);
           if (H5Pclose(gcpl_id) < 0)
              BAIL(NC_EHDFERR);
\endcode

Each dataset in the HDF5 file must be created with a property list for
which the attribute creation order has been set to creation
ordering. The H5Pset_attr_creation_order function is used to set the
creation ordering of attributes of a variable.

The following example code (from libsrc4/nc4hdf.c) shows how the
creation ordering is turned on by the netCDF library.

\code
        /* Turn on creation order tracking. */
        if (H5Pset_attr_creation_order(plistid, H5P_CRT_ORDER_TRACKED|
                                       H5P_CRT_ORDER_INDEXED) < 0)
           BAIL(NC_EHDFERR);
\endcode

I recently hit this issue again (and due to limited long-term memory) spent some time triaging it and then remembered the issue...
I pulsed the THG group again and they are willing to look into increasing the maximum attribute creation index from 16 bits to a larger value, but that probably won't happen until 1.14.0 at the earliest...
Since I had users who needed to write their files now (although writing to netcdf-3 / netcdf-5 format works), and I will probably have users start hitting the limit more frequently as model complexity increases... I decided to experiment.  I removed all calls to H5Pset_attr_creation_order() and was able to "successfully" create the file and it seems to run through subsequent read/write calls with no discernable issues.
Question
What is the disadvantage of writing a netCDF-4 file with the attribute creation order turned off?  Will this cause issues?  If so, what issues and are they problem-specific (i.e., do some applications/uses of netCDF need this turned on and some can function fine without it?)
Proposal
If it is OK for some applications to work with netCDF files with the attribute creation ordering turned off, would it be possible to add a configuration (preferably run-time or less desirably compile-time) to the netCDF library which would disable the setting of the attribute creation ordering.  This would give me a solution now instead of in a year or two (plus the time waiting for associated applications to catch up and be able to handle hdf5-1.14.X format files...)
I can work up a PR for consideration if this seems like a possibility...

Note that it is probably OK to retain the attribute creation order tracking for the groups. The main area I am hitting the limit on is with datasets.

The difference is that when opening an existing dataset, the assigned attribute numbers
would differ from at creation time. It is possible, I suppose that some users do a bad
thing and access attributes by attribute number rather than name when reading a dataset.

This is unfortunately very important. If we don't turn on creation ordering, the varids will change. They will be reordered into alphabetical order.
Plenty of codes out there depend on var 0 being something, var 1 being something else, etc. So reodering the vars will break all kinds of user code and be a major violation of backwards compatibility.

I am not suggesting this as a change to default behavior; I'm asking whether it would be possible to provide an option that the application / library could set to disable the use of the ordering if and only if the application / library knew that it could work correctly with non-deterministially-ordered variables...

I believe the attribute table that keeps track of dimensions may know about varids - I don't know how what you propose can work, but I am certainly open to suggestions...

I have run a few tests on my netCDF library that has the attribute creation order tracking disabled and other than a different ordering for a few attributes, I can see no differences in the files.  All of the vars appear in the exact same ordering in the file as they do with the attribute creation order tracking enabled (which I think makes sense...)
I'm not sure where to look to verify that the attribute table bookkeeping does not get changed, but the files that I am creating seem to be valid and are readable by tools linked with either a "pre-change" netcdf library or a "post-change" library.
If I run "ctest", I get some failures, but they all seem to be related to attribute ordering. I can see that if an application relied on the attributes appearing in a certain order, this change would break them, but in my uses, I always query the attribute by name and not position, so unless internally the library is relying on a specific ordering of the "hidden" attributes, I'm not sure how this would affect my subset of files. [I do agree that this breaks backward-compatibility so must be selectable at run-time and not the default behavior]

See #2056 for a proof-of-concept implementation of what I am proposing.

@edwardhartnett
< This is unfortunately very important. If we don't turn on creation ordering, the varids will change.
I guess I am not understanding how the attribute creation ordering affects the varids.  Could you explain?   In my tests, it looks like the only thing that changes is the order of the attributes themselves.

Do you really have a variable with 2^16 attributes?

No, but I have a file that has more than 2^16 attributes since each variable has a few hidden attributes and the count is global over the file and not local to a particular variable.

Is there a separate counter for variables and for attributes? That would lessen the problem
since it would only affect users who access attributes by attribute number (rather than name).
Not sure how common that is.

OK, I thought you were talking about changing the order of varids.
I agree that changing the order of attids has a far smaller impact.
One way to do this would be as a mode flag for the whole file at nc_create time. Another would be a variable setting. Unfortunately we don't have a mode flag for vars, so this would require a new function nc_def_var_att_ordering() or something.
In either case, I think old versions of netCDF-4 would still be able to work with the file. The user would have to know that attribute ordering is alphabetical not by creation.

Another solution is for netcdf to track creation order something like we track dimension ids.

Good idea!
We would lose a minor backward compatibility. Let's suppose we turn off
HDF5 creation ordering, and rely on our own tracking of creation order, and
we do a new release with that code (4.8.2). In this release we keep track
of attribute creation order in our hidden attribute metadata.
Then, files created by 4.8.2 would be readable to previous versions, but
the attribute order might be different from intended. Because versions of
netcdf-c before 4.8.2 would not be able to interpret the attribute ordering
information we are storing in our metadata hidden attributes.
This could actually be pretty serious. For example, if NOAA upgraded to
4.8.2 all downstream users of NOAA data would see their attribute order
change, until they upgraded to 4.8.2.
For that reason, maybe this should still be something that has to be
explicitly turned on via a mode flag or some var function?
On Sun, Aug 8, 2021 at 2:21 PM Dennis Heimbigner @.***>
wrote:

Another solution is for netcdf to track creation order something like we
track dimension ids.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Unidata/netcdf-c/issues/2054#issuecomment-894850462,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABCSXXAPN7PSCLLX5LS6HDTT33ROPANCNFSM5BUODBMA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

We also need to consider the reverse situation:

library with creation order disabled creates a file
someone else reads the file but their library has creation order enabled.

Will they experience problems?

No I don't think that would be a problem, the read code takes the attributes in whatever order the HDF5 library presents them. Creation order is determined at dataset creation time for the HDF5 file.


Replace XGBoost image for E2E with community hosted

What this PR does / why we need it:
I replaced the image with community hosted one.
follow up: https://github.com/kubeflow/training-operator/pull/1913
Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #
Checklist:

[ ] Docs included if any changes are user facing

/assign @johnugeorge @terrytangyuan

/assign @kuizhiqing

lgtm

lgtm

Could you say /lgtm?


Possibility of moving to YARD-style documentation in README

I find some of the documentation and READMEs in this project relatively difficult to use. For example, in Faker::Lorem:
# Optional arguments: word_count=4, supplemental=false, random_words_to_add=6
# The 'random_words_to_add' argument increases the sentence's word count by a random value within (0..random_words_to_add).
# To specify an exact word count for a sentence, set word_count to the number you want and random_words_to_add equal to 0.
# By default, sentences will have a random number of words within the range (4..10).
Faker::Lorem.sentence #=> "Dolore illum animi et neque accusantium."
Faker::Lorem.sentence(3) #=> "Commodi qui minus deserunt sed vero quia."
Faker::Lorem.sentence(3, true) #=> "Inflammatio denego necessitatibus caelestis autus illum."
Faker::Lorem.sentence(3, false, 4) #=> "Aut voluptatem illum fugit ut sit."
Faker::Lorem.sentence(3, true, 4) #=> "Accusantium tantillus dolorem timor."

There's nothing here specifying what the supplemental parameter does, and there's also no solid indication in the examples. Visiting the code itself isn't any more helpful:
def sentence(word_count = 4, supplemental = false, random_words_to_add = 0)
  words(word_count + rand(random_words_to_add.to_i), supplemental).join(' ').capitalize + locale_period
end

I ended up being able to most easily find the purpose of this parameter on this GitHub issue.
What I would expect in the README:
# Optional arguments: word_count=4, supplemental=false, random_words_to_add=6
# Faker::Lorem.sentence(word_count = 4, supplemental = false, random_words_to_add = 6)
#
# @param [Integer] word_count the number of words to include in the generated sentence.
# @param [Boolean] supplemental whether or not to pull from an additional,  supplemental set of Lorem text.
# @param [Integer] random_words_to_add a number of words between 0 and this value will be added to the generated sentence.
Faker::Lorem.sentence #=> "Dolore illum animi et neque accusantium."
Faker::Lorem.sentence(3) #=> "Commodi qui minus deserunt sed vero quia."
Faker::Lorem.sentence(3, true) #=> "Inflammatio denego necessitatibus caelestis autus illum."
Faker::Lorem.sentence(3, false, 4) #=> "Aut voluptatem illum fugit ut sit."
Faker::Lorem.sentence(3, true, 4) #=> "Accusantium tantillus dolorem timor."
Faker::Lorem.sentence(3, false, 4) #=> "Aut voluptatem illum fugit ut sit."
Faker::Lorem.sentence(3, true, 4) #=> "Accusantium tantillus dolorem timor."

What I would expect in the code:
# Generate a sentence with Lorem Ipsum text.
#
# @param [Integer] word_count the number of words to include in the generated sentence.
# @param [Boolean] supplemental whether or not to pull from an additional,  supplemental set of Lorem text.
# @param [Integer] random_words_to_add a number of words between 0 and this value will be added to the generated sentence.
#
# @return [String] a random sentence generated from Lorem Ipsum text.
def sentence(word_count = 4, supplemental = false, random_words_to_add = 0)
  words(word_count + rand(random_words_to_add.to_i), supplemental).join(' ').capitalize + locale_period
end

I wanted to start a discussion here to see whether or not there's an interest in beginning a larger effort to improve this library's documentation. It is such an excellent source of data that it is a shame to see any of its cool options hidden.
It's also entirely possible that I've missed an additional source of documentation somewhere :)

YARD documentation is supported in RubyMine, and I think VS Code as support for it as well.

I've started to add YARD style docs. an example can be seen here: https://www.rubydoc.info/github/SpyMaster356/faker/382145/Faker/Blockchain/Bitcoin

I really like this. Who's willing to take up the challenge of converting all the docs? :)

I don't think that there necessarily needs to be a conversion effort to move all of the documentation immediately, though that would certainly be the dream :).
I was more envisioning a discussion here about which YARD documentation elements should be required, how/where to go about documenting those in the CONTRIBUTING.md, and then how to go about enforcing those documentation requirements for every new pull request before it is allowed to be merged.

I agree with that, we can ask existing/new pull requests that haven't been merged yet to include YARD doc blocks.
An full example would be:
##
# @return [string] A hex string
#
# @param length [integer] Length of the string
# @param upcase [boolean] Upcase the letters
#
# @example Faker::Example.hex
#   "d6964ad8b71d46a62e5e64cd00b9e476"
#   "955b1fdf9971cda5745604ab081cc61d"
#   "73d3dc6fa0ba67911024d61eb6c1e382"
# @example Faker::Example.hex(length: 10)
#   "170b43193e"
#   "181450a4ae"
#   "d7b0a72da6"
# @example Faker::Example.hex(upcase: true)
#   "4BE0CB215BE62A06306BF21872EA9603"
#   "8A73D1E0BB69E3A5388EA0E340A0B0D9"
#   "F0C078EECFFCA394336F93B54D6CDE24"
#
# @faker.version 1.9.2
def hex(length: 32, upcase: false)

I know that YARD also allows for us to specify our own CSS for the documentation, so we could so some fancy things with that, but i haven't yet delved into that.

A few comments on the proposed format provided by @SpyMaster356 :

@param length [integer] Length of the string should probably be @param length [Integer] the length of the string (note the lower case description and capitalized data type). Making explicit where we wish to depart from the style laid out in the Tags documentation for YARD will be necessary to maintain consistency.

What's the intended use of the @faker.version tag? Just want to make sure it's actually relevant for the people who will be reviewing the documentation. Personally, what version a function was introduced in wouldn't be of interest to me as an end user.

I firmly believe in having a summary statement at the top of all YARD docs, even if that summary statement is repetitive. I would probably have added something to the top of your comment along the lines of "Produces a random string of hexadecimal characters.". I find that it makes the end documentation even more consistent and easier to parse.

Without summary:

With summary:

I didn't know there was a suggested format for tags. I think we should adhere to that as much as possible . However, when i was poking around with it, i was running into the issue where [String] was resolving to Faker::String instead of standard lib's String. Do you know if there is a way to force Yard to pick the standard lib one?

In the current markdown docs, there is often a line "Available since version 1.9.0" or similar, and i think it would be good to keep that for users that have locked down their version of faker.

Nearly all the methods are going to be single line "produce a random this", "produce a random that", so i don't know if a summary is completely needed. We could drop the description on the @return tag to remove the duplication.

I'm not sure about the String resolution issue off the top of my head, but maybe prefixing it like ::String would pull it out of the Faker namespace.
Everything else makes perfect sense to me and I agree with your approach, for whatever that's worth to the group :)

So an updated example:
##
# Produces a random hexadecimal string
#
# @return [string]
#
# @param length [integer] Length of the string
# @param upcase [boolean] Upcase the letters
#
# @example Faker::Example.hex
#   "d6964ad8b71d46a62e5e64cd00b9e476"
#   "955b1fdf9971cda5745604ab081cc61d"
#   "73d3dc6fa0ba67911024d61eb6c1e382"
# @example Faker::Example.hex(length: 10)
#   "170b43193e"
#   "181450a4ae"
#   "d7b0a72da6"
# @example Faker::Example.hex(upcase: true)
#   "4BE0CB215BE62A06306BF21872EA9603"
#   "8A73D1E0BB69E3A5388EA0E340A0B0D9"
#   "F0C078EECFFCA394336F93B54D6CDE24"
#
# @faker.version next
def hex(length: 32, upcase: false)

One thing that we should discuss is the number of examples that are required.
A few ideas for that:

Require 1 example for each usage
Require 3 examples for each usage (unless it's always the same result, eg Faker::Placeholdit.image)
Require 3 examples for the primary usage, and 1 for each of the secondary usages

I'm fine with requiring just one example and encouraging examples for optional arguments.

I don't think anything more than 1 example showing the main use case should be required (of course, the more the better). The @param and @return tags should be detailed enough to inform users about the various uses on their own, and is where the focus should be placed during documentation review.

For me, I think each usage should have at least one example, as that can speak more about what will be generated. But generally @param and the summary should be enough in most cases.

Oh cool, Rubocop has a documentation cop, we could make use of that once all the methods are documented.

When utilizing YARD for documentation, I've found the yard stats command that comes with the gem to be very helpful as well. You can use some simple command line tools to ensure that your documentation percentage doesn't drop.
It seems like we've got enough of a start here to open up a pull request. Does it make sense to include these guidelines in CONTRIBUTING.md, or is it worth having something like a DOCUMENTATION.md document that can be linked from CONTRIBUTING.md?

To me, having the new documentation style as part of CONTRIBUTING.md makes the most sense to me. And looking at the file, it actually doesn't mention documentation at all >.<
Adding the check to ensure that the documentation level doesn't drop (btw, currently sits at 7% excluding lib/faker/deprecate) would be good to have from the start.
Sounds good to you @stympy?

That works for me.

@stephendolan i've already started a bit on my fork here: https://github.com/SpyMaster356/faker/tree/yard-powered-docs

Just so I'm clear @SpyMaster356 , are you submitting the initial pull request, or should I?

We could work off of my fork as it's already got Yard added, and some basic setup done. Up to you, as you opened this thread.
On another note, using ::String for the type didn't fix the name-spacing issue, and it looks like even YARD has a similar issue: https://www.rubydoc.info/gems/yard/String

Should i open the pull request @stephendolan?

I'm happy to open one, @SpyMaster356 . It will just be the week of March 8th before I can devote the time to getting it done correctly. If we'd like to see progress before then, someone else will have to open the PR.
We've got a big release at work this week, followed by a big vacation for me :D.

Have fun on the vacation. I'll open a PR for the stuff i have on my fork.


Context menu in SQL Editor doesn't respect cursor position on which it is called

System information:

Operating system (distribution) and version Win10
DBeaver version 22.0.3

Steps to reproduce, if exist:

Type two different queries in SQL Editor (e.g. two selects from different table)
Left-click on one of the queries to put a text cursor on it
Right-click on another query and pick Export from query in the context menu
Perform export

Expected: Exported data is fetched from the query on which the context menu was called
Actual: Data is fetched from the query that had the caret prior to the right-click

Hi, @Matvey16 . I'd like to work on this issue!

@Matvey16 I'd like to work on this issue! But I still have some questions that need your clarification.
To demonstrate, for example, there are two lines of code. I put a text cursor in the first line(regardless of inserted between letters or selecting letters), and I right-click on the second line:

In some cases using other editors, like coding python in PyCharm, it will first place the text cursor to the second line, and the context menu will always be from wherever the text cursor is,  which is the second line in this case.

In other cases, like using Jupyter Notebook in PyCharm, it does not re-place the text cursor. The text cursor will still be in the first line, and the context menu will still regard to the first line, regardless of where I right clicked.

I think either way is can be chosen by developers intentionally, as both have advantages and disadvantages. To me, it seems like DBeaver has chosen the second way.
I'd like to work on it to adopt the first way above, but I still need your confirmation if this is really considered a bug.


Move workflow archival from ReqMgr2 CherryPy to MSRuleCleaner

Impact of the new feature
MicroServices
Is your feature request related to a problem? Please describe.
Once we have the MSRuleCleaner developed It will be the service to take care  of the final workflow status transition to {normal, aborted, rejected}-archived, which means we will need to remove the logic from the CherryPy thread which is currently dealing with that. The place where the change should happen is here  [1].
[1]
https://github.com/dmwm/WMCore/blob/5961d2229b1e548e58259c06af154f33bce36c68/src/python/WMCore/ReqMgr/CherryPyThreads/StatusChangeTasks.py#L101
Describe the solution you'd like
Let MSRuleCleaner take care of this status transition. and remove the code from StatusChangeTasks.py
Describe alternatives you've considered
No allternative
Additional context
No

Hello @todor-ivanov @amaltaro I see that this issue is closed, but I have something to add here:
In Unified, we have this issue to address the output cleanup for aborted/rejected workflows. Last week in P&R Dev meeting, we thought that this WMCore issue could be the correct place to address this feature. What do you think about it?

@haozturk Hasan, I think that Unified issue can be closed then. MSRuleCleaner has already been deployed in production and it's deleting all the rucio rules found for the output datasets, made under the rucio account used by WMAgents wma_prod.
It does not use what you called as "rucio erase" though, MSRuleCleaner simply deletes a given rule id from Rucio.

@amaltaro thanks for your comment. One question to be on the same page: Does MSRuleCleaner invalidate outputs at DBS and deletes the rules at Rucio for workflows that are rejected/aborted in a state before the archival? So, for instance, does it cover the case where the workflow is rejected after the announced state?

Hi @haozturk, the immediate answer is No. I know we had the long discussion in the P&R dev meeting a week ago about this use case and the eventual race condition that we may face in case we want to invalidate also datasets which already have rules on tape, but we somehow missed to reflect that at least with a comment in the current issue (it is my fault I would say).  But anyway, this one just went through and is already in testbed. So we should consider a dedicated issue just for those workflows which meet the conditions:

They follow the path announced -> rejected
They already have rules for datasets on tape which needs to be invalidated

I would say even though we missed to put that exact check in the current version of the service, we are still safe since the path announced -> rejected itself is one that should be triggered by a manual intervention and it should not be that often, hence the person who is triggering it could also (for the time being) make the proper check for rules on tape.
Upon a second thought, we may even have an issue which is covering also this use case and is the next one to work on [1].
Maybe just a comment there giving some details will be enough too.
[1]
https://github.com/dmwm/WMCore/issues/10027


How Embed Videos with Fluid Player width: 100% height: 100% ?

in my phone HUAWEI, Does not open in full screen!?

some one can help me?

Hi @LeoLuanElezi, the logo and logoPosition can be set in the Fluid Player options parameter. See below for an example on how to set this:
<script type="text/javascript">
    var testVideo = fluidPlayer(
        'my-video',
        'http://example.com/vast.xml',
        {
            logo: 'exampleImage.jpg',
            logoPosition: 'top right'
        }
    );
</script>

Note that the default logoPosition will be top left, and so it is not required to set this property.
The full list of the optional Fluid Player parameters can be found at the following - https://github.com/fluid-player/fluid-player#parameters
I hope this information helps, and if you have any further questions do not hesitate to ask. Thank you very much.

Thanks 👍


Add Feature to move and modify elements on Sketch Page

Sample Snapshot

Description

The title explains the issue well!

I didn't understand what this issue is about, can you explain me a little bit because I want to work on this issue, but I'm not getting what exactly you want me to do

Basically once a figure/shape has been drawn on the canvas, we should be able to change it's dimensions as well it's position

hey @smaranjitghose i would like to work on the issue . can i get assigned?


Creates the crawler for the Brazil Central Bank Disqualified Persons List

This PR is related to the issue https://github.com/opensanctions/crawlers/issues/15

Thanks! keep description - it's a nice explanatory touch
On Mon, 22 Jan 2024 at 16:29, Bruno Görresen Mello @.***>
wrote:

@.**** commented on this pull request.
In datasets/br/bcb_disqualified_persons/crawler.py
https://github.com/opensanctions/opensanctions/pull/519#discussion_r1462111482
:

sanction = h.make_sanction(context, entity)

sanction.add(

   "program", "Brazil's Central Bank General Register of Disqualified Persons"

)
sanction.add("authority", "Brazil's Central Bank")

The ID of the process

sanction.add(

   "description",

   "Administrative Sanctioning Process Number: {}".format(input_dict.pop("PAS")),

)

I'll add that, should I keep the description? I added it because I
couldn't find where to put the PAS, but now it might be irrelevant
—
Reply to this email directly, view it on GitHub
https://github.com/opensanctions/opensanctions/pull/519#discussion_r1462111482,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABZSGLPHDP26TTXI6MAUMDYP2HVRAVCNFSM6AAAAABB4UAMDCVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQMZWHA3TMNRWGM
.
You are receiving this because you commented.Message ID:
@.***>

I think I resolved all changes @jbothma


Allow ServerURL to be set through environment variables

We use mkdocs-kroki-plugin to render techdocs in our organization's Backstage.
We want to enforce ServerURL while building our docs in the techdocs builder, rather than having users set the ServerURL in their kroki: config.

We don't want to risk a user forgetting to set this and sending diagrams to kroki.io
We'd like to set different ServerURL per environment

Alternatively you can also use environment variables via mkdocs config:
  - kroki:
      ServerURL: !ENV KROKI_SERVER_URL


Add ability to run inference directly on OEMol

Add ability to run inference directly on OEMol for the StructuralInference class. Once the new docking stuff is merged we can also add a method to predict from DockingResult object if we want. Closes #253

Codecov Report

Merging #528 (8d10b3d) into main (d57c4d3) will increase coverage by 0.07%.
The diff coverage is 100.00%.

Additional details and impacted files

oh yep good point, will do

@hmacdope should be good to go once CI finishes


WSO2 App Server Utils tests and refactored project.

Associated JIRAs
https://wso2.org/jira/browse/WSAS-2208
https://wso2.org/jira/browse/WSAS-2204

We don't need it now, it was used earlier for accessing Tomcat listener methods in integration tests. Removed in last commit.


set parameter stop in repl removes other stop words

if i am in the repl and I type /set parameter stop <|system> all other stop words are removed. I just wanted to add one.

Hey Matt! The other /set commands are "replace" behaviour rather than append so I think this may be the expected behaviour for now. Leaving this open as a possible enhancement.

Yeah, it's not ideal, but I couldn't think of a way around this. How would you remove a different parameter otherwise?

Will close this for now


feat(detector-aws): add DetectorSync implementation for beanstalk detector

Context
Another PR moving away from the deprecated Detector interface in favour to the DetectorSync interface. This time is for AWS Beanstalk detector and it might help a bit in https://github.com/open-telemetry/opentelemetry-js/issues/3582
Not doing all detectors at once:

detectors are independent from each other
smaller PRs for faster reviews

Short description of the changes

add AwsBeanstalkDetectorSync class and instance
make AwsBeanstalkDetector reuse the new sync instance
update tests

Note: since this package is marked as stable (major version is 1) we are doing a non breaking change by adding the new class and mark the current as deprecated. The async detector changes behavior, it does not throw anymore, and it was discussed in last SIG meeting (2024-07-31). The outcome was we're okay with that.
A couple of PRs will follow:

one to suppress internal tracing.
one to add the sync detector to auto-instrumentations package.

closed in favour of #2376 following this suggestion


Checking for already registered collections

Closes #81

Codecov Report

Merging #82 into master will increase coverage by 0.04%.
The diff coverage is 83.33%.

@@             Coverage Diff              @@
##             master      #82      +/-   ##
============================================
+ Coverage     74.49%   74.53%   +0.04%
- Complexity      660      666       +6
============================================
  Files            86       86
  Lines          1988     1999      +11
  Branches        198      199       +1
============================================
+ Hits           1481     1490       +9
- Misses          366      367       +1
- Partials        141      142       +1

Impacted Files
Coverage Δ
Complexity Δ

.../java/org/smof/collection/CollectionsPoolImpl.java
66.66% <100%> (+2.38%)
14 <2> (+2)
:arrow_up:

...rc/main/java/org/smof/exception/SmofException.java
100% <100%> (ø)
2 <1> (+1)
:arrow_up:

...n/java/org/smof/collection/SmofDispatcherImpl.java
84.9% <100%> (+0.59%)
21 <2> (+2)
:arrow_up:

src/main/java/org/smof/collection/Smof.java
75.29% <50%> (-1.25%)
22 <2> (+1)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4120b67...f8a192d. Read the comment docs.


General Discussions

I'm creating this issue as a catch all for random conversations.

Hi @msphn,
I saw your issue #59 on the parent fork. I recently revamped my fork which is incompatible with the parent but I implemented many resources including refs/branches. Just a PSA.
Cheers,
David

Hi @msphn,
I saw your issue 59 on the parent fork. I recently revamped my fork which is incompatible with the parent but I implemented many resources including refs/branches. Just a PSA.
Cheers,
David

Hi @charl,
I noticed your issue https://github.com/ktrysmt/go-bitbucket/issues/63 and wanted to mention to you that my fork, which is completely incompatible with upstream, has querying/filtering: https://github.com/davidji99/go-bitbucket#query-parameters
Hopefully this helps you. If not, ignore me =)
Thanks.


Bad Requests always with POST/PUT method.

Not matter what I tried. I'm receiving for every resource that I have in my application that POST/PUT method fail. GET method works fine.
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json
Content-Length: 53
Server: Werkzeug/0.10.1 Python/2.7.8
Date: Sat, 11 Jul 2015 21:11:07 GMT

{
    "message": "Bad Request",
    "status": 400
}

I prepared the following environment for testing and for frustration.

server: test.py

from flask import Flask, Blueprint
from flask.ext.script import Manager, Shell

app = Flask(__name__)

from apitest import api_bp
app.register_blueprint(api_bp)

manager = Manager(app)

if __name__ == '__main__':
    manager.run()

resource: apitest.py

from flask import Blueprint
from flask.ext.restful import Api, Resource

api_bp = Blueprint('api', __name__)
api = Api(api_bp)

class Test(Resource):

    def get(self):
        print 'GETTTTT'
        return

    def put(self):
        print 'PUTTTT'
        return

    def post(self):
        print 'POSTTT'
        return

api.add_resource(Test, '/test')

As like in the documentation, the code above works perfectly:
curl -i http://<IP_ADDRESS>:5000/test -XPOST -d  ""
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 4
Server: Werkzeug/0.10.1 Python/2.7.8
Date: Sat, 11 Jul 2015 21:22:51 GMT

null%

But If I put the same code in my app in app/api/v1/__init__.py it always fails, not matters what I've tried, POST had never worked. In the console, by example, always print GETT with GET but with POST. The console never shows POSTTT from print.
My last trie was to put the modify of Resource class in the code for see what's happening. And, I added a print line with something in the method dispatch_request from Resource class. So after this. I'm sure that when I request the respective POST to Test, the method dispatch_request never execute. For PUThas the same behavior. But withGET` always works fine.
I don't know how to proceed next, some help, I'll appreciate it so much.  :weary: :gun:

I have the feeling your production code is getting bad JSON (I've run into this issue when I type it out by hand).

Yes, that is my initial guess as well. An empty string is not valid JSON and will raise 400 Bad Request.

No. Guys. I found the origin of the problem, but not the solution yet. I wake up this morning with the feeling was about some in the middle between  and I came up to review the initialization of the app. The error was caused by:

from flask import Flask
...
from flask_wtf.csrf import CsrfProtect
...
csrf = CsrfProtect() #1
...

def create_app(config_name):
    app = Flask(__name__)
    ...
  csrf.init_app(app) #2

If I comment this line where csrf appears. Then, the Bad Request disappear and after that, all become normal.

Thanks for the update! I think your explanation here should be sufficient documentation wise.


Better error-handling for create/init commands  (node < v18)

Love this project! Small [maybe] bug report.
In node 16.16.0, npx partykit@latest init and npm create partykit@latest seem to generally work, but they throw an error.
This makes sense in light of fetch being added as experimental in v18.
npx partykit@latest init

Text of error...
Could not fetch latest versions of partykit and partysocket, defaulting to *

https://github.com/partykit/partykit/blob/e525f2480e01aa37e5902ba3f8d8299d08f0add8/packages/partykit/src/cli.tsx#L145
npm create partykit@latest

Text of error...
Could not fetch latest versions of partykit and partysocket, defaulting to *
ReferenceError: fetch is not defined
    at null.init (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/create-partykit/src/index.tsx:166:7)
    at _Command.<anonymous> (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/create-partykit/src/index.tsx:529:11)
    at _Command.listener (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/commander/lib/command.js:482:17)
    at null.<anonymous> (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/commander/lib/command.js:1300:65)
    at _Command._chainOrCall (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/commander/lib/command.js:1197:12)
    at _Command._parseCommand (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/commander/lib/command.js:1300:27)
    at _Command.parse (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/commander/lib/command.js:909:10)
    at null.<anonymous> (/Users/patcon/.npm/_npx/cadd21533ff9f374/node_modules/create-partykit/src/index.tsx:540:9)
Where should we create your project?

Proposed Solution
Better error-handling when fetch isn't available, or more explicitly saying that anything below some node version is unsupported and may display errors.

fixed node 16 support in https://github.com/partykit/partykit/pull/410
(but we'll probably deprecate node 16 support soon, with good error messages)


MAIN Improve DatetimeEncoder

References
Addresses issues raised in https://github.com/skrub-data/skrub/pull/743 and detailed in https://github.com/skrub-data/skrub/issues/768.
What changes?

DatetimeEncoder exposes format_per_column_, which maps datetime parsable columns (seen during fit) to their first non-null entry. We aim to use this column selection in the TableVectorizer to unify the datetime column selection logic.
We now keep constant features, as removing them dynamically changes the shape of the output. This previous behavior might break pipelines requiring a specific shape and is surprising for the user.
On the other hand, we detect date columns (in opposition to datetime columns), e.g. formatted like "2023-01-01", and we only extract features up to the "day" level for those.
extract_until = None doesn't create an extra column "total_second". It simply doesn't extract features among {"year", ..., "nanoseconds"}.
The parameter add_total_second creates the "total_second" feature. The default is True.
The parameter errors either raises errors (errors="raise") or outputs NaT (errors="coerce") when non-datetime-parsable values are given during transform. The default is "coerce".

This PR required additional tests.
cc @LeoGrin @jeromedockes @GaelVaroquaux

in a later PR the TableVectorizer will start using this module's is_datetime_parsable to identify datetime columns? Note there is some logic in the TableVectorizer to handle month first vs day first that may need to be moved into the DateTimeEncoder module

I wonder if we could have a transformer that just parses string columns that contain dates and replaces them with columns with datetime64 dtype? This is one thing that is annoying to do in pandas if we don't know in advance which columns are dates
import pandas as pd

df = pd.DataFrame({"date": ["2023-10-06", "2023-10-07"]})
df["date"] = pd.to_datetime(df["date"]) # avoid having to do this

his is one thing that is annoying to do in pandas if we don't know in advance which columns are dates

I like this suggestion a lot, this is indeed very annoying. Does it need to be a transformer? I feel we could use a "switch-bait" function in the sense of fuzzy_join, like skrub.to_datetime().  WDYT? cc @GaelVaroquaux

@jeromedockes I went for the switch-bait to_datetime with a rework on the backend side to decouple DatetimeEncoder from the column selection, the format guessing, and the parsing logic. LMKWYT :)

Thanks, it looks great! I wonder if we should make it for dataframes only (and possibly 2d arrays), and maybe give it a different names because not all columns are converted. I get your "switchbait" point but it may make the function more complex, 1d or less inputs are well handled by polars and pandas functions, and the skrub use case is DataFrames

Thanks! I get your idea, but we might end up with different behavior for series (pd.to_datetime) vs dataframes (skrub.to_datetime). For consistency, I'd advocate for handling both 1d and 2d outputs, so that the user can always rely on skrub.to_datetime, WDYT? I also simplified the logic for series, to be the same as dataframes.

I found a couple of issues, anything related to datetimes, timezones, and the interactions between standard library datetimes, pandas and polars is always a bit tricky :D

Thanks for pointing out the issues! I haven't tested it extensively with Polars yet.

The CI errors with the old Pandas version. This is linked to the missing "format=mixing" argument, recently introduced in https://github.com/pandas-dev/pandas/pull/50939.

Thanks! I get your idea, but we might end up with different behavior for series (pd.to_datetime) vs dataframes (skrub.to_datetime). For consistency, I'd advocate for handling both 1d and 2d outputs, so that the user can always rely on skrub.to_datetime, WDYT? I also simplified the logic for series, to be the same as dataframes.

yes that makes sense. skrub.to_datetime will be different than pd.to_datetime and handle only a subset of usecases, so we should be careful to document that and about how we handle parameters that don't apply (such as "unit")
for example pandas can convert to datetime a dataframe with separate columns for year, month, etc. whereas skrub looks at each column separately
import pandas as pd
from skrub import to_datetime

df = pd.DataFrame({"year": [2023, 2024], "month": [1, 2], "day": [12, 13]})
print("pandas:")
dt = pd.to_datetime(df)
print(dt)
print("""
------------
skrub:""")
sdt = to_datetime(df)
print(sdt)

pandas:
0   2023-01-12
1   2024-02-13
dtype: datetime64[ns]

------------
skrub:
   year  month  day
0  2023      1   12
1  2024      2   13

also I like your choice to handle series basically as dataframes with one column and I wonder if we should have this harmonization for arrays and scalars too?
import pandas as pd
from skrub import to_datetime

s = pd.Series(["a", "b"])
print(to_datetime(s))
print("""
-----------------
""")
print(to_datetime(s.values))

0    a
1    b
dtype: object

-----------------

/home/jerome/workspace/backedup_repositories/skrub/skrub/_datetime_encoder.py:85: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  return pd.to_datetime(X, **kwargs)
DatetimeIndex(['NaT', 'NaT'], dtype='datetime64[ns]', freq=None)

We also have doctest errors everywhere. It seems very picky.

yes doctests compares the output as strings, so from this point of view "2022" and "2.022e+03" are not the same. I'll look into setting numpy and pandas formatting options so we get more sensible outputs

yes that makes sense. skrub.to_datetime will be different than pd.to_datetime and handle only a subset of usecases, so we should be careful to document that and about how we handle parameters that don't apply (such as "unit")

Yes, there are efforts to be made on the documentation.

for example pandas can convert to datetime a dataframe with separate columns for year, month, etc. whereas skrub looks at each column separately

I wasn't even aware of this feature 😅 but I got your point

also I like your choice to handle series basically as dataframes with one column and I wonder if we should have this harmonization for arrays and scalars too?

Maybe we should, to bring consistency in skrub even further?

We may need to discuss it a bit more before setting options for the whole
codebase, because the numpy default is actually quite good in most cases. It
looks weird in this particular examples because the array mixes numbers like
2022 which should be formatted with a decimal notation, and very large numbers
which should be formatted with a scientific notation.
so maybe for now you can set the printoptions just for this module
diff --git a/pyproject.toml b/pyproject.toml
index 3afcafd3..3a182a71 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -39,3 +39,4 @@ filterwarnings = [
     'ignore:elementwise\ comparison\ failed:FutureWarning',
 ]
 addopts = "--doctest-modules"
+doctest_optionflags = "NORMALIZE_WHITESPACE ELLIPSIS"
diff --git a/skrub/_datetime_encoder.py b/skrub/_datetime_encoder.py
index a94c89e1..9e530e07 100644
--- a/skrub/_datetime_encoder.py
+++ b/skrub/_datetime_encoder.py
@@ -1,3 +1,4 @@
+""">>> np.set_printoptions(formatter={"float": "{:g}".format})"""
 import warnings
 from collections import defaultdict
 from typing import Iterable
@@ -445,10 +446,10 @@ class DatetimeEncoder(TransformerMixin, BaseEstimator):
     with five columns ("year", "month", "day", "hour" and "total_seconds"):

     >>> enc.transform(X)
-    array([[2022.,   10.,   15.,    0.,    1.6657920e+09],
-           [2021.,   12.,   25.,    0.,    1.6403904e+09],
-           [2020.,    5.,   18.,    0.,    1.5897600e+09],
-           [2019.,   10.,   15.,   12.,    1.5711408e+09]])
+    array([[2022,   10,   15,    0,    1.66579e+09],
+           [2021,   12,   25,    0,    1.64039e+09],
+           [2020,    5,   18,    0,    1.58976e+09],
+           [2019,   10,   15,   12,    1.57114e+09]])
     """

     def __init__(

note the added line in the pyproject.toml -- I can add this one in a small separate PR if you think it makes more sense
The set_printoptions can be done in the module's docstring if you want to hide it from readers of the classe's docstring, or in the classe's docstring if you want to show it. Note as this is a private module we don't mind stuffing it in there as no one reads it.
you can use any other format you prefer, here are a couple more options

'{}':
[2022.0 166579456.0]

'{:g}':
[2022 1.66579e+08]

'{:#.0f}':
[2022. 166579456.]

@jeromedockes can you elaborate on the choices of flag? I fail to see how NORMALIZE_WHITESPACE or ELIPSIS would help.

@jeromedockes can you elaborate on the choices of flag? I fail to see how NORMALIZE_WHITESPACE or ELIPSIS would help.

NORMALIZE_WHITESPACE is because with the numpy.set_printoptions call, the printed numpy array will not have its columns aligned anymore. this flag makes differences in whitespace unimportant so it allows you to display in the doctest output a "prettier" array where you use the custom formatting but still add whitespace so columns line up. another option would be to use alignment options in the format eg "{: >4g}" but the NORMALIZE_WHITESPACE is probably good to have anyway
the ELLIPSIS is not actually needed and is actually the default but is just in the direction of your point that doctest seems picky
I opened a separate PR for the doctest options: #794

Also, actually my suggestion to use set_printoptions is not a good one because  it might affect subsequent tests. So we would have to use the context manager:
diff --git a/pyproject.toml b/pyproject.toml
index 3afcafd3..3a182a71 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -39,3 +39,4 @@ filterwarnings = [
     'ignore:elementwise\ comparison\ failed:FutureWarning',
 ]
 addopts = "--doctest-modules"
+doctest_optionflags = "NORMALIZE_WHITESPACE ELLIPSIS"
diff --git a/skrub/_datetime_encoder.py b/skrub/_datetime_encoder.py
index a94c89e1..03a83408 100644
--- a/skrub/_datetime_encoder.py
+++ b/skrub/_datetime_encoder.py
@@ -444,11 +444,12 @@ class DatetimeEncoder(TransformerMixin, BaseEstimator):
     The encoder will output a transformed array
     with five columns ("year", "month", "day", "hour" and "total_seconds"):

-    >>> enc.transform(X)
-    array([[2022.,   10.,   15.,    0.,    1.6657920e+09],
-           [2021.,   12.,   25.,    0.,    1.6403904e+09],
-           [2020.,    5.,   18.,    0.,    1.5897600e+09],
-           [2019.,   10.,   15.,   12.,    1.5711408e+09]])
+    >>> with np.printoptions(formatter={"float": "{:g}".format}):
+    ...     enc.transform(X)
+    array([[2022,   10,   15,    0,    1.66579e+09],
+           [2021,   12,   25,    0,    1.64039e+09],
+           [2020,    5,   18,    0,    1.58976e+09],
+           [2019,   10,   15,   12,    1.57114e+09]])
     """

     def __init__(

but that adds noise to the example :/

another possible compromise would be to use total_seconds=False in the example

another possible compromise would be to use total_seconds=False in the example

This feels like a good compromise 😄

Remaining questions:

How should we handle timestamps? By default, passing a regular float timestamp to pandas will result in an incorrect result unless specifying unit:>>> pd.to_datetime(1697471201.980721)  # (equivalent to setting unit="ns")
Timestamp('1970-01-01 00:00:01.697471201')

>>> pd.to_datetime(1697471201.980721, unit="s")
Timestamp('2023-10-16 15:47:12')

This is also quite challenging because it involves parsing float or int as timestamps.

How should we handle timestamps? By default, passing a regular float timestamp to pandas will result in an incorrect result unless specifying unit:

It seems really error-prone to try to parse these automatically, and it seems okay to parse actual timestamps as numerical features for most use cases, while the opposite is really annoying and surprising. So I would let users handle it on their own. Another option would be to accept a list of columns which should be parsed as dates, but I don't think the benefit would be worth the cost.

It seems really error-prone to try to parse these automatically, and it seems okay to parse actual timestamps as numerical features for most use cases, while the opposite is really annoying and surprising. So I would let users handle it on their own. Another option would be to accept a list of columns which should be parsed as dates, but I don't think the benefit would be worth the cost.

Yes, and I'm also thinking about the case when we would run the DatetimeEncoder multiple times on the same dataset and end-up transforming the total_seconds column each time, which would be extremely confusing.

@jeromedockes I think we lost the codecov report in the CI?

yes I guess that must have been introduced in #2615 and we missed it because codecov upload still  exits with status 0 even if it fails to upload anything. I'll try to fix it in #797

For the example, the prediction plot looks a bit weird with the total_seconds features added, I'm not sure why, but it seems a bit nicer with a cutoff on 06-05, I'll let you decide if you want to change it.

Thanks @LeoGrin. I realize I've neglected the example in my revamp of DatetimeEncoder; I need to focus on it. Interestingly, notice that adding total_seconds has improved the NMSE:
from:
array([-120.29551054, -192.65273375, -170.69606296, -225.30467065,
       -214.21981371])

to:
array([-108.89923208, -189.50744903, -181.15543923, -143.86698948,
       -207.90131453])

Also, total_seconds is now the most important feature.

I have just changed a few of the formatting of the x-labels of the graphs in example 3 in #809

@jeromedockes I think this is now reviewable :)

One caveat of this first implementation of skrub.to_datetime is that when we pass a numpy array of object dtypes, with mixed-typed entries, we might end up with some numeric representations of the date that are not helpful:
import numpy as np
import pandas as pd
from skrub import to_datetime

df = pd.DataFrame(
    dict(
        a=["2020-01-01", "2021-01-01"],
        b=[2020, 2021],
    )
)

# OK
to_datetime(df)
# a	b
# 0	2020-01-01	2020
# 1	2021-01-01	2021

# Not very helpful
to_datetime(df.values)
# array([[1609459200000000000, 2021],
#       [1609545600000000000, 1021]], dtype=object)

# Better, but needs all columns to be datetime
to_datetime(df.values[:, [0]])
# array([['2020-01-01T00:00:00.000000000'],
#       ['2021-01-01T00:00:00.000000000']], dtype='datetime64[ns]')

# OK, by converting to pydatetime
X_split = list(df.values.T)
X_split[0] = pd.to_datetime(X_split[0]).to_pydatetime()
out = np.vstack(X_split).T
out
# array([[datetime.datetime(2020, 1, 1, 0, 0), 2020],
#      [datetime.datetime(2021, 1, 1, 0, 0), 2021]], dtype=object)

# Notice how we still lose the integer representation of the second column.
pd.DataFrame(out).dtypes
# 0    datetime64[ns]
# 1            object

Also, we convert dataframes to a set of numpy 1d array, which might be problematic for Polars datetime representation for example.
Therefore, we should consider improving the to_datetime logic in a subsequent PR to specialize the ndarray, Polars, and Pandas dataframes / series cases, instead of reducing everything to a set of 1d numpy array.

# Not very helpful
to_datetime(df.values)
# array([[1609459200000000000, 2020],
#       [1609545600000000000, 2021]], dtype=object)

for this one I see a different result:
[[1577836800000000000 2020]
 [1609459200000000000 2021]]

which corresponds to the timestamps of the 2 dates. Why do you think that is not helpful?

in the case where we have to output a numerical numpy array, the timestamp is probably the best we can do. but as you said in most cases we probably want dataframes as input and outputs, with proper datetime columns in the outputs

for this one I see a different result:

Yes, the results are just for illustration don't mind them.

which corresponds to the timestamps of the 2 dates. Why do you think that is not helpful?

More specifically, not helpful in the context of working with dataframes because we lose the datetime representation (using pd.to_datetime with timestamps is always tricky because of the unit kwarg, and this is not something we can do with skrub.to_datetime), but helpful for ML downstream applications.

@LeoGrin  it has changed a bit since your approval, do you want to have another look? If not I will merge it

@LeoGrin it has changed a bit since your approval, do you want to have another look? If not I will merge it

I'll have a look tonight, thanks!

Just a few comments but otherwise LGTM, thanks!

Let's merge then!


fix null_count on compute_record_batch_statistics to report null counts across partitions

Rationale for this change
Maybe I'm missing something or being dumb, but while reading datafusion::physical_plan::common::compute_record_batch_statistics I noticed that the null_count took the value from the last partition, not the sum of all partition.
What changes are included in this PR?

fix compute_record_batch_statistics
add a test

Are these changes tested?
yes
Are there any user-facing changes?
I don't think so, perhaps some queries are now correct that were previously incorrect.

I took the liberty of merging up from main and runnning cargo fmt to fix the CI test

Thanks again @samuelcolvin  -- sorry for the delay in merging


Added PHP Censor

Added PHP Censor - An open source self-hosted continuous integration server for PHP projects (Fork of PHPCI)

Any reason why this is a fork and not a contribution back to the main project?

@josegonzalez Direct quote from https://github.com/ziadoz/awesome-php/blob/master/CONTRIBUTING.md ...

Unfortunately, not every library, tool or framework can be considered for inclusion. The aim of Awesome PHP is to be a concise list of noteworthy and interesting software written in modern PHP. Therefore, suggested software should:
a) Be widely recommended regardless of personal opinion b) Well known or discussed within the PHP community c) Be unique in its approach or function d) Fill a niche gap in the market

@pothi thanks for quoting the contributing docs at me.
@corpsee Any reason why your fork should be included over the existing PHPCI entry?

@josegonzalez PHPCI in fact abandoned, pull requests waits for years. PHP Censor fixed a lot of PHPCI bugs, It has LDAP authentication and PostgreSQL support unlike PHPCI etc.

 If anything, I'd rather we have one or the other.


Update update Doris tzfiles to 2024a

Proposed changes
Issue Number: close #xxx
doc pr: https://github.com/apache/doris-website/pull/451
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


fix(boards): hide button for deleted bodies if no permission

:tada: This PR is included in version 1.34.5 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Allow the voxel that produced an UnorientedUnitQuad to be preserved

Basically what the title says. The objective behind this is to allow custom texture coords to be generated depending on the type/data of the original voxel

Why is this necessary? You already have the coordinates of the voxel on the quad, so you can easily access it.

Well for my use case I want to implement a single texture atlas for a chunk mesh with per voxel texture coordinates, but without information on what the original voxel is I cannot generate the correct texture atlas coordinates.

but without information on what the original voxel

But you do have this information. It's just the voxel from the array you provided to the meshing function.


Cannot write integration test: 'Lock cannot be acquired after being cancelled' (regression since Mongock 4)

Description
I write integration tests of migrations written using Mongock: I execute them using MongockApplicationRunner.
What works in Mongock 4 crashes in Mongock 5. During Spring context load, Mongock enters a state where it refuses to run any further migrations.
PRIORITY
NORMAL
Version and environment
Mongock

Mongock version: 5.0.36
mongock-springboot + mongodb-springdata-v3-driver
annotation approach

Environment

Spring Boot 2.6.4
Infrastructure: JUnit Jupiter + Spring Test

Steps to Reproduce

Write a @SpringBootTest
Let Spring inject MongockApplicationRunner into your test
Use MongockApplicationRunner to execute your migration

Behaviour
Expected behavior: Mongock runner executes my migration
Actual behavior: Runner does not even start, it crashes on acquiring lock:
2022-03-02 20:20:52.237 ERROR 81672 --- [    Test worker] i.m.r.core.executor.MongockRunnerImpl    : Mongock did not acquire process lock. EXITING WITHOUT RUNNING DATA MIGRATION

io.mongock.driver.api.lock.LockCheckException: Lock cannot be acquired after being cancelled
	at io.mongock.driver.core.lock.LockManagerDefault.initialize(LockManagerDefault.java:374) ~[mongock-driver-core-5.0.36.jar:na]
	at io.mongock.driver.core.lock.LockManagerDefault.acquireLockDefault(LockManagerDefault.java:145) ~[mongock-driver-core-5.0.36.jar:na]
	at io.mongock.runner.core.executor.operation.change.MigrationExecutorBase.executeMigration(MigrationExecutorBase.java:96) ~[mongock-runner-core-5.0.36.jar:na]
	at io.mongock.runner.core.executor.operation.change.MigrationExecutorBase.executeMigration(MigrationExecutorBase.java:46) ~[mongock-runner-core-5.0.36.jar:na]

The exception is misleading: nothing is ever cancelled. The previous run of Mongock (during Spring context load) finishes successfully.
io.mongock.driver.core.lock.LockManagerDefault#initialize documentation claims the method is 'idempotent'. Of course it isn't because it starts with:
    if (releaseStarted) {
      throw new LockCheckException("Lock cannot be acquired after being cancelled");
    }

releaseStarted is only set to true, never set to false. This means in Mongock 5, the lock manager becomes useless the moment after it's started for the first time. There is no way to correctly use it again.
How often the bug happens: Always
Link to repository using Mongock
https://github.com/pkubowicz/mongock5-springboot-test
A GitHub action proves that 100% identical code works in Mongock 4 but crashes in Mongock 5.
Additional context
Maybe this is not the most effective way of testing a migration, but the Mongock documentation does not offer any advice. Since my real-life system is built on Spring Boot, it seems logical to write tests in Spring Boot way. I have to control running migrations in my test, because I need to save initial state into the database.

Hello @pkubowicz ,
this is the expected behaviour. A MongockRunner is expected to be used just once. After finished, it cannot be re-executed.
Anyway, I don't see why an application needs to run multiple times the same MongockRunner instance.Unless we are speaking about a  multi-tenant scenario, for which Mongock provides a built-in solution.
The same way an integration test shouldn't neither and I believe there are ways to solve this issue. Let me take a look to the sample project you provide, between tomorrow and Friday, and I will try to come up with solution

Hello @pkubowicz , we received a priority bug for a client that has took longer that we expected. We'll try to take a look and get back to you as soon as possible.

Hello, do you have any suggestions? Just reminding, it seems that in Mongock 5, there is no way of testing migrations.

Being frank, we have been to busy, I totally forgot about this.
But. what do you mean with "there is no way of testing migrations"? The only limitation you have is that a MongockRunner can only be executed once. But I don't see why that's an obstacle to test migration.
Anyway, I think we can have some free time this week to look at this. We'll try to and will let you know

So how do you write your tests if:

a Spring test begins with loading Spring context
loading Spring context starts MongockRunner
MongockRunner is not usable after being started
so when a test method is started, test code cannot use MongockRunner to execute migrations

Hello @pkubowicz .
Aplogies for the delay - I finally got the chance to take a look at your project. I forked it and did a PR with some suggested changes on how it is best practice to write the tests with Mongock.
During the analysis, I found out that you are loading the SpringContext -  which already executes the Mongock migration - then you clean the data and execute Mongock again.
This is not the best approach for Mongock given the migration  will already run during the loading time of the Spring Boot Application.
I'd suggest to do segregate the test cases:

Have a unit test for that can test the changeUnit independently and
Have an integration test that can test the entire Mongock process independently from Spring boot
Have an integration test the entire Spring boot application with the Mongock migration in it.

This issue mainly covers the last one, testing the entire Spring boot application so that the Mongock runner again is not executed again.
To summarise: run the Spring boot application, with the Mongock process, ensuring some preconditions (in this case some pre-populated data in the database). This can be achieved in different ways, depending on the case, here you can create a InitializingBean like this:
package com.example.mongock;

@TestConfiguration
public class InitializerContext {

    @Bean
    @Order(1)
    public InitializingBean initData(MongoDatabaseFactory mongoDatabaseFactory) {
        return new InitializingBean() {
            @Override
            public void afterPropertiesSet() throws Exception {

                MongoDatabase mongoDatabase = mongoDatabaseFactory.getMongoDatabase();
                mongoDatabase.getCollection("user").deleteMany(new Document());
                mongoDatabase.getCollection("user").insertMany(
                        Stream.of("alice", "bob", "chris")
                                .map(name -> new Document().append("email", name + "@mail.com"))
                                .collect(toList())
                );
                mongoDatabase.getCollection("summary").deleteMany(new Document());
            }
        };
    }
}

and adding @Import(InitializerContext.class):
@SpringBootTest
@Import(InitializerContext.class)
@ContextConfiguration(initializers = MongoInitializer.class)
public abstract class BaseCountingMigrationTest {

Then you can remove the MongockRunner and leave the test as simple as:
    @Test
    public void works() {
        var found = mongoDatabase.getCollection("summary").find();
        assertThat(found).hasSize(1);
        //noinspection ConstantConditions
        assertThat(found.first().getInteger("userCount")).isEqualTo(3);
    }

I hope this clarifies your case :)

Thank you for the answer.

I'd suggest to do segregate the test cases:

I'm sure you already use Mongock in real-life projects. Is the segregation you suggest the one you use in your projects?

unit test for that can test the changeUnit independently

I used to have tests like this. For me the test was like - skip annotations, and manually call the migration as plain Java code:
void myTest(@Autowired MongoTemplate template) {
  new CountingMigration().run(template);
}

Very quickly I came into a situation where all tests passed, but the migration did not happen on production. The reason was that my migration was not annotated at all.
So I don't want any tests where I simulate what Mongock runner does. I don't want to follow changes in Mongock API and internals, and update my tests as annotations and their handling change over time. I want a black-box test that tells Mongock "here is the migration, run it", and inspect what is in the database.
This black-box approach (showed in a simplified way in my example repo) worked very well in Mongock 4. It allowed me to focus on data and not on the migration tool. But I am unable to continue using this approach.
Maybe I misunderstood what you suggest and your way works well. Can you share a simplified test code?
Sorry for bothering you, but, as I mentioned before, Mongock documentation offers no suggestions how to approach testing. I think it should provide even a tiniest example of a migration test so that users know where to start.
Imagine Project Reactor documentation with no word on how to approach testing when your code uses Reactive Streams.

InitializingBean

I don't want to write tests this way. This bean contains part of test scenario: it knows which elements populate the database before the migration is started. Just a few of numerous drawbacks of this approach:

unreadable and unmaintainable: I need to read two files to understand what are inputs and outputs of a single simple scenario
explosion of classes: if I have 2 scenarios: 1) database initially has 1 element 2) database initially has 3 elements, then I need to write 2 beans like this. What if I have 3 migrations, each tested with 2 scenarios? 6 files to add, and later remove if I clean up too old migrations from my code. 6 files which I may forget to remove, and will eat time during compilation, may break when I update major Spring version etc.
need to make those independent: @TestConfiguration will be picked by all tests. Now I have this InitializingBean for CountingMigration operating on user collection. Imagine I add a GreetingMigration operating on greetings migration. I write GreetingMigrationTest. Spring picks up InitializingBean test configuration, which results in users being put into the database. But GreetingMigration does not know about user collection, there is no sense it doing so. So maybe a Spring profile to separate initializing beans? Ok, it will work, but this is one more thing to think about: add migration, add its own test, add its own initializing bean, add its own Spring test profile.

Hello again @pkubowicz ,
first thing, we need to create a section in the documentation to provide some advices to how to test it. However, there is no one way of testing and depends very much on what framework the team is using and actually how that framework is used, as there are many ways of using Spring boot, for example. Anyway, we really need to provide some help on that.
Said that, like with any other tool or project there are different levels of testing. The one you mentioned without annotations, it's basically a unit test for the changeUnit. That can be very useful in specific scenarios, but obviously is far from being enough.
In simple words you are just trying to provide an integration test, in which you run spring boot and the migration is supposed to be part of the startup. You don't care, you just need to check the output is in the database as expected. Although still insufficient as if you don't set any precondition you are assuming the database is always empty, we can start from that. The problem with the test you had is that you were running Spring boot(and Mongock with it), then in your test you clean the database, populate some data into the database, run manually Mongock again and then check the database state.
That's not ideal and it's an approach somehow in the middle. Or you either run manually a MongockRunner instance without Spring boot or you run Spring boot with Mongock, but not both. I really recommend the second one, as it's more complete.
To achieve you can see the  PR I raised. You just need to remove the @TestConfiguration(and obviously update the checks accordingly in your testing, as it now changes a bit). With this approach, you can run Springboot(Mongock will be executed as part of the application startup) and you can check the expected output of tour changeUnits.
But, as said, you very likely want to test it with some pre-populated data. You can choose the way you want, I think to create an InitializingBean for test is a decent approach. To be honest, I don't fully agree all your points you make about it:

unreadable and unmaintainable: I need to read two files to understand what are inputs and outputs of a single simple scenario

Not really, you can set that TestConfiguration class(with your the bean) in the actual test class

Explosion of classes: if I have 2 scenarios: 1) database initially has 1 element 2) d...

This is actual true. But it's more a limitation(or design) of the test framework. However, I see
a cool and easy way to get around this , but it will require a new release.

Being totally honest, we cannot commit to a new release this week, but we can have it soon.

need to make those independent: @TestConfiguration will be picked by all tests...

Also not really. You don't need to share the @TestConfiguration class among all your tests, you
 can have it locally per test

My suggestion is to apply something similar to the PR I raised and then do the small change once the new release provide that feature(with documentation). Otherwise, you can just wait until it's released.

@pkubowicz ,
I have thinking and while we develop this small mechanism to make it easier(plus documentation) and assuming that you don't want to use that configuration I provided, you can try to keep your current code and in your test, instead of autowiring the MongockRunner, autowire the RunnerSpringbootBuilder.
So instead of
    @Autowired
    private MigrationExecutor migrationExecutor;

you should use
    @Autowired
    private RunnerSpringbootBuilder runnerSpringbootBuilder;

and then in you test:
    @Test
    public void works() {
        mongoDatabase.getCollection("user").deleteMany(new Document());
        mongoDatabase.getCollection("user").insertMany(
                Stream.of("alice", "bob", "chris")
                        .map(name -> new Document().append("email", name + "@mail.com"))
                        .collect(toList())
        );
        mongoDatabase.getCollection("summary").deleteMany(new Document());

        System.out.println("RUNNING MIGRATIONS");
        runnerSpringbootBuilder.buildRunner().execute();
        //migrationExecutor.runAllMigrations(mongoDatabase, migrationCollectionName);

        var found = mongoDatabase.getCollection("summary").find();
        assertThat(found).hasSize(1);
        //noinspection ConstantConditions
        assertThat(found.first().getInteger("userCount")).isEqualTo(3);
    }

This is still not right, as Mongock is running twice, but it should work. And you can perform as many test as you wish.
Please let me know if it works and on our side will notify you when the release is done as the mentioned mechanism is available 😉

I tried RunnerSpringbootBuilder. If I execute what you suggest (runnerSpringbootBuilder instead of migrationExecutor), I get:
MigrationExecutorBase      : Mongock skipping the data migration. All change set items are already executed or there is no change set item.

Which is no surprise, since the migration is marked as 'applied' before the test start, at the time when the Spring context starts.
To be able to execute a migration after the context is loaded, I need to clean change-log-repository-name collection. Which is exactly why I have MigrationExecutor: it first cleans the collection, then calls a Mongock runner.
I tried first cleaning the collection, and then starting Mongock with RunnerSpringbootBuilder. But then we return to the starting point:
MongockRunnerImpl    : Mongock did not acquire process lock. EXITING WITHOUT RUNNING DATA MIGRATION
io.mongock.driver.api.lock.LockCheckException: Lock cannot be acquired after being cancelled
	at io.mongock.driver.core.lock.LockManagerDefault.initialize(LockManagerDefault.java:374) ~[mongock-driver-core-5.0.36.jar:na]

and, as I have written:

releaseStarted is only set to true, never set to false. This means in Mongock 5, the lock manager becomes useless the moment after it's started for the first time

I managed to write a test I want using StandaloneRunner.
In short:

I use mongock-standalone as a test-only dependency
I declare a @SpringBootConfiguration to prevent Spring Boot test from picking up @EnableMongock from the production configuration, and to prevent starting mongock before the test logic
I inject values from application.yml to make MongockStandalone configuration as close to the production configuration as possible

https://github.com/pkubowicz/mongock5-springboot-test/pull/2/files
I think I can use this PoC as an example how to update my real project. If you find anything there useful, feel free to use it when writing the documentation.


Am I doing something wrong or is PocketHost down?

Trying to access my instance at $instanceName.pockethost.io/_/, it returns "The requested resource wasn't found."
Here's my logs from Pockethost dashboard for today:
2023-11-30T06:37:54.694Z stdout
Instance stopped
2023-11-30T06:37:54.730Z stdout
Starting instance
2023-11-30T06:37:55.494Z stdout
2023/11/30 06:37:55 Server started at http://<IP_ADDRESS>:8090
2023-11-30T06:37:55.494Z stdout
├─ REST API: http://<IP_ADDRESS>:8090/api/
2023-11-30T06:37:55.494Z stdout
└─ Admin UI: http://<IP_ADDRESS>:8090/_/
Running version 0.19.*
Any reason why the instance has been stopped?

PocketHost is not directly related to PocketBase and I can't help you with it.
Note that everyone is free to offer PocketBase as a service if they want to and PocketHost is just one such example by @benallfree.
You can contact PocketHost and submit the report to their repo or discord channel - https://github.com/pockethost/pockethost/discussions.


Add Jenny Justh to Contributors.md

Hi  @jennyjj, I'm quite elated about your pull request. I wanna evolve this project to addresses various problems faced by first-time contributors. I'd love to learn about your journey in open source community, the problems, pain points you had etc.
Could you explain how you felt when you went through the tutorial, made a pull request and learned that I merged it?
We’ve recently added social share  to our web app. Could you please go to https://roshanjossey.github.io/first-contributions/#social-share and share your first contribution to open source?
Also, check out projects with easy issues while you’re there.

Hi Roshan, It was a great tutorial.  One thing missing was reminding us to add the files to our git (git add .).  I didn't do this the first time and the changes were not made in the repository.  But then I figured it out.  But it felt GREAT to commit it and receive your confirmation that you accepted the pull request:).

@jennyjj, git add is there. Please read through making necessary changes section again. git add . adds everything in a directory. This might not always be ideal. Try to read about git add -p too.
I’d urge you to continue contributing to other open source projects.
We also have a slack team to help/mentor people trying to contribute to open source projects.
You can join the team by going to https://goo.gl/PM1Jdh


Get the key of item pushed onto a list

There was a closed issue for this #199 but changes in the push method mean the answer no longer works.
I am trying to do something like this but value is a promise and using preserveSnapshot on the list doesn't seem ideal:
const mylist = af.database.list('myFirebasePath');
const promise = mylist.push({key: val}).then((value) => {
console.log(value.$key);
} );

let key = af.database.list('myFirebasePath').push{key:val}).getKey();

.push() does not return a Promise rather than a ThenableReference which contains the key value.