pdfDoc.images[] is empty

What were you trying to do?
Extract images from PDF
How did you attempt to do it?
pdfDoc has specific object properties (Great!), including images, but it's an empty array.
This works:
var fs = require('fs')
var path = require('path')
var Jimp = require('jimp')

const { PDFDocument, PDFRawStream, PDFName } = require('pdf-lib')

async function runTest() {
    //console.log("propName::" + propName);
    var templ = 'brochure.pdf'
    console.log("File:" + templ)
    //my load statement
    const pdfDoc = await PDFDocument.load(fs.readFileSync(templ))
    const enumeratedIndirectObjects = pdfDoc.context.enumerateIndirectObjects()
    enumeratedIndirectObjects.forEach(async (x, objIdx) => {
        const pdfRef = x[0]
        const pdfObject = x[1]

        if (!(pdfObject instanceof PDFRawStream)) return

        //const { dict } = pdfObject
        console.log('objIdx', objIdx, 'pdfObject', pdfObject)
        var buffer = Buffer.from(pdfObject.contents)
        try {
            const img = await Jimp.read(buffer)
            console.log(`${objIdx.toString()}.jpg`)
            img.writeAsync(`images/${objIdx.toString()}.jpg`, Jimp.MIME_JPEG)
        } catch (e) {
            // console.log(buffer);
            console.log([objIdx, e.message])
        }
    })

}

runTest()

What actually happened?
empty array
What did you expect to happen?
get a list of images. Each image should have buffer (that can be extracted/written to disk) and page index (from what page in PDF it came)
brochure.pdf
toyota.pdf
How can we reproduce the issue?
const pdfDoc = await PDFDocument.load(fs.readFileSync(originalPdfPath))
console.log('pdfDoc', pdfDoc)

Version
1.17.1
What environment are you running pdf-lib in?
Node
Checklist

[X] My report includes a Short, Self Contained, Correct (Compilable) Example.
[X] I have attached all PDFs, images, and other files needed to run my SSCCE.

Additional Notes
No response

Hello @fancydev18! Looks like you've misunderstood what the images property is used for. It's a private readonly property that is not part of the public API: https://github.com/Hopding/pdf-lib/blob/93dd36e85aa659a3bca09867d2d8fac172501fbe/src/api/PDFDocument.ts#L187
See https://github.com/Hopding/pdf-lib/issues/83#issuecomment-487383843 for more details on image extraction with pdf-lib.


Clear up makeRequest_ naming conventions

We follow a few different patterns throughout the codebase, so I think we should try to sync up.
In some constructors, you might find this.makeRequest_ = storage.bucket.makeRequest.bind(storage.bucket). In other places, we define a makeRequest_ on the prototype. (props to @callmehiphop for first calling us out on this).
We should also clear up the naming conventions around making requests. Currently:
function Class() {
  this.makeAuthorizedRequest_ = util.makeAuthenticatedRequestFactory();
}

Class.prototype.makeReq_ = function(method, path, qs, body, callback) {
  var reqOpts = {
    method: method,
    uri: path,
    // ...
  };

  this.makeAuthorizedRequest_(reqOpts);
};

I think we should go through and have only one style; Class.prototype.request instead of constructor-initialized functions.
function Class() {
  this.authClient = util.getAuthClient();
}

Class.prototype.request = function(reqOpts, callback) {
  util.request(this.authClient, reqOpts, callback);
};

It should make it easier to follow along with, and it should be possible.
Also note using (reqOpts, callback) over (method, path, query, body, callback). It's pretty non-JS to have multiple arguments. We should just use the standard request options format, with reqOpts.uri, reqOpts.method, reqOpts.qs, reqOpts.json, etc.
LMKWYT!

:+1:
I'm all for this!

Fixed by #904.


Sync from getambassador.io 2021-09-06

Closed in favor of https://github.com/datawire/ambassador-docs/pull/432

Closed in favor of https://github.com/datawire/ambassador-docs/pull/432


Update meilisearch js to v0.27.0

Update meilisearch-js to latest none-beta version

closed in favor of #153


@me instead of user id  to get private channel

I guess i accidentally did this with the cleanup-fix before 1.4 xD Pls don't hate me :sob:


MM-54499 Patch react-native-navigation to fix button text wrapping

Summary
We upgraded to the most recent version of react-native-navigation in #7491, but it has an issue with text wrapping in those buttons on iOS. There's a fix in for it (https://github.com/wix/react-native-navigation/pull/7781), but a version with that fix hasn't been released yet, so I just reverted for the time being.
Let me know if I should patch the new version of the library with that fix instead of downgrading.
Ticket Link
https://mattermost.atlassian.net/browse/MM-54499
Checklist

[x] Has UI change

Device Information
This PR was tested on: iOS Simulator (iOS 16.4, iPhone SE 3rd gen)
Screenshots

Release Note
Fixed text in Follow/Following button wrapping incorrectly on iOS

I rather you apply the patch than downgrade

I rather you apply the patch than downgrade

K, I'll make that change. Hopefully it's just the one change from the linked PR that's needed to fix this

@enahum I changed that. I did have to remove some .DS_Store files from the patch file, but it looks like I removed those correctly

Cherry pick is scheduled.


Ability to pass custom http port, TLS support, scale support

Custom port can be passed via metadata annotations, key http.port:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  annotations:
    http.port: "99"
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: nginx-service
          servicePort: 80

Scale can be passed via "scale" annotation:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  annotations:
    scale: "2"
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: nginx-service
          servicePort: 80


[protocolv2] always expect init, input sometimes (docs only)

Why
Currently we expect:

0 init messages for rpc and subscription, and 0-1 init messages for upload and stream
1-n input messages for upload and stream, and 0 input for

This works but if we did:

1 init message for all
0-n input messages (always 0 for rpc and subscription)

We'd have a shape of streams that is a little more consistent, making implementations simpler, although practically it's not much simpler since it's papered over in the libraries.
What changed
Updated documentation for init and input.

In practice this doesn't affect the implementations since wire messages don't differentiate between init and input.


Change link order for static links

zlib needs to be later in the link list when linking statically on Ubuntu 12.04. Static linking used to work on Ubuntu 12.04. I have not investigated what specific change caused the failure.
I tested this change on Ubuntu 12.04, 14.04, 15.10 and Fedora 22 or both static and dynamic linking.

Reviewers: @ximinez @miguelportilla

Current coverage is 49.39%

Merging #1417 into develop will increase coverage by +0.01% as of 8fd7e5d

@@            develop   #1417   diff @@
=======================================
  Files           964     964
  Stmts         67065   67065
  Branches          0       0
  Methods           0       0
=======================================
+ Hit           33122   33126     +4
  Partial           0       0
+ Missed        33943   33939     -4

Review entire Coverage Diff as of 8fd7e5d

Powered by Codecov. Updated on successful CI builds.

Looks good. Default builds work in Windows ("Static linking is only implemented for linux."). Default and static builds work in Linux. (I used ./Builds/Test.py -k -v -- -j 8  debug && rm -rf build/ && ./Builds/Test.py -k -v -- -j 8  debug  --static to test.)

@ximinez Push a commit (hopefully) addressing your comment on add_static_libs

Tested last commit. All default builds work in Windows and OS X. Default and static builds work in Linux 15.10 (./Builds/Test.py -k -v -- -j8 --static).

:+1:

:+1:


Add existing publishedClaim to argument in Draft

Server search is looking for an exact match... well that just will not do! Going to try and get it to do as the client is doing - see if the search text exists within and existing publishedClaim

Turns out this is now native in mongodb!! Just got to upgrade

Mongodb upgraded. Text search in place. Descriptions indexed. It now matches words but there's a problem in adding them into the argument group. That's the next bug fix!

Ok so it is saving the published claim as a reason. But we have no indication that it is published. Methinks reasons are going to have to hold a bit ore meta info

Refactoring reason objects - instead of a straight claim object they now have a meta object and a claim object. But for some reason when the reason is saved (as part of the current draft save) an id is being placed next to the reasonObject as well as within the reasonObject... I'm thinking there's a problem with the current draft save.

Argh. mre weirdness going on.  Somehow, when a new claim is written up as a reason and the user clicks 'save' - it already has an "_id" from mongo... wtf. Then when the current draft is saved, the extra _id pops in.
And - just noticed, each argument group is being assigned an _id. It's like mongo is caching old schemas... maybe, just maybe, it might be that

Nope - not caching. But one more bit of info - the argument _id is being created first, then the additional reason _id.

Aaaahhh - mongoose or mongo gives an _id to any object stored in an array. Seems like they just don't like saving plain old arrays!

so the populate functions aren't working anymore. got to find and fix

Such a nightmare. Looks like its saving fine - I'm totally cool with what's in the database.  But the stuff that's being put together in express - not good. This is where my efforts must lie!

Found the bug. in draftClaim-route.js when I try to set the reason object as = an object, something fails - it refuses and keeps the string... too weird, it must be a scope thing.

Giving up for now... this is the guy that's failing: replaceRefWithReason() in draftClaim-route.js

So - it's fixed!! not sure how. did some stringifying then parsing, made up a new object to edit... weirdness will be found in the future but for now - KABOOM this crazy bug is fixed.
Now. Back to why I was doing all that refactoring in the first place: adding a meta object so we know if a linked reason object is a draft or claim, so that when the thing loads we know where to look.

Ok - back onto this today!
Problem 1: When clicking insert on a published claim after a draft claim has already been added - all the reasons loose their text.

Found a bug - a listener is created on the search results every time we type a letter

FIXED IT!! That also seems to have fixed our problem.  Now when the published result is the only thing added to a reason, the text shows up - that's a first!  Time to test it a bit more

Spoke too soon - it made a new draftclaim.

So the result in the search results is the plain claim object. It's not being added with the extra meta telling us that it is actually published.

This may be because the insert button doesn't appear to be working (or watched?)

Ok - got the insert button working, listener is working, thoughan error did pop up after applying. BUT when saving, the published text still isn't appearing. Also, a weird new id is being applied in the DB. Possibly because it let me 'save' an 'imported' reason, which it shouldn't.

Ok - cleared out that search watcher error, new var name. Imported reason isn't being 'saved'. BUT Saving the big current draft gives us a "hit default..." when it checkes through the arg reasons.

Getting further with this!! Now we need to get the save reason button to 'save the link' to the published claim... I'm not really sure what that means yet.  Will probably have to do the same for links to existing draft claims... but I thought that was working already?

So, if we don't 'save the link' to an exitsting draft / claim and just go on with calling the update draft service what happens?
The update service just passes the entire current draft object to the server to be updated.

Oop, lost a comment there - left it open too long. Back on it for a bit - line 89 of argument editor controller, it's applying 'draft' state when it should be 'published'. hmm.

It's been over a month - but I'm back on this!
$rootScope.search.selectedResult.claimType; is referenced in argumentEditor-controller.js:102 to set the state of the new reason, for some reason it's not being set correctly when an existing published claim is added as a reason.

Think I got it figured! The db call is still going to the draftclaims collection for the published claim, not the claims collection.


[Teams] Can't get Input.Date data

Platform

JavaScript

Author or host
I'n an Author
Version of SDK
Node JS SDK

Adaptivecards 1.1.3
Botbuilder 3.11.0
Botbuilder Teams 0.1.6

Issue
Hi, we are developing a bot for MS Teams and using the Adaptive Cards. They are awesome, but one issue breaks all the stuff. The Input.Date field is not being passed back into code. For instance, we have Input.Date and Input.Time fields. After these fields were filled out and the card was submitted we've got the time only, no date data.
Here is the card
{
        "contentType": "application/vnd.microsoft.card.adaptive",
        "content": {
          "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
          "type": "AdaptiveCard",
          "version": "1.0",
          "body": [
            {
              "type": "Container",
              "items": [
                {
                  "type": "TextBlock",
                  "text": "Please fill out the time",
                  "wrap": true,
                  "weight": "bolder"
                },
                {
                  "type": "Input.Date",
                  "id": "date",
                  "placeholder": "Date",
                  "value": "2019-02-20"
                },
                {
                  "type": "Input.Time",
                  "id": "time",
                  "placeholder": "Time",
                  "value": "09:00"
                }
              ]
            }
          ],
          "actions": [
            {
              "type": "Action.Submit",
              "title": "Submit",
              "data": {
                "from": "getTime",
                "date": date,
                "time": time
              }
            }
          ]
        }
      };

Hi @matthidinger, thank you for a fast response.
Here is the new card
static get(originalText = false) {
      return {
        "contentType": "application/vnd.microsoft.card.adaptive",
        "content": {
          "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
          "type": "AdaptiveCard",
          "version": "1.0",
          "body": [
            {
              "type": "Container",
              "items": [
                {
                  "type": "TextBlock",
                  "text": "Sorry, I cannot recognize some of the data in the message. Please fill it out manually:",
                  "wrap": true
                },
                {
                  "type": "TextBlock",
                  "text": "What to do",
                  "wrap": true,
                  "weight": "bolder"
                },
                {
                  "type": "Input.Text",
                  "id": "text",
                  "placeholder": "Type an action name...",
                  "maxLength": 500
                },
                {
                  "type": "TextBlock",
                  "text": "When to do",
                  "wrap": true,
                  "weight": "bolder"
                },
                {
                  "type": "Input.Date",
                  "id": "date",
                  "placeholder": "2020-05-29 for instance..."
                },
                {
                  "type": "Input.Time",
                  "id": "time",
                  "placeholder": "Time..."
                }
              ]
            }
          ],
          "actions": [
            {
              "type": "Action.Submit",
              "title": "Submit",
              "data": { // for messageBack see this https://docs.microsoft.com/en-us/microsoftteams/platform/concepts/cards/cards-actions#adaptive-card-actions
                "from": "help",
                "originalText": originalText
              }
            }
          ]
        }
      };
    }

And here is what we've got after submitting it:
{
   from:"help"
   originalText:"aasd"
   text:"take a shower"
   time:"16:30"
}

As far as you can see all the data is returned fine. originalText parameter is passed to the card with the function - we receive the parameter and return the JSON of the card with this parameter. So, all the data is returned ok excepting the date.
Thank you for your help

This should be fixed in the latest builds; please re-open if you find that is not the case. Thanks!


Blocklist for intellisense suggestions

The Storybook project has recently started exporting hook functions from its @storybook/addons modules that shadow the names of the core hook functions exported by React project.
The result of this is that VSCode intellisense often suggests importing these hook functions instead of the ones from React, which is almost never what you want:

This is a real nightmare when coding, as often you'll end up importing the wrong one without even noticing, resulting in a error when you run the app.
It would be good if VSCode could provide a mechanism to block particular suggestions, based on their name and source.

The IntelliSense feature is powered by the TypeScript project. Please file a feature request upstream with an example of where this would be useful


question

Is charge via fawry return an code to users to pay their money for the nearest Fawry center?

Yes, after send request to fairy servers , the response return with payment code


Examples are a glitchy mess

In some instances the examples may look like this:

This is due to somewhat old pygame/SDL versions being shipped with Raspberry Pi OS.
Raspberry Pi 3 etc
On Pi 3B+, Pi Zero and other pre-Pi4 boards you can update pygame from pip to fix glitchiness:
sudo python3 -m pip install pygame --upgrade

Raspberry Pi 4
On Pi 4, you should upgrade pygame as above but it will probably result in this error when you try and run an example:
* failed to add service - already in use?

I believe this, followed by the unhelpful hard exit(1) is caused by this code: https://github.com/raspberrypi/userland/blob/97bc8180ad682b004ea224d1db7b8e108eda4397/interface/khronos/common/linux/khrn_client_rpc_linux.c#L214-L216
In any case the only way to work around this seems to be by compiling SDL from source... :grimacing:  ... and using your shiny new SDL to replace the system one used by pygame.
You can do that with the following steps:
wget https://www.libsdl.org/release/SDL2-2.0.14.zip
unzip SDL2-2.0.14.zip
cd SDL2-2.0.14
./configure --enable-video-rpi --enable-video-kmsdrm --prefix=`pwd`/install
make
make install

This will install SDL into an "install" directory within the "SDL2-2.0.14" directory, so don't delete it. (You can change the --prefix when configuring to something like /opt/SDL2-2.0.14 if you're so inclined, it doesn't really matter where you put it.)
Once built the new SDL libraries will be available in install/lib, find the full path by typing:
cd install/lib
pwd

In my case the path is /home/pi/SDL2-2.0.14/install/lib.
Now when running an example from this repository, you should prefix the command line with LD_LIBRARY_PATH=/home/pi/SDL2-2.0.14/install/lib, for example:
sudo LD_LIBRARY_PATH=/home/pi/SDL2-2.0.14/install/lib python3 clock.py

If everything is set up correctly on your Pi 4 you should see "Using driver: kmsdrm"

Thank you for this info - updating pygame fixed the issue on the Pi Zero. 👍

this helped me today I got the error about pygame.error: No video mode large enough for 640x480 however adding the config to boot did not help
I upgrade pygame without sudo in that worked pip3 install pygame --upgrade


fix: pagination ts error

fix pagination ts error
logs page enable  copy function name

An error is still reported at build time.


Adam

team name AyeshaAnaerotica

app-micropython was introduced by PR #13, so I will close this PR.


Positional arguments do not work for TransformerModel and wrong pipeline name used in the code for plagiarism demo

For the example app at https://github.com/elastic/elasticsearch-labs/blob/main/supporting-blog-content/plagiarism-detection-with-elasticsearch/plagiarism_detection_es.ipynb. I find there are following issues:

According to code at https://github.com/elastic/eland/blob/main/eland/ml/pytorch/transformers.py,
class TransformerModel: def __init__(     self,     *,     model_id: str,     task_type: str,     es_version: Optional[Tuple[int, int, int]] = None,     quantize: bool = False,     access_token: Optional[str] = None, ):
It needs to have keyword arguments for the constructor. I am not sure the demo uses which eland version. I use the latest version:
$ pip3 list | grep eland eland                     8.11.1

The call in the code:
tm = TransformerModel(hf_model_id, "text_embedding")
does not work. The working one is:
tm = TransformerModel(model_id=hf_model_id, task_type="text_embedding")
This applies to the code as well:
tm = TransformerModel(hf_model_id, "text_classification")

pipeline name is spelled wrongly in the code:

client.reindex(wait_for_completion=True,                source={                   "index": "plagiarism-docs"     },                dest= {                   "index": "plagiarism-checker",                   "pipeline": "plagiarism-pipeline"     } )
In the code, "plagiarism-pipeline" pipeline is never created. It should be "plagiarism-checker-pipeline".

The synchronous call for the reindex seems not working at all.

client.reindex(wait_for_completion=True,                source={                   "index": "plagiarism-docs"     },                dest= {                   "index": "plagiarism-checker",                   "pipeline": "plagiarism-pipeline"     } )
It always returns "Connection time out" error. I have to change it to False to make it work.

Issue 2 has already been address, as we had another bug report for it (#148).
@priscillaparodi Could you look at items 1 and 3 above?

3 is solved after the pipeline name update (item 2). I can’t reproduce the item 1 error, it is working, the model id was defined before.

@priscillaparodi The eland code referenced above agrees with what @liu-xiao-guo is saying. The constructor of the latest version of TransformerModel only takes keyword arguments, but you are passing the model id as a positional argument. Wouldn't it make sense to change the notebook to use keyword arguments here? The current version of the code seems broken.

I will check based on the latest version and update accordingly.

https://github.com/elastic/elasticsearch-labs/pull/156


Options Disappear after update to 4d24249

Adapt brightness, adapt rgb/x/y, and adapt color temp have disappeared after update.

Lights now no longer adapt any of those things.

That is because those options are now switches, such that you can automate it.
See the docs here: https://deploy-preview-14877--home-assistant-docs.netlify.app/integrations/adaptive_lighting/

Roger, thank you!


feat(richtext-lexical): add maxDepth property to various lexical features

Description

[ ] I have read and understand the CONTRIBUTING.md document in this repository.

Type of change

[ ] Chore (non-breaking change which does not add functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Change to the templates directory (does not affect core functionality)
[ ] Change to the examples directory (does not affect core functionality)
[ ] This change requires a documentation update

Checklist:

[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Existing test suite passes locally with my changes
[ ] I have made corresponding changes to the documentation

@AlessioGr how hard would it be to add the max depth feature on main?


record algorithm output area in scenario metrics

output area allows us to compute cost per output area
perhaps it's best to have 2 metrics:
output pixels, in megapixel
output resolution, in m²
Then we can avoid that low resolution algorithms have a seemingly low cost per km², but we can still compute it if we want to?

added these two new metrics:
[
        "results:proj:shape:area:megapixel",
        0.033369
      ],
      [
        "results:proj:bbox:area:utm:km2",
        3.3369
      ]

closing this one for now: new metrics have been added, and are being picked up by dashboard


Latest vstest builds do not build in sourcebuild with nullability enabled

As discussed in https://github.com/dotnet/installer/pull/16005#issuecomment-1500588135 there is a missing null check that causes the sourcebuild build to fail.

fix is https://github.com/microsoft/vstest/pull/4391 this is time sensitive for the dotnet org so quick triage would be appreciated


gitignored files aren't being ignore in the fuzzy finder

I've got some files ignored in .gitignore but they aren't ignored by fuzzy-finder.
Are they supposed to be?

 But in this one project I get ignored files. The .gitignore looks like follows:
# common dev and OS files
.DS_Store
node_modules/
dist/
npm-debug.log
yarn-error.log

# Editor directories and files
.idea
*.suo
*.ntvs*
*.njsproj
*.sln

### ignore everything
/*

### except for the following:

# folders
!/src
!/tests
!/examples

# global build, to be published
!/global.js
!/global.js.map

# config files
!/.builderrc
!/.editorconfig
!/.npmrc
!/.gitignore
!/.npmignore
!/package.json
!/*.config.js

# all good projects have a README!
!/README.md

I guess gitignore is working properly, because the Atom file explorer shows items grayed out as expected, just that fuzzy-finder apparently isn't following exactly the same rules.
In the following shot, I am able to fuzzy-find files in the grayed-out folders which are gitignored:

Thanks @trusktr - can you update your issue to add the issue template details?

I guess gitignore is working properly, because the Atom file explorer shows items grayed out as expected, just that fuzzy-finder apparently isn't following exactly the same rules.

Tree view uses its own setting Settings > Packages > Tree View > Hide VCS Ignored Files (this setting is toggled by pressing i when the tree view is focused).  Fuzzy finder uses the Settings > Core > Exclude VCS Ignored Paths setting (see the description for that setting).  What do you have for these settings?


CellProfiler workflow: optional scatter

This is an attempt to unify the "distributed" and "single-VM" CellProfiler workflows into the same workflow.  It will scatter if you specify the input splitby_metadata as something other than "".
It has the option to copy outputs to a google bucket.
But it will also work without copying to a google bucket: the output files are outputs of the workflow (i.e. they will be copied to the Cromwell execution directory).  The outputs are arrays.  Each array element will be one scatter unit, if scattering occurs, and otherwise the array will have only one element.
Totally untested.

I'm not sure why the tests are not running...


Support @param?

The plugin seems to happily generate the doc (despite the Undocumentable method, missing name warnings mentioned in another issue.)
So my source looks like this:
# Retrieve child node IDs from a given node ID
#
# @param [Number] nodeID path arg the node identifier to search
# @return [Array] a list of child IDs
get '/node/:nodeID/children' do

And the generated doc looks like this:
- (Array) GET '/node/:nodeID/children'

Retrieve descendant node IDs from a given node ID

Returns:
    * (Array) — a list of child IDs

So it looks like my @param tags are getting dropped from the generated doc.  This is even more important if @param is documenting URL query parameters that aren't visible in the path route.
Also, what would be the recommended method of documenting a POST body?  A @body tag would be nice :)  Thanks!

I'd be interested in a working @param too

Do you have an idea how to implement this, @tjeden?


New crypt function test fails with system python2.6 on OSX

/cc @dhermes
i didn't dig too far, but it looks like the system python doesn't behave as expected:
======================================================================
FAIL: test_succeeds (tests.test_crypt.Test_pkcs12_key_as_pem)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/craigcitro/gh/oauth2client/tests/test_crypt.py", line 57, in test_succeeds
    self.assertEqual(pem_contents, pkcs12_key_as_pem)
AssertionError: '-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA4ej0p7bQ7L/r4rVGUz9RN4VQWoej1Bg1mYWIDYslvKrk1gpj\n7wZgkdmM7oVK2OfgrSj/FCTkInKPqaCR0gD7K80q+mLBrN3PUkDrJQZpvRZIff3/\nxmVU1WeruQLFJjnFb2dqu0s/FY/2kWiJtBCakXvXEOb7zfbINuayL+MSsCGSdVYs\nSliS5qQpgyDap+8b5fpXZVJkq92hrcNtbkg7hCYUJczt8n9hcCTJCfUpApvaFQ18\npe+zpyl4+WzkP66I28hniMQyUlA1hBiskT7qiouq0m8IOodhv2fagSZKjOTTU2xk\nSBc//fy3ZpsL7WqgsZS7Q+0VRK8gKfqkxg5OYQIDAQABAoIBAQDGGHzQxGKX+ANk\nnQi53v/c6632dJKYXVJC+PDAz4+bzU800Y+n/bOYsWf/kCp94XcG4Lgsdd0Gx+Zq\nHD9CI1IcqqBRR2AFscsmmX6YzPLTuEKBGMW8twaYy3utlFxElMwoUEsrSWRcCA1y\nnHSDzTt871c7nxCXHxuZ6Nm/XCL7Bg8uidRTSC1sQrQyKgTPhtQdYrPQ4WZ1A4J9\nIisyDYmZodSNZe5P+LTJ6M1SCgH8KH9ZGIxv3diMwzNNpk3kxJc9yCnja4mjiGE2\nYCNusSycU5IhZwVeCTlhQGcNeV/skfg64xkiJE34c2y2ttFbdwBTPixStGaF09nU\nZ422D40BAoGBAPvVyRRsC3BF+qZdaSMFwI1yiXY7vQw5+JZh01tD28NuYdRFzjcJ\nvzT2n8LFpj5ZfZFvSMLMVEFVMgQvWnN0O6xdXvGov6qlRUSGaH9u+TCPNnIldjMP\nB8+xTwFMqI7uQr54wBB+Poq7dVRP+0oHb0NYAwUBXoEuvYo3c/nDoRcZAoGBAOWl\naLHjMv4CJbArzT8sPfic/8waSiLV9Ixs3Re5YREUTtnLq7LoymqB57UXJB3BNz/2\neCueuW71avlWlRtE/wXASj5jx6y5mIrlV4nZbVuyYff0QlcG+fgb6pcJQuO9DxMI\naqFGrWP3zye+LK87a6iR76dS9vRU+bHZpSVvGMKJAoGAFGt3TIKeQtJJyqeUWNSk\nklORNdcOMymYMIlqG+JatXQD1rR6ThgqOt8sgRyJqFCVT++YFMOAqXOBBLnaObZZ\nCFbh1fJ66BlSjoXff0W+SuOx5HuJJAa5+WtFHrPajwxeuRcNa8jwxUsB7n41wADu\nUqWWSRedVBg4Ijbw3nWwYDECgYB0pLew4z4bVuvdt+HgnJA9n0EuYowVdadpTEJg\nsoBjNHV4msLzdNqbjrAqgz6M/n8Ztg8D2PNHMNDNJPVHjJwcR7duSTA6w2p/4k28\nbvvk/45Ta3XmzlxZcZSOct3O31Cw0i2XDVc018IY5be8qendDYM08icNo7vQYkRH\n504kQQKBgQDjx60zpz8ozvm1XAj0wVhi7GwXe+5lTxiLi9Fxq721WDxPMiHDW2XL\nYXfFVy/9/GIMvEiGYdmarK1NW+VhWl1DC5xhDg0kvMfxplt4tynoq1uTsQTY31Mx\nBeF5CT/JuNYk3bEBF0H/Q3VGO1/ggVS+YezdFbLWIRoMnLj6XCFEGg==\n-----END RSA PRIVATE KEY-----\n' != '-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDh6PSnttDsv+vi\ntUZTP1E3hVBah6PUGDWZhYgNiyW8quTWCmPvBmCR2YzuhUrY5+CtKP8UJOQico+p\noJHSAPsrzSr6YsGs3c9SQOslBmm9Fkh9/f/GZVTVZ6u5AsUmOcVvZ2q7Sz8Vj/aR\naIm0EJqRe9cQ5vvN9sg25rIv4xKwIZJ1VixKWJLmpCmDINqn7xvl+ldlUmSr3aGt\nw21uSDuEJhQlzO3yf2FwJMkJ9SkCm9oVDXyl77OnKXj5bOQ/rojbyGeIxDJSUDWE\nGKyRPuqKi6rSbwg6h2G/Z9qBJkqM5NNTbGRIFz/9/LdmmwvtaqCxlLtD7RVEryAp\n+qTGDk5hAgMBAAECggEBAMYYfNDEYpf4A2SdCLne/9zrrfZ0kphdUkL48MDPj5vN\nTzTRj6f9s5ixZ/+QKn3hdwbguCx13QbH5mocP0IjUhyqoFFHYAWxyyaZfpjM8tO4\nQoEYxby3BpjLe62UXESUzChQSytJZFwIDXKcdIPNO3zvVzufEJcfG5no2b9cIvsG\nDy6J1FNILWxCtDIqBM+G1B1is9DhZnUDgn0iKzINiZmh1I1l7k/4tMnozVIKAfwo\nf1kYjG/d2IzDM02mTeTElz3IKeNriaOIYTZgI26xLJxTkiFnBV4JOWFAZw15X+yR\n+DrjGSIkTfhzbLa20Vt3AFM+LFK0ZoXT2dRnjbYPjQECgYEA+9XJFGwLcEX6pl1p\nIwXAjXKJdju9DDn4lmHTW0Pbw25h1EXONwm/NPafwsWmPll9kW9IwsxUQVUyBC9a\nc3Q7rF1e8ai/qqVFRIZof275MI82ciV2Mw8Hz7FPAUyoju5CvnjAEH4+irt1VE/7\nSgdvQ1gDBQFegS69ijdz+cOhFxkCgYEA5aVoseMy/gIlsCvNPyw9+Jz/zBpKItX0\njGzdF7lhERRO2cursujKaoHntRckHcE3P/Z4K565bvVq+VaVG0T/BcBKPmPHrLmY\niuVXidltW7Jh9/RCVwb5+BvqlwlC470PEwhqoUatY/fPJ74srztrqJHvp1L29FT5\nsdmlJW8YwokCgYAUa3dMgp5C0knKp5RY1KSSU5E11w4zKZgwiWob4lq1dAPWtHpO\nGCo63yyBHImoUJVP75gUw4Cpc4EEudo5tlkIVuHV8nroGVKOhd9/Rb5K47Hke4kk\nBrn5a0Ues9qPDF65Fw1ryPDFSwHufjXAAO5SpZZJF51UGDgiNvDedbBgMQKBgHSk\nt7DjPhtW69234eCckD2fQS5ijBV1p2lMQmCygGM0dXiawvN02puOsCqDPoz+fxm2\nDwPY80cw0M0k9UeMnBxHt25JMDrDan/iTbxu++T/jlNrdebOXFlxlI5y3c7fULDS\nLZcNVzTXwhjlt7yp6d0NgzTyJw2ju9BiREfnTiRBAoGBAOPHrTOnPyjO+bVcCPTB\nWGLsbBd77mVPGIuL0XGrvbVYPE8yIcNbZcthd8VXL/38Ygy8SIZh2ZqsrU1b5WFa\nXUMLnGEODSS8x/GmW3i3KeirW5OxBNjfUzEF4XkJP8m41iTdsQEXQf9DdUY7X+CB\nVL5h7N0VstYhGgycuPpcIUQa\n-----END PRIVATE KEY-----\n'

The issue is (likely?) a difference in some crypto dependency, but we should check that things work (or fail appropriately).

Did you run via tox or other?

both -- fails in tox and directly using /usr/bin/python2.6.

I'm trying to find documentation, but doesn't Apple ship a fake version of openssl or their own custom substitute not even named openssl?

On my machines:
$ # Linux
$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

$ # Mac OS X
$ openssl version
OpenSSL 0.9.8za 5 Jun 2014

So, regarding GoogleCloudPlatform/gcloud-python#537, I tested with this and it works even though OpenSSL 0.9.8za spits out a different PEM key than OpenSSL 1.0.1f. (Like I actually visited the signed URL, gave an expiration of 60 seconds and I could view the content for the whole 60 seconds on both OSes.)
It still means the tests fails on OS X. Can you get one of your auth contacts to weigh in?

More context:
OS X puts stuff in /System/... and then puts /System/... ahead of standard location (i.e. the one pip uses) on the import path.
SRSLY APPLE? Thanks but no thanks.
On the flip-side: Heartbleed never impacted OS X since they use a custom OpenSSL (at C level, not just Python).

@anthmgoogle Anthony, an auth question for you: we're seeing that some of the auth libraries we use for signing behave slightly differently. what OS/openssl version(s) do we want to support? "as many as possible" is my guess, but want to confirm.
@dhermes same command as you on the machine where it fails:
$ which openssl
/usr/bin/openssl
$ openssl version
OpenSSL 0.9.8zc 15 Oct 2014

maybe za and zc differ in some small way?

@craigcitro We still support Mac OS X. The test just fails because the signed content is different so comparing to pem_from_pkcs12.pem fails.
As I mentioned above, using the signed content still works, i.e. it's still valid.

ah, maybe i misread that above -- should we just loosen the test to check for either key?
@anthmgoogle if everything seems to function properly, do you have any strong feelings from the auth side?

Yeah we could just create the equivalent for the openssl that Apple uses and then use that file in the test if sys.platform == 'darwin'.

i was thinking something more lowbrow -- the input is fixed, and we have a small finite number of possible outputs. just check that it's one of them and call it a day?

SGTM

PTAL @dhermes

Why don't you name _alternate Mac OS X or something? Other than the history, the test gives no indication of why more than one file is needed.

(test failure is because i cancelled the second run when i pushed to correct author; passing version is here: https://travis-ci.org/google/oauth2client/builds/49809411
as to the filename -- i don't think we're going to encode enough into it to nail down the failing version, so it seems misleading. (for instance, it works for you on OSX -- someone else in the same boat might say "oh maybe we can delete this now".)

I was just suggesting a comment as to the origin of the file. Do as you wish.

No concerns with checking for either key type.
On Fri, Feb 6, 2015 at 3:54 PM, Craig Citro<EMAIL_ADDRESS>wrote:

Merged #117 https://github.com/google/oauth2client/pull/117.
—
Reply to this email directly or view it on GitHub
https://github.com/google/oauth2client/pull/117#event-231842841.


[iOS]Flutter which is integrated into iOS projects as a Flutter Module, does not take effect when the code is modified.

I have integrated Flutter into my iOS project as a Flutter Module, and every time I change the Flutter code and then re-run it in Xcode, the code does not take effect.
I have to remove the stuff inside Xcode->DerivedData and then re-run it for it to take effect.
I have created a new iOS project and a new Flutter Module project and have the same problem.
This problem does not exist in 2.2.0.
Podfile
target 'flutter_test' do

     flutter_application_path ='../f_f'
     load File.join(flutter_application_path, '.ios', 'Flutter', 'podhelper.rb')
     install_all_flutter_pods(flutter_application_path)

  # Pods for flutter_test

end

flutter doctor
[✓] Flutter (Channel stable, 2.10.3, on macOS 12.2.1 21D62 darwin-arm, locale
    zh-Hans-CN)
    • Flutter version 2.10.3 at /Users/choikarl/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 7e9793dee1 (6 weeks ago), 2022-03-02 11:23:12 -0600
    • Engine revision bd539267b4
    • Dart version 2.16.1
    • DevTools version 2.9.2
    • Pub download mirror https://pub.flutter-io.cn
    • Flutter download mirror https://storage.flutter-io.cn
[!] Android toolchain - develop for Android devices (Android SDK version
32.1.0-rc1)
• Android SDK at /Users/choikarl/Library/Android/sdk
✗ cmdline-tools component is missing
Run path/to/sdkmanager --install "cmdline-tools;latest"
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run flutter doctor --android-licenses to accept the SDK licenses.
See https://flutter.dev/docs/get-started/install/macos#android-setup for
more details.
[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
• Xcode at /Applications/Xcode.app/Contents/Developer
• CocoaPods version 1.11.3
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[!] Android Studio
• Android Studio at /Applications/Android Studio Preview.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
✗ Unable to find bundled Java version.
• Try updating or re-installing Android Studio.
[✓] Android Studio (version 2021.1)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7772763)
[☠] IntelliJ IDEA Ultimate Edition (the doctor check crashed)
✗ Due to an error, the doctor check did not complete. If the error message
below is not helpful, please let us know about this issue at
https://github.com/flutter/flutter/issues.
✗ FormatException: Unexpected extension byte (at offset 5)
• #0      _Utf8Decoder.convertSingle
(dart:convert-patch/convert_patch.dart:1789:7)
#1      Utf8Decoder.convert (dart:convert/utf.dart:351:42)
#2      InputStream.readString
(package:archive/src/util/input_stream.dart:207:30)
#3      new ZipDirectory.read
(package:archive/src/zip/zip_directory.dart:40:30)
#4      ZipDecoder.decodeBuffer
(package:archive/src/zip_decoder.dart:19:30)
#5      ZipDecoder.decodeBytes
(package:archive/src/zip_decoder.dart:14:12)
#6      IntelliJPlugins._findPluginXml
(package:flutter_tools/src/intellij/intellij.dart:130:44)
#7      IntelliJPlugins._readPackageVersion
(package:flutter_tools/src/intellij/intellij.dart:141:40)
#8      IntelliJPlugins.validatePackage
(package:flutter_tools/src/intellij/intellij.dart:63:35)
#9      IntelliJValidator.validate
(package:flutter_tools/src/intellij/intellij_validator.dart:103:15)
#10     asyncGuard.
(package:flutter_tools/src/base/async_guard.dart:111:32)
#11     asyncGuard.
(package:flutter_tools/src/base/async_guard.dart:109:18)
#12     _rootRun (dart:async/zone.dart:1426:13)
#13     _CustomZone.run (dart:async/zone.dart:1328:19)
#14     _runZoned (dart:async/zone.dart:1861:10)
#15     runZonedGuarded (dart:async/zone.dart:1849:12)
#16     runZoned (dart:async/zone.dart:1780:12)
#17     asyncGuard (package:flutter_tools/src/base/async_guard.dart:109:3)
#18     Doctor.startValidatorTasks
(package:flutter_tools/src/doctor.dart:205:9)
#19     Doctor.diagnose (package:flutter_tools/src/doctor.dart:309:47)
#20     DoctorCommand.runCommand
(package:flutter_tools/src/commands/doctor.dart:50:48)
#21     FlutterCommand.verifyThenRunCommand
(package:flutter_tools/src/runner/flutter_command.dart:1320:12)

#22     FlutterCommand.run.
(package:flutter_tools/src/runner/flutter_command.dart:1161:27)

#23     AppContext.run.
(package:flutter_tools/src/base/context.dart:150:19)

#24     CommandRunner.runCommand (package:args/command_runner.dart:209:13)

#25     FlutterCommandRunner.runCommand.
(package:flutter_tools/src/runner/flutter_command_runner.dart:281:9)

#26     AppContext.run.
(package:flutter_tools/src/base/context.dart:150:19)

#27     FlutterCommandRunner.runCommand
(package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)

#28     run..
(package:flutter_tools/runner.dart:62:9)

#29     AppContext.run.
(package:flutter_tools/src/base/context.dart:150:19)

#30     main (package:flutter_tools/executable.dart:94:3)

[✓] VS Code (version 1.66.0)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension can be installed from:
🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter
[✓] Connected device (2 available)
• iPhone 13 Pro (mobile) • 9EF2F616-DB7E-4813-BB9B-BC3C3139B3B7 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-15-2
(simulator)
• Chrome (web)           • chrome                               • web-javascript • Google Chrome 100.0.4896.88
! Error: 林夕 is not connected. Xcode will continue when 林夕 is connected. (code -13)
[!] HTTP Host Availability
✗ HTTP host https://maven.google.com/ is not reachable. Reason: An error occurred while checking the HTTP host: Operation timed out
! Doctor found issues in 4 categories.

@ChoiKarl
Not sure if this is a bug or maybe a caching issue at your end. I tried a sample add-2-app on iOS simulator, modified a text in flutter's main.dart and ran from XCode which showed updated code as below:

appBar: AppBar(
        title: const Text('Full-screen Flutter Demo'),
      ),`

Earlier the text was Full-screen Flutter.
I suggest you to try the samples app and see if you still get same behavior or not.
https://github.com/flutter/samples

I have this problem and so do my colleagues.
Demo.zip
https://user-images.githubusercontent.com/12951859/163785470-1931ecd5-de4f-4e7d-9637-995d1667d3ab.mp4

@ChoiKarl
Please try to upgrade to latest stable or master version and see if it still shows the behavior or not.

@darshankawar
Yes, still have the same problem, I tried to upgrade to 2.10.5 also have the problem.

@ChoiKarl
From the video you shared, you are making changes in Android Studio but trying to run from XCode. Did you try hot restarting from Android Studio itself where you made the changes ?

@darshankawar
If I connect with Flutter Attach and execute hot reset it takes effect.
But I want it to work straight from Xcode.

@ChoiKarl
Since you are using add2app, it requires you to use flutter attach and hot reload to see the updates.
See this related link for your reference: https://docs.flutter.dev/development/add-to-app#add-to-ios-applications
Closing this as using flutter attach and hot reloading should work and per your comment. If you disagree, write in comments and I'll reopen it.

I think this should be reopened, every time the app is compiled it should also recompile the dart code.  I can see from the recording that's not what's happening, Xcode is running the app but it's not reflecting the dart code edited before the app compiled.
Can you reproduce again, and export your Xcode build logs?

@jmagman
log.txt

I deleted it and re-ran,  it’s OK.

I deleted it and re-ran, it’s OK.

From those logs I see the dart code recompiling, were you reproducing the issue when this happened?
[ +863 ms] kernel_snapshot: Starting due to {InvalidatedReasonKind.inputChanged: The following inputs have updated contents: /Users/choikarl/Desktop/-项目/market-flutter/lib/shop_cart/view/item/shop_cart_app_bar.dart}
[   +8 ms] /Users/choikarl/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev /Users/choikarl/flutter/bin/cache/artifacts/engine/darwin-x64/frontend_server.dart.snapshot --sdk-root /Users/choikarl/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk/ --target=flutter --no-print-incremental-dependencies -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --no-link-platform --packages /Users/choikarl/Desktop/-项目/market-flutter/.dart_tool/package_config.json --output-dill /Users/choikarl/Desktop/-项目/market-flutter/.dart_tool/flutter_build/35f2c5933da9d4a69480d25ea0491bce/app.dill --depfile /Users/choikarl/Desktop/-项目/market-flutter/.dart_tool/flutter_build/35f2c5933da9d4a69480d25ea0491bce/kernel_snapshot.d /Users/choikarl/Desktop/-项目/market-flutter/.dart_tool/flutter_build/generated_main.dart
[+8366 ms] kernel_snapshot: Complete

Ideally I need:

Run from Xcode, see app run, stop app
Change dart code
Run again in Xcode, see that the dart code was not updated in the app
Export logs.

@jmagman
I have always been able to have problems.
https://user-images.githubusercontent.com/12951859/164584496-8346812b-2029-4572-9fe5-75d5454b54f7.mp4
log.txt

Interesting, the code actually is recompiling, but it's not being embedded into the app?
[ +409 ms] kernel_snapshot: Starting due to {InvalidatedReasonKind.inputChanged: The following inputs have updated contents: /Users/choikarl/Downloads/Test/flutter_module_project/lib/main.dart}
[   +4 ms] /Users/choikarl/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev /Users/choikarl/flutter/bin/cache/artifacts/engine/darwin-x64/frontend_server.dart.snapshot --sdk-root /Users/choikarl/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk/ --target=flutter --no-print-incremental-dependencies -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --no-link-platform --packages /Users/choikarl/Downloads/Test/flutter_module_project/.dart_tool/package_config.json --output-dill /Users/choikarl/Downloads/Test/flutter_module_project/.dart_tool/flutter_build/2299146171019cc8b5384988fa59ae00/app.dill --depfile /Users/choikarl/Downloads/Test/flutter_module_project/.dart_tool/flutter_build/2299146171019cc8b5384988fa59ae00/kernel_snapshot.d package:flutter_module_project/main.dart
[+3950 ms] kernel_snapshot: Complete

@jmagman
yes, it's not embedded into the app.
https://user-images.githubusercontent.com/12951859/164882982-cad059bc-8e9d-49f5-92c2-ec4df0170483.mp4
build app time.

product app.framework time.

I was able to reproduce this! The App.framework podspec is pointed to flutter_module_project/.ios/Flutter/App.framework but the Flutter tooling is building it subsequent times to the DerivedData products directory /path/to/DerivedData/project/Build/Products/Debug-iphonesimulator/App.framework

CocoaPods is copying the frameworks from BUILT_PRODUCTS_DIR correctly the first time, but the script based in xcfilelist inputs that point to the stale versions in the .ios module directory.

@jmagman
How can I solve this problem without upgrading Flutter? My current version is 2.10.3.


WIP: Bin packing optimization

Description
An optimization implementation provided by Databricks is bin-packing which coalesces small files into a larger file. This reduces the number of calls to the underlying storage and results in faster table reads.
This is a high level description of the process. A user can provide a filter for which partitions they want to optimize. Active add actions for those partitions are obtain and then placed into bins labeled by the partition. Once actions are in their respective bin, start building additional bins which consists of a list files to be merged together. Files with a size larger than delta.targetFileSize or bins with only a single file are not optimized.
Each bin is then processed and the smaller files are written to a larger one. Corresponding Add and Remove actions are created. Metrics on how many files were considered and total file size are also captured.
Related Issue(s)

MVP for #98

Issues

Currently the writer's provided by delta-rs write partition information to the parquet file. This differs from the Databricks implementation which does not. This causes a schema mismatch to occur when packing files from different writers. This is currently handled by dropping partition columns when rewriting.

Similar to (1.), schema evolution allows the addition of new columns, converting NullTypes -> any other type, and upcasts. Currently this is not handled and will cause the a Schema mismatch error. Looking for some guidance on how these scenarios should be handled.

Todo

[x] Add file statistics using new writers
[ ] Implement back fill and partition column drop on partition write
[ ] Determine criteria for when optimize should fail
[ ] idempotent tests

Hi @houqp, let me know if there's anything additional you would like to see in the PR and would appreciate any feedback with the issues items. I'll work on finishing this up by adding statistics on writes.

Thanks @Blajda - this is a great update to delta-rs. Personally I think it would be good to re-use the existing writers and fix/extend them where they fall short for the optimize scenario. At least the RecordBatchWriter should handle dropping columns that are represented as partition folders within the divide_by_partitions function. There are some tests for that, but if they don't that's a bug we should definitely fix. They also handle the file naming and computing file statistics. That way we don't duplicate that logic.
Right now do not handle muti-part files or desired file sizes, which might be use case here. I.e. we could extend the writer to either eagerly flush files when the size of the in-memory writer reaches the desired files, or create a new underlying arrow writer and flush only when flush is called. In case of multiple files, these could be the multi-part files.
A downside of the current implementation is, that we always try to partition the data written into the writer, which might be a fairly expensive operation when its not needed - not sure however how it will behave, if we don't have the partition columns present, since there is nothing to partition by :).
We could try and use the PartitionWriter directly, but the logic for add actions / statistics is currently part of the higher level writer - i think.

very cool @Blajda ! I agree with  @roeap the code reuse suggestion.

Currently the writer's provided by delta-rs write partition information to the parquet file. This differs from the Databricks implementation which does not. This causes a schema mismatch to occur when packing files from different writers. This is currently handled by dropping partition columns when rewriting.

Ideally, we should also not write out partition columns into the actual parquet file. I think this is an oversight from our end. User of delta-rs should be responsible for populating the partition columns. We could track it as a follow up clean up task.

Similar to (1.), schema evolution allows the addition of new columns, converting NullTypes -> any other type, and upcasts. Currently this is not handled and will cause the a Schema mismatch error. Looking for some guidance on how these scenarios should be handled.

As far as I know, there is no easy way around this other than we need to manually compare the table schema with the schema we got after reading from the parquet file and backfill new columns in the arrow record batches before writing them out to the new parquet file.

Just had a quick look into the RecordbatchWriter. To me it seems we do make sure the partition columns are not part of the data written out to storage.
https://github.com/delta-io/delta-rs/blob/5b2a09c59e1fa18f38c275475e41c2148ae02b5e/rust/src/writer/record_batch.rs#L184-L194
We use that partitioned schema to initialize the PartitonWriter and validate the Schema of data written into the PartitionWriter. Then again, I might have missed something, or there could be a bug ...

Hi @roeap,
I originally wrote this at the start of April with the original JSON writer and I haven't tested that scenario with the new writers. Glad to hear it was resolved and sorry about the confusion. The protocol link was helpful, I was trying determine if default values are supported when adding columns and this confirms that null for back-filling is acceptable.
I want to avoid the usage of divide_by_partition_values since the sort is not required in this case but I do agree there are utilities such as next_data_path which can be reused.
Next steps for me is factoring out the drop partition columns functionality and creating some utility for back-filling.

@Blajda - what do you think about adding a write_partition method to the Writer trait and essentially carving out the inner loop over partitions in the current write method? It would then be great to have the writer aware of the desired file size as well, since right not it is not. In the case of optimize you do have more prior knowledge about how to bin the files, that the optimize command can already leverage - but the general logic would be a great addition to the writer.
That way you get the create add (foremost stats) for free. More importantly though, as we move to support higher writer versions we will need to handle things like column alias / renames, column invariants, identity- and calculated columns etc. which I suspect will be non trivial logic. By leveraging a single writer struct I feel our lives might be a lot easier if we have a single code path that writes out data.

what do you think about adding a write_partition method to the Writer trait and essentially carving out the inner loop over partitions in the current write method?

Yes that sounds like the right approach to ensure the writer's are unified. I assume the signature would look something like this.
async fn write_partition(&mut self, values: RecordBatch, partition: &?) -> Result<(), DeltaWriterError>

Now it's mostly a decision on how to represent the full partition path. We can reuse get_partition_key and have ? be a String. My only concern is that it might be bit confusing for new users. Maybe we can make a new type called PartitionPath which can be created in the same way as get_partition_key and simply encapsulates a String.
I think that abstraction would be very helpful and would also help cleanup the bin packing implementation.
Let me know what you think

I'm ready for another review of this PR.
For the scope of this work I did not implement backfill so it expected to fail when called on tables that had schema evolution. I got blocked when handling the evolution of Arrow Lists and had trouble determining the best way forward.
This is expected to be used with tables that only perform appends. Support for different writing patterns would only require a check for remove actions on the impacted partitions so I think we are safe to ignore add actions to a partition. Once we have pattern for performing those check it should be small change.
@roeap I first tried adding a write_partition to the Writer trait but I wasn't completely sure what to do for the JSON writer case. For now I've only added it to the record batch writer and ensured that partitioned related columns are dropped when called. Let me know what you think and if anything else should be changed.

Looking good!
Left some minor questions. Reading this I realized that we should probably treat other operations - like vacuum - the same as here. I.e. having it as a separtae struct that can be executed ... But this is something for a follow up.

Hi @roeap @wjones127
I've implemented the suggested changes and add additional tests. Let me know what you think! :smiley:

Thanks @wjones127 and @roeap  for helping carry this to completion.

Amazing work on this @Blajda !


Update the wsk installation instructions

This updates the wsk install docs and adds in the (uncommented) brew docs so it'll be real easy to add them if we want, once brew has approved our pull request.
The docs here are a little lengthier and provide a little more help and context for users that aren't super experienced with setting a PATH in their shell.
~Caveat: I suspect these docs will not work on Windows, however the other docs we had also gave you no help when it came to Windows users so I'm not sure if this makes it worse or not.~

I updated the docs to include a separate version for Windows. This removes the earlier concern I had.

Hey @Juice10, do we have support for wsk CLI to download/install through brew?

@pritidesai nearly, I need to apply a patch to it and then it can be merged. I'm going to work on that today: https://github.com/Homebrew/homebrew-core/pull/31231#issuecomment-415289712

Hey @Juice10 please let me whenever we are ready to merge this PR.

Hey @pritidesai please merge this PR

Update: I uncommented the brew docs now that wsk has been accepted in Homebrew


Update android sample and add UTXOPlanner to test

Description
Added UTXOPlanner call to debug crash in Android

it works locally and in CI, I updated sample/android to use AnySigner and UTXOPlanner