A Pandas-compatible dataframe API for Beam

Iñigo González

Jun 19, 2020

I miss Pandas very useful i/o operations in this list like .read_csv(), to_csv(). or to_sql().

I suggest to either leave them out by now (and document how feed data into it) or implement just the most straightforward ones.

At first sight pandas.DataFrame.to_sql() seems hard to port; but pandas.DataFrame.read_gbq() seems much easier since we already have BigQuery io.

Robert Bradshaw

Aug 15, 2020

Yes, I was just thinking about how to implement read_csv (and other IOs) the other day; one interesting question is how to deal with (liquid) sharding.

The current offering requires providing the input as a PCollection, but this is certainly in the roadmap.

Robert Bradshaw

Sep 21, 2020

This was implemented at https://github.com/apache/beam/pull/12841 , with follow-up at https://github.com/apache/beam/pull/12844 (and more to come).

2 replies

Rui Wang

Apr 20, 2020

Beam SQL has received a similar request. E.g. users asked a support of https://cloud.google.com/bigquery/docs/reference/standard-sql/numbering_functions#row_number

Kenneth Knowles

Apr 20, 2020

Does pandas have tests we can crib?

Kenneth Knowles

Apr 20, 2020

Wait now I'm confused, since that seems to be point 2, so what is this point?

Robert Bradshaw

Apr 20, 2020

We have less control over the tests in point (2), and they may be harder to run (or at least have a lot more indirection in setting them up).

Kenneth Knowles

Apr 22, 2020

I see. Fair enough.

3 replies

You're suggesting