Use insert_job in the BigQueryToGCPOpertor and adjust links - #24416
Conversation
turbaszek
left a comment
There was a problem hiding this comment.
One comment regarding persisting links. Otherwise looks good to me.
| conf = job["configuration"]["extract"]["sourceTable"] | ||
|
|
||
| try: | ||
| self.log.info(f"Executing: {configuration}") |
There was a problem hiding this comment.
We should use %-formatting in logging so the formatting is deferred until the latest possible moment.
| if job_type in job.to_api_repr()["configuration"]: | ||
| for table_prop in tables_prop: | ||
| if table_prop in job.to_api_repr()["configuration"][job_type]: | ||
| table = job.to_api_repr()["configuration"][job_type][table_prop] |
There was a problem hiding this comment.
Do we need to write job.to_api_repr()["configuration"] 3 times? Could you please introduce a variable for this value?
|
Ah. How could I missed it ! |
|
Yeah. I think that last comment from @turbaszek and we can merge (and release 8.0.1) |
|
Soory @lwyszomi - it TOTALLY got under the radar - not sure how, I was waiting for that PR! |
|
Ah yeah. there are also few comments from @josh-fell - so @lwyszomi -> if you can address them tomorrow morning, I am happy to wait. |
|
@potiuk I pushed fixes for @josh-fell and @turbaszek comments. |
All addressed and we want to release quickly :)
Cherry-picked changes from the community: * apache/airflow#24416 * apache/airflow#23078 Unit tests executed: https://screenshot.googleplex.com/5AJvqmrXwoUA83L Dags executed: https://screenshot.googleplex.com/67ABr73jTfZETb7 Change-Id: I6f4795eed4a8ecc67b0f7997f228cd92d464d004 GitOrigin-RevId: 0fa3a13797a21cd5d6c3e08d315bb93f7af4e787
Refactor of the BigQueryToGCPOperator, now we using
insert_jobmethod notrun_extract. Also adjustment for the links in the BigQueryInsertJobOperatorFYI @potiuk
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named
{pr_number}.significant.rst, in newsfragments.