![]() ![]() ![]() You can implement custom retention or backup policies.The size of your XCom information is not limited by the size of your metadata database.XComs don’t need to be JSON serializable because you can store the data however you want. ![]() XCom backends provide much more flexibility than you would have using traditional XComs with Airflow’s metadata database. #Xcom 2 demo build how toWhen to Use Custom XCom Backendsīefore we dive into the details on how to set up an XCom backend, we want to briefly touch on when you might want to do so and, importantly, when you might not. If you’re new to working with XComs or the TaskFlow API and want some background before diving into custom XCom Backends, check out the Airflow documentation on XComs and our webinar on using the TaskFlow API. This guide discusses the benefits of using an XCom backend, shows an example of implementing an XCom backend with S3, and describes how to set this up if you’re running Airflow on Astro. You can also implement your own serialization / deserialization methods to define how XComs are handled. This means that rather than store XComs in Airflow’s metadata database by default, you can push and pull XComs to and from an external system such as S3, GCS, or HDFS. One of the features of the TaskFlow API that increases the flexibility of XComs is support for a custom XCom backend. The TaskFlow API abstracts the task and dependency management layer away from users, which greatly improves the experience of working with XComs. One of the main new features in this release was the TaskFlow API, which was introduced to solve the challenge of explicitly passing messages between Airflow tasks. In December 2020, the Apache Airflow project took a huge step forward with the release of Airflow 2.0. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |