Apache Iceberg Explained

Apache Iceberg
Author:Ryan Blue, Daniel Weeks
Programming Language:Java, Python
Operating System:Cross-platform
Genre:Data warehouse, Data lake
License:Apache License 2.0
Website:

    Apache Iceberg is an open-source high-performance format for huge analytic tables. Iceberg enables the use of SQL tables for big data while making it possible for engines like Spark, Trino, Flink, Presto, Hive, Impala, StarRocks, Doris, and Pig to safely work with the same tables, at the same time.[1] Iceberg is released under the Apache License.[2] Iceberg addresses the performance and usability challenges of using Apache Hive tables in large and demanding data lake environments.[3] Vendors currently supporting Apache Iceberg tables in their products include CelerData, Cloudera, Dremio, IOMETE, Snowflake, Starburst, Tabular,[4] and AWS.[5]

    History

    Iceberg was started at Netflix by Ryan Blue and Dan Weeks. Hive was used by many different services and engines in the Netflix infrastructure. Hive was never able to guarantee correctness and did not provide stable atomic transactions.[3] Many at Netflix avoided using these services and making changes to the data to avert unintended consequences from the Hive format.[3] Ryan Blue set out to address three issues that faced the Hive table by creating Iceberg:[3]

    1. Ensure the correctness of the data and support ACID transactions.
    2. Improve performance by enabling finer-grained operations to be done at the file granularity for optimal writes.
    3. Simplify and obfuscate general operation and maintenance of tables.

    Iceberg development started in 2017.[6] The project was open-sourced and donated to the Apache Software Foundation in November 2018.[7] In May 2020, the Iceberg project graduated to become a top-level Apache project.[7]

    Iceberg is used by multiple companies including Airbnb,[8] Apple,[3] Expedia,[9] LinkedIn,[10] Adobe,[11] Lyft, and many more.[12]

    See also

    Notes and References

    1. Web site: Apache Iceberg. iceberg.apache.org. 5 October 2022.
    2. Web site: apache/iceberg GitHub License. The Apache Software Foundation. 5 October 2022. 5 October 2022.
    3. Web site: Woodie. Alex. Apache Iceberg: The Hub of an Emerging Data Service Ecosystem?. Datanami. 8 February 2021.
    4. Web site: Vendors. 2023-05-05. iceberg.apache.org.
    5. Web site: Using Apache Iceberg tables – Amazon Athena. Amazon Web Services, Inc..
    6. Web site: Initial public release in apache/iceberg. GitHub. 5 October 2022. en.
    7. Web site: Incubation Status Template - Apache Incubator. incubator.apache.org.
    8. Web site: Zhu. Ronnie. Upgrading Data Warehouse Infrastructure at Airbnb. The Airbnb Tech Blog. en. 26 September 2022.
    9. Web site: Mathiesen. Christine. A Short Introduction to Apache Iceberg. Expedia Group Technology. 5 October 2022. en. 26 January 2021.
    10. Web site: FastIngest: Low-latency Gobblin with Apache Iceberg and ORC format. engineering.linkedin.com. en.
    11. Web site: Bremner. Jaemi. Iceberg at Adobe. Medium. en. 3 December 2020.
    12. Web site: Council. Data. Open Source Highlight: Apache Iceberg. www.datacouncil.ai. 5 October 2022. en-ie.

    This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article "Apache Iceberg".

    Except where otherwise indicated, Everything.Explained.Today is © Copyright 2009-2024, A B Cryer, All Rights Reserved. Cookie policy.