RocksDB explained

RocksDB
Author:Dhruba Borthakur
Developer:Meta Platforms (was Facebook, Inc.)
Programming Language:C++
Operating System:Windows, macOS, Linux, FreeBSD, OpenBSD, Solaris, AIX
Platform:Cross-platform
Genre:Embedded database
License:Apache 2.0 or GPL 2

RocksDB is a high performance[1] [2] [3] [4] [5] embedded database for key-value data. It is a fork of Google's LevelDB optimized to exploit multi-core processors (CPUs), and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads. It is based on a log-structured merge-tree (LSM tree) data structure. It is written in C++ and provides official language bindings for C++, C, and Java. Many third-party language bindings exist. RocksDB is free and open-source software, released originally under a BSD 3-clause license.[6] [7] [8] However, in July 2017 the project was migrated to a dual license of both Apache 2.0 and GPLv2 license.[9] This change helped its adoption in Apache Software Foundation's projects after blacklist of the previous BSD+Patents license clause.[10] [11]

RocksDB is used in production systems at various web-scale enterprises[12] including Facebook, Yahoo!,[13] and LinkedIn.[14]

Features

RocksDB, like LevelDB, stores keys and values in arbitrary byte arrays, and data is sorted byte-wise by key or by providing a custom comparator.

RocksDB provides all of the features of LevelDB, plus:

and others.[25]

RocksDB is not an SQL database (although MyRocks combines RocksDB with MySQL). Like other NoSQL and dbm stores, it has no relational data model, and it does not support SQL queries. Also, it has no direct support for secondary indexes, however a user may build their own internally using Column Families or externally. Applications use RocksDB as a library, as it provides no server or command-line interface.

History

RocksDB was created at Facebook by Dhruba Borthakur[26] [27] in April 2012, as a fork of LevelDB with the initial stated goal of improving performance for server workloads.[28] [29]

Integration

As an embeddable database, RocksDB can be used as a storage engine within a larger database management system (DBMS). For example, Rockset uses RocksDB[30] mostly for analytical data processing.

Alternative backend

The following projects have been started to replace or offer alternative storage engines for already-established database systems with RocksDB:

ArangoDB

ArangoDB has added RocksDB to its previous storage engine ("mmfiles").[31] RocksDB is be the default storage engine since ArangoDB 3.4.[32]

Cassandra

Cassandra on RocksDB can improve the performance of Apache Cassandra significantly (3–4 times faster in general, 100 times faster in some use-cases). The Instagram team at Facebook developed and open-sourced their code, along with benchmarks of their performance results.[33]

MariaDB

MariaDB can use the MyRocks storage engine (which is forked from RocksDB) since MariaDB 10.2.5 (Alpha status) [34] and stable since MariaDB 10.2.16 in 2018.[35]

MongoDB

The MongoRocks project provides a storage module for MongoDB where the storage engine is RocksDB.[36] [37] [38]

A related program is Rocks Strata, a tool written in Go, which allows managing incremental backups of MongoDB when RocksDB is used as the storage engine.[39]

MySQL

The MyRocks project created a new RocksDB-based storage engine for MySQL.[40] [41] In-depth details about MyRocks were presented at Percona Live 2016.[42]

Oxigraph

Oxigraph[43] is a graph database implementing the SPARQL standard, based on RocksDB

UKV

The UKV[44] project allows users to use RocksDB on par with LevelDB as the underlying key-value store. It represents a shared abstraction for create, read, update and delete (CRUD) operations common to every storage engine. It augments it with structured bindings for several high-level languages, including Python, Java, and Go.

Embedded

The following database systems and applications have chosen to use RocksDB as their embedded storage engine:

Ceph's BlueStore

The Ceph's BlueStore storage layer uses RocksDB for metadata management in OSD devices.[45]

Apache Flink

Apache Flink uses RocksDB to store checkpoints.[46]

FusionDB

FusionDB[47] uses RocksDB as its storage engine for XML, Key/Value, and JSON.[48]

LogDevice LogsDB

LogDevice's LogsDB is built atop RocksDB.[49]

Kafka Streams

Kafka Streams uses RocksDB for its state stores.[50]

Manhattan

The Manhattan Distributed Key-Value Store has used RocksDB as its primary engine to store Twitter data since 2018.[51]

Rockset

The Rockset[52] service that is used for operational data analytics uses RocksDB as its storage engine.[53]

SSDB

The ssdb-rocks[54] project uses RocksDB as the storage engine for the SSDB[55] NoSQL Database.

TiDB

The TiDB[56] project uses RocksDB as its storage engine.[57]

YugabyteDB

The YugabyteDB[58] database uses a modified version of RocksDB as part of its DocDB storage engine

Third-party language bindings

Third-party programming language bindings available for RocksDB include:

Notes and References

  1. Web site: Performance Benchmarks . . November 29, 2015.
  2. Web site: Benchmarking the leveldb family . 7 July 2014 . March 10, 2016.
  3. Web site: Comparing LevelDB and RocksDB, take 2 . 27 April 2015 . March 10, 2016.
  4. Web site: Benchmarking LevelDB vs. RocksDB vs. HyperLevelDB vs. LMDB Performance for InfluxDB . 20 June 2014 . March 10, 2016.
  5. Book: Golan-Gueta . Guy . Bortnikov . Edward . Hillel . Eschar . Keidar . Idit . Idit Keidar. Proceedings of the Tenth European Conference on Computer Systems . Scaling concurrent log-structured data stores . April 21, 2015 . 10.1145/2741948.2741973 . 1–14 . 9781450332385 . 5849146.
  6. Web site: Facebook's latest open source effort: a flash-powered database called RocksDB . 21 November 2013 . March 10, 2016 . 24 February 2020 . https://web.archive.org/web/20200224065917/https://gigaom.com/2013/11/21/facebooks-latest-open-source-effort-a-flash-powered-database-called-rocksdb/ . dead .
  7. Web site: Under the Hood: Building and open-sourcing RocksDB . . March 10, 2016.
  8. Web site: RocksDB - Facebook's Database Now Open Source . March 10, 2016.
  9. Web site: GitHub pull request . . July 20, 2017.
  10. Web site: Apache says 'no' to Facebook code libraries . . July 20, 2017.
  11. Web site: GitHub issue . . July 20, 2017.
  12. Web site: Users.md . . December 1, 2015.
  13. Web site: RocksDB on Steroids . March 10, 2016.
  14. Web site: Benchmarking Apache Samza: 1.2 million messages per second on a single node . March 10, 2016.
  15. Web site: RocksDB transactions . GitHub. 2016-04-04.
  16. Web site: How to backup RocksDB? . GitHub. 2017-07-19.
  17. Web site: Checkpoints . GitHub. 2017-07-19.
  18. Web site: Column families in RocksDB . GitHub. 2016-04-04.
  19. Web site: RocksDB bloom filters . GitHub. 2016-04-04.
  20. Web site: RocksDB TTL support . GitHub. 2016-04-04.
  21. Web site: Universal compaction . GitHub. 2016-04-04.
  22. Web site: RocksDB merge operator . GitHub. 2016-04-04.
  23. Web site: RocksDB perf context and IO stats context . GitHub. 2016-04-04.
  24. Web site: Spatial indexing in RocksDB . rocksdb.org. 2018-07-19.
  25. Web site: Features Not in LevelDB. GitHub.
  26. Web site: First commit where RocksDB diverges from LevelDB . . May 10, 2012 . March 15, 2016.
  27. Web site: Rocksdb readme file . . Nov 30, 2012 . March 15, 2016.
  28. Web site: The History of RocksDB . November 24, 2013 . March 10, 2016.
  29. Web site: RocksDB: A High Performance Embedded Key-Value Store for Flash Storage - Data@Scale . November 22, 2013 . March 10, 2016 . ... The story of why we decided to do RocksDB ... . Dhruba . Borthakur . YouTube.
  30. Web site: How We Use RocksDB at Rockset . Sandeep . Dhoot . 2019-06-27 . 2023-03-01 . rockset.com.
  31. Web site: Comparing new RocksDB and MMFiles storage engines. ArangoDB.
  32. Web site: RC1 ArangoDB 3.4 - Whats new? . 6 September 2018.
  33. Web site: Open-sourcing a 10x reduction in Apache Cassandra tail latency . 5 March 2018.
  34. Web site: MyRocks . MariaDB KnowledgeBase. 2019-04-28.
  35. Web site: MariaDB 10.2.16 Release Notes. MariaDB KnowledgeBase.
  36. Web site: mongodb-partners/mongo-rocks . . 29 October 2021.
  37. Web site: Integrating RocksDB with MongoDB . July 19, 2018.
  38. Web site: MongoDB + RocksDB at Parse . December 1, 2015.
  39. Web site: facebookgo/rocks-strata . . 31 October 2021.
  40. Web site: facebook/mysql-5.6 . . 2 November 2021.
  41. Web site: MyRocks: MySQL on RocksDB . November 29, 2015.
  42. Web site: MyRocks Deep Dive . 19 April 2016 . May 9, 2016.
  43. Web site: Oxigraph. Thomas. Pellissier Tanon. July 12, 2024. GitHub.
  44. Web site: unum-cloud/ukv . . 28 December 2022.
  45. Web site: Storage Devices -- Ceph Documentation . 2017-11-08 . 2020-02-24 . https://web.archive.org/web/20200224065932/https://docs.ceph.com/docs/luminous/rados/configuration/storage-devices/ . dead .
  46. Web site: Apache Flink 1.8 Documentation: State Backends . ci.apache.org. 2019-08-11.
  47. Web site: FusionDB. Evolved Binary.
  48. Web site: The Design and Implementation of FusionDB. XML Prague.
  49. Web site: LogDevice: a distributed data store for logs . 31 August 2017. Mark Marchukov, Facebook.
  50. Web site: Configuring a streams application . kafka.apache.org. 2024-03-11.
  51. Web site: Adopting RocksDB within Manhattan . . 28 December 2022.
  52. Web site: Rockset: Search and analytics database. rockset.com.
  53. Web site: How we use RocksDB at Rockset . rockset.com. en. 2019-07-10.
  54. Web site: ideawu/ssdb-rocks . . 21 August 2021.
  55. Web site: Home. ID-2Sbo.
  56. Web site: pingcap/tidb . . 4 November 2021.
  57. Web site: TiDB Internal (I) - Data Storage . 11 July 2017 . Shen Li.
  58. Web site: How We Built a High Performance Document Store on RocksDB? . 20 February 2019.
  59. Web site: warrenfalk/rocksdb-sharp . . 28 September 2021.
  60. Web site: RocksDB bindings for CHICKEN Scheme 5. wiki.call-cc.org . 2024-07-13.
  61. Web site: b1naryth1ef/rocksdb . . 22 October 2019.
  62. Web site: urbint/rox . . September 2021.
  63. Web site: leo-project/erocksdb . . September 2021.
  64. Web site: barrel-db / erlang-rocksdb · GitLab. GitLab.
  65. Web site: tecbot/gorocksdb . . 29 October 2021.
  66. Web site: rocksdb-haskell. Hackage.
  67. Web site: RocksJava . GitHub.
  68. Web site: rocksdb. 25 March 2022 .
  69. Web site: rocksdb. .
  70. Web site: iabudiab/ObjectiveRocks . . 2 August 2021.
  71. Web site: OCaml bindings for RocksDB . . 8 October 2021.
  72. Web site: An OCaml RocksDb binding using ocaml-ctypes . . 28 September 2020.
  73. Web site: RocksDB. MetaCPAN.
  74. Web site: Photonios/rocksdb-php . . 11 August 2021.
  75. Web site: "rocksdb" pack for SWI-Prolog. www.swi-prolog.org.
  76. Web site: stephan-hof/pyrocksdb . . 27 October 2021.
  77. Web site: rocksdb-ruby | RubyGems.org | your community gem host. rubygems.org.
  78. Web site: rust-rocksdb/rust-rocksdb . . 11 July 2024.

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia article "RocksDB".

Except where otherwise indicated, Everything.Explained.Today is © Copyright 2009-2024, A B Cryer, All Rights Reserved. Cookie policy.