BitFunnel explained

BitFunnel
Developer:	Microsoft
Programming Language:	C++
Platform:	Windows, macOS, Ubuntu
Genre:	Search engine indexing algorithm
License:	MIT License

BitFunnel is the search engine indexing algorithm and a set of components used in the Bing search engine,^[1] which were made open source in 2016.^[2] BitFunnel uses bit-sliced signatures instead of an inverted index in an attempt to reduce operations cost.^[3]

History

Progress on the implementation of BitFunnel was made public in early 2016, with the expectation that there would be a usable implementation later that year.^[4] In September 2016, the source code was made available via GitHub. A paper discussing the BitFunnel algorithm and implementation was released as through the Special Interest Group on Information Retrieval of the Association for Computing Machinery in 2017 and won the Best Paper Award.^[5]

Components

BitFunnel consists of three major components:

• BitFunnel – the text search/retrieval system itself
• WorkBench – a tool for preparing text for use in BitFunnel
• NativeJIT – a software component that takes expressions that use C data structures and transforms them into highly optimized assembly code

Algorithm

Initial problem and solution overview

The BitFunnel paper describes the "matching problem", which occurs when an algorithm must identify documents through the usage of keywords. The goal of the problem is to identify a set of matches given a corpus to search and a query of keyword terms to match against. This problem is commonly solved through inverted indexes, where each searchable item is maintained with a map of keywords.

In contrast, BitFunnel represents each searchable item through a signature. A signature is a sequence of bits which describe a Bloom filter of the searchable terms in a given searchable item. The bloom filter is constructed through hashing through several bit positions.

Theoretical implementation of bit-string signatures

The signature of a document (D) can be described as the logical-or of its term signatures:

$$\backslash overrightarrow=\backslash bigcup\_\; \backslash overrightarrow$$

Similarly, a query for a document (Q) can be defined as a union:

$$\backslash overrightarrow=\backslash bigcup\_\; \backslash overrightarrow$$

Additionally, a document D is a member of the set M when the following condition is satisfied:

$$\backslash overrightarrow\; \backslash cap\; \backslash overrightarrow\; =\; \backslash overrightarrow$$

This knowledge is then combined to produce a formula where M is identified by documents which match the query signature:

$$M\text{'}=\; \backslash left\backslash$$

These steps and their proofs are discussed in the 2017 paper.

Pseudocode for bit-string signatures

This algorithm is described in the 2017 paper.$$\backslash beginM\text{'}=\backslash emptyset\; \backslash \backslash \backslash texttt\backslash \; D\; \backslash in\; C\backslash \; \backslash texttt\; \backslash \backslash \backslash qquad\; \backslash texttt\backslash \; \backslash overrightarrow\; \backslash cap\; \backslash overrightarrow\; =\; \backslash overrightarrow\backslash \; \backslash texttt\; \backslash \backslash \backslash qquad\; \backslash qquad\; M\text{'}\; =\; M\text{'}\; \backslash cup\; \backslash \; \backslash \backslash \backslash qquad\; \backslash texttt\; \backslash \backslash \backslash texttt\backslash end$$

External links

• BitFunnel · GitHub
• BitFunnel · BitFunnel

Notes and References

1. Web site: Microsoft open-sources Bing components for fast code compilation. Serdar. Yegulalp. September 6, 2016. InfoWorld.
2. Web site: Verma. Arpit. 2016-09-07. Microsoft Open Sources Major Components Of Bing Search Engine, Here's Why It Matters. 2020-06-12. Fossbytes. en-US.
3. Book: Goodwin. Bob. Hopcroft. Michael. Luu. Dan. Clemmer. Alex. Curmei. Mihaela. Elnikety. Sameh. He. Yuxiong. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval . BitFunnel . 2017-08-07. 605–614. New York, NY, USA. ACM. 10.1145/3077136.3080789. 978-1-4503-5022-8. free.
4. Web site: When will BitFunnel be usable? · BitFunnel. 2020-06-12. bitfunnel.org.
5. Web site: SIGIR Best Paper Awards . ACM . 8 July 2020.