In computing, the count–min sketch (CM sketch) is a probabilistic data structure that serves as a frequency table of events in a stream of data. It uses hash functions to map events to frequencies, but unlike a hash table uses only sub-linear space, at the expense of overcounting some events due to collisions. The count–min sketch was invented in 2003 by Graham Cormode and S. Muthu Muthukrishnan[1] and described by them in a 2005 paper.[2]
Count–min sketch is an alternative to count sketch and AMS sketch and can be considered an implementation of a counting Bloom filter (Fan et al., 1998[3]) or multistage-filter.[1] However, they are used differently and therefore sized differently: a count–min sketch typically has a sublinear number of cells, related to the desired approximation quality of the sketch, while a counting Bloom filter is more typically sized to match the number of elements in the set.
The goal of the basic version of the count–min sketch is to consume a stream of events, one at a time, and count the frequency of the different types of events in the stream. At any time, the sketch can be queried for the frequency of a particular event type from a universe of event types
l{U}
The actual sketch data structure is a two-dimensional array of columns and rows. The parameters and are fixed when the sketch is created, and determine the time and space needs and the probability of error when the sketch is queried for a frequency or inner product. Associated with each of the rows is a separate hash function; the hash functions must be pairwise independent. The parameters and can be chosen by setting and, where the error in answering a query is within an additive factor of with probability (see below), and is Euler's number.
When a new event of type arrives we update as follows: for each row of the table, apply the corresponding hash function to obtain a column index . Then increment the value in row, column by one.
Several types of queries are possible on the stream.
\hatai=minjcount[j,hj(i)]
count
Obviously, for each, one has
ai\leq\hatai
ai
Additionally, this estimate has the guarantee that
\hatai\leqai+\varepsilonN
1-\delta
N=\sumi
counta
countb
Small modifications to the data structure can be used to sketch other different stream statistics.
Like the count sketch, the Count–min sketch is a linear sketch. That is, given two streams, constructing a sketch on each stream and summing the sketches yields the same result as concatenating the streams and constructing a sketch on the concatenated streams. This makes the sketch mergeable and appropriate for use in distributed settings in addition to streaming ones.
One potential problem with the usual min estimator for count–min sketches is that they are biased estimators of the true frequency of events: they may overestimate, but never underestimate the true count in a point query. Furthermore, while the min estimator works well when the distribution is highly skewed, other sketches such as the Count sketch based on means are more accurate when the distribution is not sufficiently skewed. Several variations on the sketch have been proposed to reduce error and reduce or eliminate bias.[4]
To remove bias, the estimatorrepeatedly randomly selects d random entries in the sketch and takes the minimum to obtain an unbiased estimate of the bias and subtracts it off.
A maximum likelihood estimator (MLE) was derived in Ting.[5] By using the MLE, the estimator is always able to match or better the min estimator and works well even if the distribution is not skewed. This paper also showed the hCount* debiasing operation is a bootstrapping procedure that can be efficiently computed without random sampling and can be generalized to any estimator.
Since errors arise from hash collisions with unknown items from the universe, several approaches correct for the collisions when multiple elements of the universe are known or queried for simultaneously [6] . For each of these, a large proportion of the universe must be known to observe a significant benefit.
Conservative updating changes the update, but not the query algorithms. To count instances of event type, one first computes an estimate
\hatai=minjcount[j,hj(i)]
count[j,hj(i)]\leftarrowmax\{count[j,hj(i)],\hat{ai}+c\}