Normalized compression distance explained

Normalized compression distance (NCD) is a way of measuring the similarity between two objects, be it two documents, two letters, two emails, two music scores, two languages, two programs, two pictures, two systems, two genomes, to name a few. Such a measurement should not be application dependent or arbitrary. A reasonable definition for the similarity between two objects is how difficult it is to transform them into each other.

It can be used in information retrieval and data mining for cluster analysis.

Information distance

We assume that the objects one talks about are finite strings of 0s and 1s. Thus we mean string similarity. Every computer file is of this form, that is, if an object is a file in a computer it is of this form. One can define the information distance between strings

x and y as the length of the shortest program p that computes x from y and vice versa. This shortest program is in a fixed programming language. For technical reasons one uses the theoretical notion of Turing machines. Moreover, to express the length of p one uses the notion of Kolmogorov complexity. Then, it has been shown ^[1] |p| = max \{K(x\mid y),K(y\mid x)\} up to logarithmic additive terms which can be ignored. This information distance is shown to be a metric(it satisfies the metric inequalities up to a logarithmic additive term), is universal (it minorizes every computable distance as computed for example from features up to a constant additive term).^[1]

Normalized information distance (similarity metric)

The information distance is absolute, but if we want to express similarity, then we are more interested in relative ones. For example, if two strings of length 1,000,000 differ by 1000 bits, then we consider that those strings are relatively more similar than two strings of 1000 bits that differ by 1000 bits. Hence we need to normalize to obtain a similarity metric. This way one obtains the normalized information distance (NID),

NID(x,y) = (max \{K{(x\mid y))/(,K{(y\mid) x)}\} }{ max \{K(x),K(y)\}}, where K(x\mid y) is algorithmic information of x given y as input. The NID is called `the similarity metric.' since the function NID(x,y) has been shown to satisfy the basic requirements for a metric distance measure.^{[2] [3]} However, it is not computable or even semicomputable.^[4]

Normalized compression distance

While the NID metric is not computable, it has an abundance of applications. Simply approximating

K by real-world compressors, with Z(x) is the binary length of the file x compressed with compressor Z (for example "gzip", "bzip2", "PPMZ") in order to make NID easy to apply. Vitanyi and Cilibrasi rewrote the NID to obtain the Normalized Compression Distance (NCD)NCD _{Z(x,y)} = (Z(xy) -min \{Z(x),Z(y)\)/() }. ^[3] The NCD is actually a family of distances parametrized with the compressor Z. The better Z is, the closer the NCD approaches the NID, and the better the results are.^[3]

Applications

The normalized compression distance has been used to fully automatically reconstruct language and phylogenetic trees.^{[2] [3]} It can also be used for new applications of general clustering and classification of natural data in arbitrary domains,^[3] for clustering of heterogeneous data,^[3] and for anomaly detection across domains.^[5] The NID and NCD have been applied to numerous subjects, including music classification,^[3] to analyze network traffic and cluster computer worms and viruses,^[6] authorship attribution,^[7] gene expression dynamics,^[8] predicting useful versus useless stem cells,^[9] critical networks,^[10] image registration,^[11] question-answer systems.^[12]

Performance

Researchers from the datamining community use NCD and variants as "parameter-free, feature-free" data-mining tools.^[5] One group have experimentally tested a closely related metric on a large variety of sequence benchmarks. Comparing their compression method with 51 major methods found in 7 major data-mining conferences over the past decade, they established superiority of the compression method for clustering heterogeneous data, and for anomaly detection, and competitiveness in clustering domain data.

NCD has an advantage of being robust to noise.^[13] However, although NCD appears "parameter-free", practical questions include which compressor to use in computing the NCD and other possible problems.^[14]

Comparison with the Normalized Relative Compression (NRC)

In order to measure the information of a string relative to another there is the need to rely on relative semi-distances (NRC).^[15] These are measures that do not need to respect symmetry and triangle inequality distance properties. Although the NCD and the NRC seem very similar, they address different questions. The NCD measures how similar both strings are, mostly using the information content, while the NRC indicates the fraction of a target string that cannot be constructed using information from another string. For a comparison, with application to the evolution of primate genomes, see.^[16]

Normalized Google distance

See main article: Normalized Google distance.

Objects can be given literally, like the literal four-letter genome of a mouse, or the literal text of War and Peace by Tolstoy. For simplicity we take it that all meaning of the object is represented by the literal object itself. Objects can also be given by name, like "the four-letter genome of a mouse," or "the text of `War and Peace' by Tolstoy." There are also objects that cannot be given literally, but only by name, and that acquire their meaning from their contexts in background common knowledge in humankind, like "home" or "red." We are interested in semantic similarity. Using code-word lengths obtained from the page-hit counts returned by Google from the web, we obtain a semantic distance using the NCD formula and viewing Google as a compressor useful for data mining, text comprehension, classification, and translation. The associated NCD, called the normalized Google distance (NGD) can be rewritten as

NGD(x,y) = (max \{ log f(x), log f(y)\)/(-) log f(x,y) }{ log N - min \{ log f(x), log f(y) \}}, where f(x) denotes the number of pages containing the search term x , and f(x,y) denotes the number of pages containing both x and y ,) as returned by Google or any search engine capable of returning an aggregate page count. The number N can be set to the number of pages indexed although it is more proper to count each page according to the number of search terms or phrases it contains. As rule of the thumb one can multiply the number of pages by, say, a thousand...^[17]

See also

• Word2vec

External links

• Efficient Estimation of Word Representations in Vector Space
• M. Li and P. Vitanyi, An Introduction to Kolmogorov Complexity and Its Applications, Springer-Verlag, New York, 4th Edition 2019

Notes and References

1. https://ieeexplore.ieee.org/document/681318/ C.H. Bennett, P. Gacs, M. Li, P.M.B. Vitányi, and W. Zurek, Information Distance, IEEE Trans. Inform. Theory, IT-44:4(1998) 1407–1423
2. M. Li, X. Chen, X. Li, B. Ma, P.M.B. Vitanyi, The similarity metric, IEEE Trans. Inform. Th., 50:12(2004), 3250–3264 . IEEE Transactions on Information Theory . 50 . 12 . 3250–3264 . 10.1109/TIT.2004.838101 . 2011-09-27 . Li . Ming . Chen . Xin . Li . Xin . Ma . Bin . Vitanyi . P. M. B. . 221927 .
3. R. Cilibrasi, P.M.B. Vitanyi, Clustering by compression . IEEE Transactions on Information Theory . 51 . 4 . 1523–1545 . 10.1109/TIT.2005.844059 . 2011-09-27 . Cilibrasi . R. . Vitanyi . P. M. B. . cs/0312044 . 911 .
4. 10.1016/j.jcss.2010.06.018 . 77 . 4 . Nonapproximability of the normalized information distance . 2011 . Journal of Computer and System Sciences . 738–742 . Terwijn . Sebastiaan A. . Torenvliet . Leen . Vitányi . Paul M.B.. 10831035 . free . 2066/92129 . free .
5. Book: E. Keogh, S. Lonardi, and C.A. Ratanamahatana. "Towards parameter-free data mining." In Conference on Knowledge Discovery in Data: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, vol. 22, no. 25, pp. 206–215. 2004 . 206 . 10.1145/1014052.1014077 . Dl.acm.org . 2004-08-22 . Towards parameter-free data mining . Keogh . Eamonn . Lonardi . Stefano . Ratanamahatana . Chotirat Ann . 978-1581138887 . 1616057 .
6. S. Wehner, Analyzing worms and network traffic using compression, Journal of Computer Security, 15:3(2007), 303–320 . Iospress.metapress.com . 2012-11-03.
7. A survey of modern authorship attribution methods . 10.1002/asi.21001. 60 . 3. Journal of the American Society for Information Science and Technology . 538–556 . 2009 . Stamatatos . Efstathios. 10.1.1.207.3310.
8. Gene expression dynamics in the macrophage exhibit criticality. 10.1073/pnas.0711525105 . 18250330. 105 . 6. Proceedings of the National Academy of Sciences . 1897–1900 . 2008 . Nykter . M.. 2538855 . 2008PNAS..105.1897N . free .
9. Computational prediction of neural progenitor cell fates. 7 . 3. 10.1038/nmeth.1424 . 20139969. Nature Methods . 213–218 . 2010 . Cohen . Andrew R. 1866/4484. 18652440 . free .
10. M. Nykter, N.D. Price, A. Larjo, T. Aho, S.A. Kauffman, O. Yli-Harja1, and I. Shmulevich, Critical networks exhibit maximal information diversity in structure-dynamics relationships, Phys. Rev. Lett. 100, 058702 (2008) . Physical Review Letters . 100 . 5 . 058702 . 10.1103/PhysRevLett.100.058702 . 18352443 . 2008 . Nykter . Matti . Price . Nathan D. . Larjo . Antti . Aho . Tommi . Kauffman . Stuart A. . Yli-Harja . Olli . Shmulevich . Ilya . 0801.3699 . 5760862 .
11. Book: M. Feixas, I. Boada, M. Sbert, Compression-based Image Registration. Proc. IEEE International Symposium on Information Theory, 2006. 436–440 . 436–440 . 10.1109/ISIT.2006.261706 . . July 2006. Compression-based Image Registration . Bardera . Anton . Feixas . Miquel . Boada . Imma . Sbert . Mateu . 978-1-4244-0505-3 . 10256/3052 . 12549455 .
12. Book: X Zhang, Y Hao, X Zhu, M Li, Information distance from a question to an answer, Proc. 13th ACM SIGKDD international conference on Knowledge discovery and data mining, 2007, 874–883 . 874 . 10.1145/1281192.1281285 . Dl.acm.org . 9781595936097 . Information distance from a question to an answer . 2007 . Zhang . Xian . Hao . Yu . Zhu . Xiaoyan . Li . Ming . Cheriton . David R. . 3040254 .
13. The normalized compression distance is resistant to noise . IEEE Transactions on Information Theory . 53 . 5 . 1895–1900 . 10.1109/TIT.2007.894669 . 2011-09-27 . Cebrian . M. . Alfonseca . M. . Ortega . A. . 10.1.1.158.2463 . 15691266 .
14. Web site: M. Cebrián, M. Alfonseca, and A. Ortega, Common pitfalls using the normalized compression distance: what to watch out for in a compressor, Commun. Inf. Syst. 5:4(2005), 367–384 . Projecteuclid.org . 2012-11-03.
15. A measure of relative entropy between individual sequences with application to universal classification. 39. 4. 10.1109/18.243444 . IEEE Transactions on Information Theory . 1270–1279 . 1993 . Ziv . J. . Merhav . N..
16. Comparison of Compression-Based Measures with Application to the Evolution of Primate Genomes. 20. 6. 10.3390/e20060393 . 2018Entrp..20..393P. Entropy . 393 . 2018 . Pratas . Diogo . Silva . Raquel M. . Pinho . Armando J.. 33265483. 7512912. free . Material was copied from this source, which is available under a Creative Commons Attribution 4.0 International License.
17. R.L. Cilibrasi, P.M.B. Vitanyi, The Google Similarity Distance, IEEE Trans. Knowledge and Data Engineering, 19:3(2007), 370-383 . IEEE Transactions on Knowledge and Data Engineering . 19 . 3 . 370–383 . 10.1109/TKDE.2007.48 . 2011-09-27 . Cilibrasi . R. L. . Vitanyi . P. M. B. . cs/0412098 . 59777 .