Sphinx (search engine) explained

Sphinx
Developer:Andrew Aksyonoff
Operating System:Linux, Windows, Solaris, FreeBSD, NetBSD, Mac OS, AIX
Programming Language:C++
Genre:Search and index
License:GPLv2 until version 2 and commercial; proprietary since version 3

Sphinx is a fulltext search engine that provides text search functionality to client applications.

Overview

Sphinx can be used either as a stand-alone server or as a storage engine ("SphinxSE") for the MySQL family of databases.When run as a standalone server Sphinx operates similar to a DBMS and can communicate with MySQL, MariaDB and PostgreSQL through their native protocols or with any ODBC-compliant DBMS via ODBC.MariaDB, a fork of MySQL, is distributed with SphinxSE.[1]

SphinxAPI

If Sphinx is run as a stand-alone server, it is possible to use SphinxAPI to connect an application to it. Official implementations of the API are available for PHP, Java, Perl, Ruby and Python languages. Unofficial implementations for other languages, as well as various third party[2] plugins and modules are also available. Other data sources can be indexed via pipe in a custom XML format.[3]

SphinxQL

The Sphinx search daemon supports the MySQL binary network protocol and can be accessed with the regular MySQL API and/or clients. Sphinx supports a subset of SQL known as SphinxQL. It supports standard querying of all index types with SELECT, modifying RealTime indexes with INSERT, REPLACE, and DELETE, and more.

SphinxSE

Sphinx can also provide a special storage engine for MariaDB and MySQL databases. This allows those MySQL, MariaDB to communicate with Sphinx's searchd to run queries and obtain results. Sphinx indices are treated like regular SQL tables. The SphinxSE storage engine is shipped with MariaDB.

Full-text fields and indexing

Sphinx is configured to examine a data set via its Indexer. The Indexer process creates a full-text index (a special data structure that enables quick keyword searches) from the given data/text. Full-text fields are the resulting content that is indexed by Sphinx; they can be (quickly) searched for keywords. Fields are named, and you can limit your searches to a single field (e.g. search through "title" only) or a subset of fields (e.g. to "title" and "abstract" only). Sphinx's index format generally supports up to 256 fields. Note that the original data is not stored in the Sphinx index, but are discarded during the Indexing process; Sphinx assumes that you store those contents elsewhere.

Attributes

Attributes are additional values associated with each document that can be used to perform additional filtering and sorting during search. Attributes are named. Attribute names are case insensitive. Attributes are not full-text indexed; they are stored in the index as is. Currently supported attribute types are:

(since 1.10-beta);

(since 2.1.1-beta);[4] [5]

JSON attributes in Sphinx

Sphinx, like classic SQL databases, works with a so-called fixed schema, that is, a set of predefined attribute columns. These work well when most of the data stored actually has values: mapping sparse data to static columns can be cumbersome. Assume for example that you're running a price comparison or an auction site with many different products categories. Some of the attributes like the price or the vendor are identical across all goods. But from there, for laptops, you also need to store the weight, screen size, HDD type, RAM size, etc. And, say, for shovels, you probably want to store the color, the handle length, and so on. So it's manageable across a single category, but all the distinct fields that you need for all the goods across all the categories are legion. The JSON field can be used to overcome this. Inside the JSON attribute you don't need a fixed structure. You can have various keys which may or may not be present in all documents. When you try to filter on one of these keys, Sphinx will ignore documents that don't have the key in the JSON attribute and will work only with those documents that have it.

License

Up until version 3, Sphinx is dual licensed; either:

  1. GNU General Public License version 2 or
  2. proprietary licensing is available for use-cases which are not within the terms of the GNU GPLv2.

Since version 3, Sphinx has become proprietary, with a promise to release its source code in the future[6]

Sphinx use examples

Feature list

Performance and scalability

Fork

In 2017, key members of the original Sphinx team forked the project under the name Manticore,[17] with the intention of fixing bugs and developing new features.[18] Unlike Sphinx, Manticore continues to be released as open source under version 3 of the GPL.[19]

See also

Further reading

External links

Notes and References

  1. Web site: AskMonty: About SphinxSE . kb.askmonty.org . Monty Program AB . 2013-08-16.
  2. Web site: Sphinx Wiki: Third Party Tools . sphinxsearch.com . Sphinx Search Wiki . 2013-08-16.
  3. Web site: xmlpipe2 . sphinxsearch.com . Sphinx Search Documentation . 2013-08-16.
  4. Web site: JSON Attributes in Sphinx 2.1.1 . sphinxsearch.com . 7 February 2013 . Sphinx Search Blog . 2013-08-16.
  5. Web site: Full JSON Support in Trunk . sphinxsearch.com . 8 August 2013 . Sphinx Search Blog . 2013-08-16.
  6. Web site: Sphinx | Open Source Search Server.
  7. Web site: Sphinx at Craigslist . craigslist.org . Craigslist . 2013-08-17.
  8. Web site: GM Recruitment . aleph-networks.com . Aleph-networks . 2012-10-01.
  9. Web site: Lightning Fast PHP Site Search . tradebit.com . Tradebit . 2013-08-17.
  10. Web site: Sphinx Search beta for Vbulletin 4.0 . vbulletin.com . Vbulletin . 2013-08-17.
  11. Web site: Sphinx Search Extension for MediaWiki . mediawiki.org . MediaWiki: Svemir Brkic, Paul Grinberg . 2013-08-17.
  12. Web site: Powered by Sphinx Search: Boardreader . sphinxsearch.com . Sphinx Search . 2013-08-17.
  13. Web site: Powered by Sphinx. sphinxsearch.com/.
  14. Web site: About Sphinx . sphinxsearch.com . Sphinx Search . 2013-08-16.
  15. Web site: Powered by Sphinx . sphinxsearch.com . Sphinx Search . 2015-05-10.
  16. Web site: Craigslist: Factsheet . craigslist.org . Craigslist . 2013-08-16 . dead . https://web.archive.org/web/20120805110958/http://www.craigslist.org/about/factsheet . 5 August 2012.
  17. Web site: About Manticore Search. 2023-04-24. manticoresearch.com. en.
  18. Web site: Manticore Search: 3 years after forking from Sphinx. 2024-05-02. manticoresearch.com. en.
  19. Web site: manticoresoftware/manticoresearch. GitHub. 2024-05-02. en.