Syncthing Explained

Syncthing
Syncthing
Author:Jakob Borg
Developer:Jakob Borg et al.[1]
Programming Language:Go
Operating System:Linux, macOS, Windows, Android, BSD, Solaris
Language:English, German, Greek, Spanish, French, Italian, Polish, Portuguese, Russian, Swedish, Norwegian
Language Count:38
Language Footnote:[2]
Genre:File synchronization
License:MPL 2.0[3]

Syncthing is a free and open source peer-to-peer file synchronization application available for Windows, macOS, Linux, Android, Solaris, Darwin, and BSD.[4] It can sync files between devices on a local network, or between remote devices over the Internet. Data security and data safety are built into its design. Version 1.0 was released in January 2019 after five years in beta.[5]

Technology

Syncthing is written in Go and implements its own, equally free Block Exchange Protocol.[6]

It is a BYO cloud model where users provide the hardware it runs on. It supports IPv6 and, for those on IPv4 networks, NAT punching and relay. Devices that connect to each other require explicit approval (unless using the Introducer feature) which increases the security of the mesh. All data, whether transferred directly between devices or via relays, is encrypted using TLS.[7] [8]

Conflicts are handled with the older file being renamed with a "sync-conflict" suffix (along with time and date stamp), enabling the user to decide how to manage two or more files of the same name that have been changed between synching.[9] GUI Wrappers can use these files to present the user with a method of resolving conflicts without having to resort to manual file handling.

Efficient syncing is achieved via compression of metadata or all transfer data,[10] block re-use[11] and lightweight scanning[12] for changed files, once a full hash has been computed and saved. Syncthing offers send-only and receive-only folder types[13] where updates from remote devices are not processed, various types of file versioning[14] (trash can, simple or staggered versioning, and handing versioning to an external program or script) and file/path ignore patterns.[15] Two different SHA256 hash implementations are currently supported, the faster of which is used dynamically after a brief benchmark on startup.[16] Moving and renaming files and folders is handled efficiently, with Syncthing intelligently processing these operations rather than re-downloading data from scratch.[17]

Infrastructure

Device discovery is achieved via publicly-accessible discovery servers hosted by the project developers,[18] local (LAN) discovery via broadcast messages, device history and static host name/addressing. The project also provides the Syncthing Discovery Server[19] program for hosting one's own discovery servers, which can be used alongside or as a replacement for the public servers.

The network of community-contributed relay servers allows devices behind different IPv4 NAT firewalls to communicate by relaying encrypted data via a third party. The relay is similar to the TURN protocol, with the traffic TLS-encrypted end-to-end between devices (thus even the relay server cannot see the data, only the encrypted stream). Private relays can also be set up and configured, with or without public relays, if desired. Syncthing automatically switches from relaying to direct device-to-device connections if it discovers a direct connection has become available.[20]

Syncthing can be used without any connection to the project or community's servers:[21] upgrades, opt-in usage data, discovery and relaying can all be disabled or configured independently, thus the mesh and its infrastructure can all be run in a closed system for privacy or confidentiality.

Configuration and management

Syncthing can be configured via a web browser either locally or remotely (and supports access via proxy server), but it is also possible to edit the configuration file directly. The REST and Events APIs or one of the community-contributed wrapper programs.[22] Links to Docker images are also provided on the community contributions page, as well as links to supported configuration management solutions such as Puppet, Ansible and others.

Reception

History

Syncthing version history (part)
Date Version Major Changes
2023-09-251.25.0[30]
2022-05-041.20.0
2021-04-061.15.0
2020-09-151.10.0
2020-04-211.5.0
2019-10-01 1.3.0
  • New parameter for adjusting database size
  • Better database performance
2019-07-09 1.2.0
  • New transport protocol (QUIC)
  • Automatic crash reporting
2019-05-09 1.1.3
  • Hotfix for release to fix a panic bug on Windows
2019-04-021.1.1
  • Add support for TLS1.3
2019-04-221.1.0
  • User/group ownership follows parent dir
  • Fix hashing performance
2019-01-011.0.0
  • Limit max simultaneous scans
  • Show limit locally changed files for receive only folders
2016-06-19 0.14 "Dysprosium Dragonfly"[31]
  • new, extensible sync protocol (full rehashing of folder contents will happen on first restart)
2016-05-17 0.13 "Copper Cockroach"[32]
  • Serve files while downloading them
  • Unique folder ID separate from label
2015-11-05 0.12 "Beryllium Bedbug"[33]
  • Connection Relaying
  • Device Discovery over HTTPS

The initial public binary release (v0.2) was made on 30 December 2013.[34]

In October 2014 it was announced by the original author that Syncthing was being rebranded as "Pulse".[35] However, on November 17, the developer decided not to change Syncthing to Pulse and is no longer working with ind.ie. Ind.ie's Pulse is now an officially sanctioned fork of Syncthing.[36]

On 22 April 2015, 0.11.0 was released and it introduced conflict handling, language selection in the UI, CPU usage and synching speed improvements, Long filename support on Windows, automatic restarting when there is a problem for example the drive being inaccessible, and support for external versioning software.[37] 0.11 is not backwards compatible with older versions of Syncthing.[37] Because of changes to the REST API Syncthing clients that were on 0.10.x wouldn't automatically update to 0.11 as it wasn't compatible with a lot of the 3rd party integrations at the time of its release.[37]

0.13.0 like many of the older releases of Syncthing is incompatible with clients that are running version 0.12.x and below. 0.13.x separates the folder ids from folder labels. It also now has the ability to serve parts of the file that have already been downloaded to other clients while it is still downloading.[38]

1.0.0, codenamed Erbium Earthworm,[39] didn't really bring any major changes to the table. It was more of a reflection by the developers on the widespread use of the program and the fact that it had already been in development for almost 5 years at that point.[40] [41] Despite the change in the major number Jakob Borg, the lead developer, stated that it was otherwise identical to 0.14.55-rc.2[39]

Alongside the 1.0.0 release the team introduced a new semver-like versioning system with the following criteria:[42]

In 1.1.0 syncthing adopted Go 1.12 and as such loses compatibility with Windows XP and Windows Server 2003[43]

1.2.0 introduces support for QUIC, can now perform automatic crash reporting and deprecates small / fixed blocks. 1.2.0 also dropped support for communicating with Syncthing clients that are running 0.14.45 or older.[44]

1.8.0 adds an experimental folder option that allows users to specify how file changes should be saved on Copy-on-write file systems and also adds TCP hole punching support.[45]

1.9.0 introduced the option caseSensitiveFS that allowed users to disable the newly added handling for case insensitive filesystems.[46]

The 1.10.0 release gave users the ability to toggle whether they would like LAN IPs to be broadcast to the global discovery network.[47]

See also

Notes and References

  1. Web site: AUTHORS. GitHub. 7 April 2016.
  2. Web site: The syncthing translation project on Transifex . www.transifex.com . 24 August 2020 . en.
  3. Web site: LICENSE. GitHub. 7 April 2016.
  4. Web site: Let Syncthing turn your desktop into a local cloud for your mobile device. Jack. Wallen. 5 January 2016. TechRepublic. CBS Interactive.
  5. Web site: 2019-01-03 . Syncthing 1.0.0 released as open-source P2P sync tool, finally leaves beta . 2023-10-18 . BetaNews . en.
  6. Web site: Block Exchange Protocol v1. 7 April 2016.
  7. Web site: Security Principles — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  8. Web site: Relaying — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  9. Web site: FAQ — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  10. Web site: FAQ — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  11. Web site: Understanding Synchronization — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  12. Web site: Understanding Synchronization — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  13. Web site: Folder Types . 2020-07-09 .
  14. Web site: File Versioning — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  15. Web site: Ignoring Files — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  16. Web site: cmd/syncthing: Add selectable sha256 package (fixes #3613, fixes #3614) . Github.com . 2016-09-22 . 2017-01-04.
  17. Web site: FAQ — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  18. Web site: Understanding Device IDs — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  19. Web site: Syncthing Discovery Server — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  20. Web site: Relaying — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  21. Web site: Security Principles — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  22. Web site: Community Contributions — Syncthing v0.14 documentation . Docs.syncthing.net . 2017-01-04.
  23. Web site: Security Now! Episode 456. Grc.com. 4 November 2014.
  24. Web site: Security Now! Episode 603. Grc.com. 7 February 2019.
  25. Web site: Security Now! Episode 698. Grc.com. 7 February 2019.
  26. Web site: Security Now! Episode 727. Grc.com. 14 August 2019.
  27. Web site: Security Now! Episode 734 on YouTube. YouTube.com. 3 October 2019.
  28. Web site: Security Now! Episode 781 on YouTube. YouTube.com. 1 September 2020.
  29. Web site: Syncing all the things. 7 September 2021.
  30. Web site: Versions & Releases — Syncthing documentation . 2023-10-18 . docs.syncthing.net.
  31. Web site: 0.14. Github.com. 19 June 2016.
  32. Web site: 0.13. Github.com. 17 May 2016.
  33. Web site: 0.12. 14 April 2016.
  34. Web site: Releases. 30 December 2013. GitHub.
  35. Web site: Introducing Pulse, and ind.ie. Jakob. Borg. 9 October 2014. 7 April 2016.
  36. Web site: Syncthing is still Syncthing. Jakob. Borg. 17 November 2014. 7 April 2016.
  37. Web site: Syncthing v0.11.0 Release Notes . Syncthing Community Forum . 15 October 2020 . 14 April 2015.
  38. Web site: Release v0.13.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.
  39. Web site: Syncthing graduation day . Syncthing Community Forum . 15 October 2020 . 1 January 2019.
  40. Web site: Release v1.0.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.
  41. Web site: Syncthing 1.0.0 released as open-source P2P sync tool, finally leaves beta . BetaNews . 15 October 2020 . en . 3 January 2019.
  42. Web site: Versions & Releases — Syncthing v1 documentation . docs.syncthing.net . 15 October 2020.
  43. Web site: Release v1.1.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.
  44. Web site: Release v1.2.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.
  45. Web site: Release v1.8.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.
  46. Web site: Release v1.9.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.
  47. Web site: Release v1.10.0 · syncthing/syncthing . GitHub . 15 October 2020 . en.