How Syncthing provides secure file syncing without sharing your files with a third party
How to bring the benefits of the cloud back to your machines
Syncthing is an open-source project that implements secure file synchronization between machines. It's one of many such services, but the more I read about how it works the more I'm amazed how well it solves this quite hard problem without relying on the trust of anything "cloud".
In this article, you'll learn about how Syncthing works and how it brings back control over your data to your hand. Hopefully, by the end of this post, you'll be as amazed by Syncthing as I am.
File synchronization
Synchronizing folders between computers became mainstream with Dropbox. This is when you put an image into a shared directory and it automatically appears on the other machine in a few seconds. It's like magic and it's extremely convenient when you get the feeling of it.
Dropbox uses a central server to store your files and the clients connect to it. When one client detects a change it uploads it to the Dropbox server. Then if another client comes online it downloads all changed files.
This is a convenient setup as it relies on a central server that is globally reachable. The clients don't need a direct way to communicate which is usually a pain due to firewalls, NATs, and dynamic IP addresses. The central server solves this problem.
The downside is that this is a machine that Dropbox controls and has access to all your files. Moreover, Dropbox provides the client too, so even if it implements zero-trust encryption Dropbox can push an update to change that. So you need to trust Dropbox. Maybe it does not look into your files and takes good care of security. And maybe not, but you can do little beyond trusting it.
This is how most cloud services work. You need to trust the provider, and it is the way of business in many services. But when there are ways to eliminate this trust they are usually more secure alternatives. I never liked the idea that I'm sending files to a provider and only "company policy" stands in the way of it abusing them.
Of course, you can use an encrypted filesystem (like eCryptfs) and share only that with Dropbox, but it's a pain to configure and run. I've done that, but it's not a solution I'd recommend to anybody.
Encrypted synchronization
The next step I took was Seafile. This works similarly to Dropbox, but there are 3 important differences:
- First, it supports client-side encryption. This way the server can not look into the files.
- Equally important, the client is open source so that I can be reasonably sure that the encryption is implemented well.
- And third, the server is also available as open-source so I can install it on a machine I control
These solve the trust problem. I can run my own server, use the client from third-party repositories (such as the main Ubuntu repo) and set a password. This setup doesn't need any trust other than the packages used.
But running and maintaining my own server is a pain. There are hosted solutions, but using one changes the financial situation. It's no longer a decision between trust and no trust but between a free solution and a paid one.
Peer-to-peer synchronization
Syncthing provides p2p synchronization. There is no central server that provides a common connection point for the clients. Why it can work this way is due to how it implements communication between the peers.
Syncthing implements the TLS protocol, the same that the browser uses when you connect to websites via HTTPS. This solves the two problems with encrypted communication:
- It provides encryption with up-to-date algorithms
- It provides authentication so that the client can be sure that it communicates with the other client and no adversary can listen in
But when you configure HTTPS for your website, you need to prove access to a domain to get a valid certificate. For example, Let's Encrypt requires that you can write a file at a specific location, or the Amazon Certificate Manager needs that you put a CNAME record to the domain. How Syncthing can use TLS but without all these complications?
The browser trusts only a few root certificates but not the ones provided by the website. There is a chain of trust here where a certificate is signed by a trusted root, so it becomes trusted too. Then another certificate is signed by this trusted one, and so on. If the website can provide a chain of certificates that can reach a trusted root then the browser will trust this connection. You need to verify domain ownership to get a valid certificate from the issuer. The web works this way as it's not realistic to add certificates of websites manually to the trusted set.
In the case of Syncthing, it generates a public-private key pair when you install it. The Device ID which you need to connect two peers is generated from the public key. Syncthing doesn't need to verify ownership to a trusted third party because the trust of the other end is established with this exchange of Device IDs.
This solves the authentication part of the encryption. As a result, it does not matter how the two peers communicate they will be the only ones who can access the files. This is end-to-end encryption done right.
Relaying
An authenticated and encrypted connection between the peers opens the way to use relays, even ones that are not trusted. Syncthing maintains a list of public relays that anybody can use, effectively providing a middleman.
Relaying solves two hard problems. First, since the connection is authenticated it does not matter how the peers find each other. This allows a discovery service where both ends register and get the current IP address of the other. With this, you don't need to provide a static IP address or domain name. When you share a folder with a device, they will be able to find each other.
And second, a relay allows connections between peers even when they are not accessible to each other, such as when both of them are behind a NAT. They can both connect to the same relay and use that to forward the communication.
In practice, after you paired the devices using each others' Device ID, synchronization just happens. You don't need to worry about connectivity issues or static addresses. By default, they will use the discovery service to find each other and a relay if they can't connect directly. This setup is as easy-to-use as using a central server but without trusting your files with the provider.
Versioning
Syncthing also supports file versioning that defines how one end handles changes to existing files coming from the other end. You can use simple versioning so that you can recover overwritten (and deleted) files up to a fixed number of changes.
Or you can use the so-called staggered versioning that automatically deletes versions as they age, leaving fewer and fewer the longer they are stored. This tries to strike a balance between storage cost and recovery.
Keeping modified files is important. Ransomware overwrites files and a file synchronization solution would happily propagate these changes to the other ends. Without a way to store old versions, you'd lose access to your files even on other devices.
While versioning works for most cases, keep in mind that Syncthing is not a backup solution. It's a good idea to take a snapshot of the synced folders from time to time.
Client-side encryption at rest
A newer feature is untrusted devices. This means one end encrypts the files with a password so that even the other end can't access them. This feature is still experimental but it opens the way of using any device as a backup.
Just use a good password.
Conclusion
With Syncthing, you get all the benefits of a cloud-based file synchronization service but without the need to trust a third party with your data. This opens the way to bring a lot of things "back from the cloud" without losing the convenience.
Gmail provides a convenient way to reach your emails from any device. I remember when I used Thunderbird (it's still around, btw) to download my emails using POP3. This was before the smartphone era and it wasn't a priority to get emails to multiple devices and it also meant that without manually backing them up, my emails were lost when I moved from one computer to another. Now, when I get a new phone it configures itself from the old one automatically using cloud backups.
File synchronization brings the cloud experience to the local data. I can download my emails on one device, and access them on another because everything is stored in files that can be made identical over a network.
All that, without privacy concerns. That's why I'm excited about Syncthing.
