How to monitor Syncthing

Syncthing monitoring

Syncthing is an awesome way to keep files synchronized between devices. It is designed to run in the background, so you configure it once and then forget about it. Which is great, but what happens when there is an error and synchronization stops? That's where monitoring comes into play.

This is usually a problem with backups also: you set it up, forget about it, and only check after a hardware failure and by that time it's too late to find out the automatic system failed years ago.

So I wanted a process that starts every now and then and takes a look at what Syncthing does and whether there are any potential problems that require investigation. I found it's not obvious what constitutes "a problem" in regards to Syncthing and in which case the monitoring should alert.

This article shows the various info Syncthing makes available and what part might indicate an error.

CLI vs REST API

Syncthing provides both a CLI and a REST API. I found that the latter is better documented and seems like a more developed way to get information about the internal state.

The REST API is available at http://localhost:8384/rest and to use it you need an API key. Fortunately, the CLI provides a simple way to get that from the running process:

import {exec} from "child_process";
import util from "util";

const apiKey = (await util.promisify(exec)("syncthing cli config gui apikey get")).stdout.trim();

With this key, it's only a matter of sending HTTP requests to the REST endpoint and parsing the results:

import http from "http";
import https from "https";

const sendRequest = (url, options, body) => new Promise((res, rej) => {
	const client = url.startsWith("https") ? https : http;
	const req = client.request(url, options, (result) => {
		const { statusCode } = result;

		result.setEncoding("utf8");
		let rawData = "";
		result.on("data", (chunk) => { rawData += chunk; });
		result.on("end", () => {
			if (statusCode < 200 || statusCode >= 300) {
				rej(rawData);
			}else {
				try {
					const parsedData = JSON.parse(rawData);
					res(parsedData);
				} catch (e) {
					rej(e);
				}
			}
		});
	}).on("error", (e) => {
		rej(e);
	});
	if (body) {
		req.write(body);
	}
	req.end();
});

const sendSyncthingRequest = (url) => sendRequest(url, {method: "GET", headers: {"X-API-Key": apiKey}});

Note: The above code does not use fetch as the current version of Node does not bundle it and I didn't want to add dependencies.

Exploring the API

All the endpoints fall into three categories:

• per-device
• per-folder
• and global

For monitoring, I'm interested in all three: any problem with a device, a folder, or any global errors should trigger a notification.

Per-folder

The list of folders is available under /config/folders:

[
	{                             
		"id": "<id1>",
		"label": "...",
		"filesystemType": "basic",                                                         
		"path": "...",
		"type": "sendreceive",      
    "paused": false,
		"devices": [          
			...
		],                         
		...
	}
]

The important part is the id as all other per-folder endpoints need the folder ID. It's also worth checking the paused property.

Statistics

Get folder statistics at /stats/folder:

{
  "<id1>": {
    "lastFile": {
      "at": "2022-04-14T08:14:24Z",
      "filename": "...",
      "deleted": true
    },
    "lastScan": "2022-04-15T07:27:24Z"
  },
}

The interesting part is the lastFile.at. I expect that files change in folders from time-to-time so it should alert me when the lastFile.at is many days in the past.

Status

Then get the folder status at /db/status?folder=${folder.id}

{
  "errors": 0,
  "globalBytes": 191167322,
  "globalDeleted": 24378,
  "...",
  "ignorePatterns": false,
  "inSyncBytes": 191167322,
  "inSyncFiles": 175,
  "invalid": "",
  "localBytes": 191167322,
  "...",
  "needBytes": 0,
  "...",
  "pullErrors": 0,
	"receiveOnlyChangedBytes": 0,
	"...",
  "sequence": 34125,
  "state": "idle",
  "stateChanged": "2022-04-15T07:27:24Z",
  "version": 34125
}

The important parts are:

• globalBytes is the size of the folder
• pullErrors indicate errors
• stateChanged: the last change in the status of the folder
• receiveOnlyChangedBytes: if the folder is receive-only, this indicates locally changed files. Since receive-only folders shouldn't be changed locally, anything > 0 should set off an alert

Errors

To get the folder errors, use the /folder/errors?folder=${folder.id} endpoint:

{
  "errors": null,
  "folder": "<id1>",
  "page": 1,
  "perpage": 65536
}

Here, the interesting part is the errors list. If it is non-null that indicates an error.

In code

The script that gets all the interesting information for folders:

const folderStats = await sendSyncthingRequest("http://localhost:8384/rest/stats/folder");

const folders = await Promise.all((await sendSyncthingRequest("http://localhost:8384/rest/config/folders")).map(async (folder) => {
	const status = await sendSyncthingRequest(`http://localhost:8384/rest/db/status?folder=${folder.id}`);
	const stats = folderStats[folder.id];
	const errors = await sendSyncthingRequest(`http://localhost:8384/rest/folder/errors?folder=${folder.id}`);

	return {
		id: folder.id,
		path: folder.path,
		paused: folder.paused,
		globalBytes: status.globalBytes,
		receiveOnlyChangedBytes: status.receiveOnlyChangedBytes,
		pullErrors: status.pullErrors,
		stateChanged: new Date(status.stateChanged).getTime() > 0 ? new Date(status.stateChanged) : undefined,
		lastFileAt: stats.lastFile && stats.lastFile.at && new Date(stats.lastFile.at).getTime() > 0 ? new Date(stats.lastFile.at) : undefined,
		errors: errors.errors,
	};
}));

Per-device

To get a list of devices, use the /config/devices endpoint:

[
  {
    "deviceID": "<deviceID>",
    "name": "Mi A2",
    "addresses": [
      "dynamic"
    ],
    "..."
    "paused": false,
		"...",
  },
	{
		"...",
	}
]

All the per-device calls will need the deviceID.

Also, the paused property indicates that the device is paused or not. It's worth monitoring this property.

Statistics

Get the device statistics at /stats/device:

{
  "<device1>": {
    "lastSeen": "2022-04-15T08:04:47Z",
    "lastConnectionDurationS": 287.07541629
  }
}

Same as the device statistics, if the device.lastSeen is further in the past that could indicate an error.

In code

The code that gets all the interesting information for devices:

const deviceStats = await sendSyncthingRequest("http://localhost:8384/rest/stats/device");

const devices = await Promise.all((await sendSyncthingRequest("http://localhost:8384/rest/config/devices")).map(async (device) => {
	const stats = deviceStats[device.deviceID];

	return {
		deviceID: device.deviceID,
		name: device.name,
		paused: device.paused,
		lastSeen: new Date(stats.lastSeen).getTime() > 0 ? new Date(stats.lastSeen) : undefined,
	}
}));

Global

Get global errors at /system/error:

{
  "errors": null
}

If the errors is non-null that indicates an error.

Pending

Apart from errors, pending devices and folders also require manual handling.

Get the pending devices at /pending/devices:

{}

And the pending folders at /pending/folders:

{}

If any of these objects are non-empty that means there is a pending device/folder waiting for configuration.

In code

const systemErrors = (await sendSyncthingRequest("http://localhost:8384/rest/system/error")).errors;
const pendingFolders = await sendSyncthingRequest("http://localhost:8384/rest/cluster/pending/folders");
const pendingDevices = await sendSyncthingRequest("http://localhost:8384/rest/cluster/pending/devices");

const global = {
	globalErrors: [
		systemErrors !== null ? JSON.stringify(systemErrors) : [],
		Object.entries(pendingFolders).length > 0 ? JSON.stringify(pendingFolders) : [],
		Object.entries(pendingDevices).length > 0 ? JSON.stringify(pendingDevices) : [],
	].flat(),
	sumGlobalBytes: folders.reduce((memo, {globalBytes}) => memo + globalBytes, 0),
};