How to monitor Syncthing
Use the Syncthing REST API to alert when the synchronization needs attention
Syncthing monitoring
Syncthing is an awesome way to keep files synchronized between devices. It is designed to run in the background, so you configure it once and then forget about it. Which is great, but what happens when there is an error and synchronization stops? That's where monitoring comes into play.
This is usually a problem with backups also: you set it up, forget about it, and only check after a hardware failure and by that time it's too late to find out the automatic system failed years ago.
So I wanted a process that starts every now and then and takes a look at what Syncthing does and whether there are any potential problems that require investigation. I found it's not obvious what constitutes "a problem" in regards to Syncthing and in which case the monitoring should alert.
This article shows the various info Syncthing makes available and what part might indicate an error.
CLI vs REST API
Syncthing provides both a CLI and a REST API. I found that the latter is better documented and seems like a more developed way to get information about the internal state.
The REST API is available at http://localhost:8384/rest and to use it you need an API key. Fortunately, the CLI provides a simple way to get that from the
running process:
import {exec} from "child_process";
import util from "util";
const apiKey = (await util.promisify(exec)("syncthing cli config gui apikey get")).stdout.trim();
With this key, it's only a matter of sending HTTP requests to the REST endpoint and parsing the results:
import http from "http";
import https from "https";
const sendRequest = (url, options, body) => new Promise((res, rej) => {
const client = url.startsWith("https") ? https : http;
const req = client.request(url, options, (result) => {
const { statusCode } = result;
result.setEncoding("utf8");
let rawData = "";
result.on("data", (chunk) => { rawData += chunk; });
result.on("end", () => {
if (statusCode < 200 || statusCode >= 300) {
rej(rawData);
}else {
try {
const parsedData = JSON.parse(rawData);
res(parsedData);
} catch (e) {
rej(e);
}
}
});
}).on("error", (e) => {
rej(e);
});
if (body) {
req.write(body);
}
req.end();
});
const sendSyncthingRequest = (url) => sendRequest(url, {method: "GET", headers: {"X-API-Key": apiKey}});
Note: The above code does not use fetch as the current version of Node does not bundle it and I didn't want to add dependencies.
Exploring the API
All the endpoints fall into three categories:
- per-device
- per-folder
- and global
For monitoring, I'm interested in all three: any problem with a device, a folder, or any global errors should trigger a notification.
Per-folder
The list of folders is available under /config/folders:
[
{
"id": "<id1>",
"label": "...",
"filesystemType": "basic",
"path": "...",
"type": "sendreceive",
"paused": false,
"devices": [
...
],
...
}
]
The important part is the id as all other per-folder endpoints need the folder ID. It's also worth checking the paused property.
Statistics
Get folder statistics at /stats/folder:
{
"<id1>": {
"lastFile": {
"at": "2022-04-14T08:14:24Z",
"filename": "...",
"deleted": true
},
"lastScan": "2022-04-15T07:27:24Z"
},
}
The interesting part is the lastFile.at. I expect that files change in folders from time-to-time so it should alert me when the lastFile.at is many
days in the past.
Status
Then get the folder status at /db/status?folder=${folder.id}
{
"errors": 0,
"globalBytes": 191167322,
"globalDeleted": 24378,
"...",
"ignorePatterns": false,
"inSyncBytes": 191167322,
"inSyncFiles": 175,
"invalid": "",
"localBytes": 191167322,
"...",
"needBytes": 0,
"...",
"pullErrors": 0,
"receiveOnlyChangedBytes": 0,
"...",
"sequence": 34125,
"state": "idle",
"stateChanged": "2022-04-15T07:27:24Z",
"version": 34125
}
The important parts are:
globalBytesis the size of the folderpullErrorsindicate errorsstateChanged: the last change in the status of the folderreceiveOnlyChangedBytes: if the folder is receive-only, this indicates locally changed files. Since receive-only folders shouldn't be changed locally, anything > 0 should set off an alert
Errors
To get the folder errors, use the /folder/errors?folder=${folder.id} endpoint:
{
"errors": null,
"folder": "<id1>",
"page": 1,
"perpage": 65536
}
Here, the interesting part is the errors list. If it is non-null that indicates an error.
In code
The script that gets all the interesting information for folders:
const folderStats = await sendSyncthingRequest("http://localhost:8384/rest/stats/folder");
const folders = await Promise.all((await sendSyncthingRequest("http://localhost:8384/rest/config/folders")).map(async (folder) => {
const status = await sendSyncthingRequest(`http://localhost:8384/rest/db/status?folder=${folder.id}`);
const stats = folderStats[folder.id];
const errors = await sendSyncthingRequest(`http://localhost:8384/rest/folder/errors?folder=${folder.id}`);
return {
id: folder.id,
path: folder.path,
paused: folder.paused,
globalBytes: status.globalBytes,
receiveOnlyChangedBytes: status.receiveOnlyChangedBytes,
pullErrors: status.pullErrors,
stateChanged: new Date(status.stateChanged).getTime() > 0 ? new Date(status.stateChanged) : undefined,
lastFileAt: stats.lastFile && stats.lastFile.at && new Date(stats.lastFile.at).getTime() > 0 ? new Date(stats.lastFile.at) : undefined,
errors: errors.errors,
};
}));
Per-device
To get a list of devices, use the /config/devices endpoint:
[
{
"deviceID": "<deviceID>",
"name": "Mi A2",
"addresses": [
"dynamic"
],
"..."
"paused": false,
"...",
},
{
"...",
}
]
All the per-device calls will need the deviceID.
Also, the paused property indicates that the device is paused or not. It's worth monitoring this property.
Statistics
Get the device statistics at /stats/device:
{
"<device1>": {
"lastSeen": "2022-04-15T08:04:47Z",
"lastConnectionDurationS": 287.07541629
}
}
Same as the device statistics, if the device.lastSeen is further in the past that could indicate an error.
In code
The code that gets all the interesting information for devices:
const deviceStats = await sendSyncthingRequest("http://localhost:8384/rest/stats/device");
const devices = await Promise.all((await sendSyncthingRequest("http://localhost:8384/rest/config/devices")).map(async (device) => {
const stats = deviceStats[device.deviceID];
return {
deviceID: device.deviceID,
name: device.name,
paused: device.paused,
lastSeen: new Date(stats.lastSeen).getTime() > 0 ? new Date(stats.lastSeen) : undefined,
}
}));
Global
Get global errors at /system/error:
{
"errors": null
}
If the errors is non-null that indicates an error.
Pending
Apart from errors, pending devices and folders also require manual handling.
Get the pending devices at /pending/devices:
{}
And the pending folders at /pending/folders:
{}
If any of these objects are non-empty that means there is a pending device/folder waiting for configuration.
In code
const systemErrors = (await sendSyncthingRequest("http://localhost:8384/rest/system/error")).errors;
const pendingFolders = await sendSyncthingRequest("http://localhost:8384/rest/cluster/pending/folders");
const pendingDevices = await sendSyncthingRequest("http://localhost:8384/rest/cluster/pending/devices");
const global = {
globalErrors: [
systemErrors !== null ? JSON.stringify(systemErrors) : [],
Object.entries(pendingFolders).length > 0 ? JSON.stringify(pendingFolders) : [],
Object.entries(pendingDevices).length > 0 ? JSON.stringify(pendingDevices) : [],
].flat(),
sumGlobalBytes: folders.reduce((memo, {globalBytes}) => memo + globalBytes, 0),
};
