Configure run detection and uploads

Tune three behaviors that decide what becomes a run and when bytes leave the instrument PC: run detection, stability period, and upload mode. Field-by-field schema lives in Watcher configuration; this page is the intent guide.

Admins can change these from the web app with watcher 1.4.0 and later: on the watcher’s Configuration tab, or with Edit watcher config on the instrument page. See Change a watcher’s config. On the PC, set them during data-hub-watcher init, with data-hub-watcher config edit, or by editing ~/.data-hub/config.yaml then syncing.

Group files into runs

A run is one acquisition: a logical group of files the watcher reports together. Run detection maps each file’s path (relative to the watch directory) to a run ID.

Common choices:

• Prefix from filename: a regex capture, for example everything before the first underscore so RUN001_data.csv → RUN001
• One subdirectory per run: each folder under the watch directory is its own run, so RUN001/data.csv → RUN001

Presets and custom regex options are listed under Run detection. Set recursive: true when files live in subfolders; otherwise only the top level is watched.

When a run gathers files from several folders, two files can share a name, such as a run.json in each capture folder. Watcher 1.1.0 and later keep every copy and add a short tag to the later names, such as sample~3f9a1c2b.tif. Older watchers kept only the first; see Files with the same name were skipped.

If files appear on disk but never become runs, check for pattern_mismatch events on the watcher detail page and run data-hub-watcher watch --dry-run.

Wait until writes finish

Stability period is how many seconds a file must keep the same size and modification time before the watcher treats it as fully written. Default is 5 seconds (allowed range 1–300). Raise it for instruments that write large files slowly.

A file that keeps changing past max_stability_wait_seconds (default 300, or 5 minutes) is abandoned with a stability_timeout error event. Raise that cap for instruments whose writes take longer; it must be at least the stability period.

A file that the instrument keeps open after its last write can look finished while the watcher still can’t read it. The watcher shows one Upload Failed event on its page for each such file, saying it can’t read the file yet, and shows another for it each time the watcher starts or its config changes. It tries the file again every minute for as long as it takes, and uploads it once the instrument lets go. A file someone requests with Upload on the run page stays requested in the meantime. Watcher 1.4.2 and later do this. Older watchers try such a file again only at the next start, and cancel a request for it after three tries.

Choose auto or manual upload

Use manual mode when someone must approve which files leave the instrument PC. Admins manage the queue from the dashboard; see Manage the watcher fleet.

Existing files on first setup

initial_scan controls the backlog already in the watch directory when an environment is first entered:

• production default: upload the backlog (full)
• staging / preview default: record a baseline and skip existing files (new-only)

Override in config when you need the other behavior. Details: Initial scan.