When a NetBox is busy

Designs keeps its own copy of what it manages in each NetBox, and plans against that copy. One operation at a time works on a NetBox: an apply, the reading-back that follows it, or a refresh. Anything else that reaches for the same NetBox meanwhile is told it is busy, with what it is doing and how far it has got, rather than waiting in silence:

The target is still syncing after an apply, 120 of 271

Try again when it is free. GET /api/v1/services/{name}/activity answers at once with the same, and the NetBox's page carries it while it lasts.

In the GUI, a plan or apply that finds the NetBox busy does not fail: the deploy page says what the NetBox is doing and how far it has got, asks again every couple of seconds, and offers Plan now or Apply now when it is free. The plan is the same one, so an apply that was waiting goes ahead with what was already accepted.

What happens after an apply

An apply returns as soon as NetBox has the objects. Reading back what the apply changed, including the objects NetBox created on its own such as a device's interfaces from its type, runs behind the response, so the next plan sees them. For a large design that reading-back takes a while, and the NetBox is busy for as long as it does: as a guide, about thirty changes a second, so a thousand devices whose type gives each eight interfaces, nine thousand changes, keep their NetBox busy for about five minutes. Plan the next batch when it is free, or leave the deploy page waiting; it plans by itself.

When the copy is behind

If that reading-back fails, say NetBox stopped answering midway, Designs does not pretend it finished. The NetBox is marked as possibly behind, with the reason, and a plan made against it says so in its notes:

Designs' copy of 'netbox-prod' may be behind NetBox: reading back what the apply changed failed (...). Refresh everything from NetBox on the target, then plan again.

Refresh from NetBox normally replays only what NetBox's changelog says changed since the last read, which is quick. When the copy is marked behind, read everything again instead:

nbd service refresh netbox-prod --full

or POST /api/v1/services/{name}/refresh?full=true. In the GUI, a NetBox marked behind says may be behind NetBox in the Deploy Targets list, and its details and the deploy page's plan step carry the reason with a Refresh everything from NetBox button beside it. The mark is cleared when the refresh completes.

When NetBox did not answer for part of a refresh

A refresh reads each kind of object in turn. If what stands in front of your NetBox answers for it, a 502, 503 or 504 from Cloudflare or a proxy meaning NetBox did not answer it properly, Designs asks again, twice, a few seconds apart. If an answer still does not come, or NetBox answers one of the reads with an error, the refresh does not finish, and Designs' copy is not changed: it is left as it was, whole, and the NetBox's page says so:

NetBox at https://… could not be read in full: 1 of 120 models failed (Notification; NetBox answered 502: …). Designs' copy was left as it was. Refresh again.

Refresh again once NetBox is answering. A refresh that completes has read every kind of object; one that could not is never saved in part.

When a part of Designs is not answering

A busy or unreachable NetBox is one thing; a part of Designs itself restarting is another, and it is said differently. When the design service, the catalogue, the service that reads your NetBox, the data model service or the database does not answer, the API answers 503 with the code service_unavailable and a sentence such as:

Designs can't reach its design service right now. This usually clears in a minute. Try again.

Nothing is wrong with your NetBox, and nothing about it needs fixing. The technical detail (which host, which error) is in the response's detail, and in the GUI behind a Details disclosure under the sentence, with a Retry button beside it. Where a lookup failed, the page says so where its result would have been: under the Deploy page's Design field (and Plan waits), where a NetBox's objects would be listed, and above the Deployments list. A Deployments list that could not be refreshed stays on show, marked with when it was loaded, so you can tell cached from current.