A sync system can return HTTP 200 for months while quietly discarding user work. The dangerous failures aren’t dramatic outages. They’re edits overwritten by an unreliable clock, operations marked complete before acknowledgment, and retries that apply the same business action twice.
Offline-first mobile sync: the local database accepts durable work and serves reads while a synchronization protocol reconciles that work with a remote authority. Offline capability isn’t a cache switch. It creates a distributed system whose replicas can diverge, restart, and observe events in different orders.
This architecture uses three safeguards that basic guides usually skip: a mutation durability contract, a conflict policy by business invariant, and a reconciliation ledger that proves what happened to every operation. It also treats authentication expiry, deletion, schema migration, and partial batches as normal states.
Define the Durability Contract First
A durability contract states exactly when the interface may tell a user that work is saved. For an offline-first app, success normally means the mutation and its synchronization intent committed atomically to local storage.
Don’t show “saved” after changing only in-memory state. Don’t mark an outbox item sent before server acknowledgment. Put the domain update and outbox operation in one database transaction. If the process dies at any instruction boundary, either both remain or neither does.
BEGIN IMMEDIATE;
UPDATE inspections
SET status = 'submitted', local_version = local_version + 1
WHERE id = :inspection_id;
INSERT INTO sync_outbox(
operation_id, entity_id, operation_type, payload, state, attempts
) VALUES (
:operation_id, :inspection_id, 'submit_inspection', :payload, 'pending', 0
);
COMMIT;
The operation ID must remain stable across retries. Generate it before the transaction, not inside a network request. The server stores accepted IDs or derives an equivalent idempotency rule so a lost response can’t turn a retry into a duplicate submission.
Separate “saved on this device” from “available everywhere.” That language isn’t cosmetic. It gives users an accurate recovery model and gives support a state they can diagnose.
Protect the outbox like primary user data
An outbox isn’t disposable transport metadata. It may contain the only durable copy of a user’s inspection, note, or order. Encrypt sensitive fields at rest, include the table in schema migrations, and exclude it from cache eviction. If device backup policy permits database backup, decide whether pending operations may restore onto another installation and how their device identity and authorization will be revalidated.
Storage exhaustion changes the contract. A transaction that can’t append to the outbox must fail visibly before the interface promises a save. Reserve practical headroom, monitor database growth, and compact acknowledged history without touching pending or blocked work. Large attachments need their own durable manifest and upload state rather than a huge opaque outbox row.
Logout also needs a written policy. Deleting a local account with unsynchronized work may destroy data. The app can require synchronization, offer an encrypted export, or warn that named pending items will be removed. On shared devices, retaining another user’s outbox after logout creates a privacy failure. Product and security owners must choose deliberately.
Sync Operations Instead of Whole Records
Operation-based synchronization preserves intent better than repeatedly uploading an opaque record snapshot. “Add tag safety” and “set due date” can commute; two complete snapshots may overwrite each other.
An outbox row needs an immutable operation ID, entity ID, operation kind, canonical payload, base version or causal context, creation sequence, state, and attempt metadata. Avoid device wall-clock time as the authority. Users change clocks, devices drift, and daylight-saving changes don’t represent causality.
import java.util.UUID
data class PendingOperation(
val operationId: UUID,
val entityId: UUID,
val kind: String,
val baseRevision: Long,
val payloadJson: String,
val sequence: Long,
val state: State = State.PENDING
) {
enum class State { PENDING, IN_FLIGHT, ACKNOWLEDGED, BLOCKED, REJECTED }
}
Not every change should become a generic patch. Domain verbs let the server preserve invariants. “Reserve stock” differs from “set quantity.” “Approve expense” differs from “set approved=true.” Commands can be revalidated after reconnect, authorized against current policy, and rejected without corrupting local truth.
Whole-record upload remains reasonable for immutable drafts owned by one user. Use it deliberately, with a base revision and an explicit conflict response.
Make the Server Idempotent
Mobile delivery is at least once in practice because a client can’t distinguish a lost request from a lost response. Idempotency makes repeating the same operation safe.
The server should claim an operation ID in the same transaction that applies its effect. A preliminary lookup followed by an unrelated write has a race: two concurrent retries can both observe absence and apply the operation.
-- One statement, so the claim and the effect cannot drift apart. The previous shape
-- needed the caller to pass whether the INSERT had claimed the operation, which the
-- caller cannot know: a concurrent retry could claim it between the two statements.
WITH claimed AS (
INSERT INTO accepted_operations(operation_id, user_id, accepted_at)
VALUES (:operation_id, :user_id, now())
ON CONFLICT (operation_id) DO NOTHING
RETURNING operation_id
)
UPDATE inventory
SET reserved = reserved + :quantity,
revision = revision + 1
WHERE sku = :sku
AND reserved + :quantity <= available
AND EXISTS (SELECT 1 FROM claimed);
Return the same logical result for a replay, including the resulting entity revision where possible. Keep idempotency records at least as long as a mobile client may retry after a long offline period. A seven-day retention policy is unsafe if field devices can stay disconnected for a month.
Idempotency doesn’t make two distinct operations equivalent. Tapping “pay” twice must create one stable operation before either request. Two freshly generated IDs describe two requests and may legitimately apply twice.
Choose Conflict Rules by Invariant
Conflict resolution should follow the business meaning of each field, not one global algorithm. Last-write-wins is safe only when losing an earlier value is acceptable and ordering is trustworthy.
| Data | Rule | Why |
|---|---|---|
| Display preference | Server-sequenced last value | Loss has low cost |
| Set of tags | Add/remove operations | Independent changes can merge |
| Long text | Three-way merge or CRDT | Concurrent edits carry value |
| Payment approval | Server validation | Current authority is mandatory |
| Delete versus edit | Tombstone plus recoverable conflict | Silent loss is unacceptable |
CRDTs help when data types and collaboration justify their metadata and conceptual cost. They don’t solve authorization, validation, deletion retention, or semantic conflicts such as two workers claiming the same scarce asset.
Create a conflict register during design. For each mutable type, write concurrent scenarios, invariant, automatic rule, user-visible outcome, and audit requirement. If the team can’t explain a merge in plain language, it isn’t ready to automate it.
Pull Changes With a Durable Cursor
Incremental pull needs a server-issued cursor that represents an ordered change feed. A client timestamp is not a safe cursor because equal timestamps, clock differences, and pagination races can skip records.
The response should contain changes plus the cursor after those changes. Apply both in one local transaction. If the app dies before commit, it repeats the page. If it dies after commit, it resumes at the new cursor.
{
"changes": [
{"sequence": 8841, "entityId": "a1", "type": "updated", "revision": 19},
{"sequence": 8842, "entityId": "b7", "type": "deleted", "revision": 4}
],
"nextCursor": "tenant-42:8842",
"hasMore": true
}
Keep tombstones long enough for the oldest supported offline client. Hard-deleting a record before a disconnected replica learns about it allows the record to reappear on its next upload. When retention can’t cover indefinite offline periods, require a full resnapshot after cursor expiry.
A cursor belongs to a scope: account, tenant, dataset, schema version, and authorization context. Changing scope should invalidate it rather than returning an incomplete view.
Design resnapshot as a normal protocol branch
Change logs can’t grow forever, and clients can’t stay offline forever without consequence. The server should return an explicit cursor-expired response with a snapshot generation, not a generic bad request. The client then preserves its outbox, replaces or reconciles replicated server state transactionally, stores the new cursor, and resumes pending uploads.
A naïve “clear and download” can erase local-only edits or break foreign keys. Stage the snapshot in temporary tables, validate counts and schema, then swap it into place while retaining unsent operations. Reapply optimistic projections from the outbox after the swap. If an operation targets an entity absent from the snapshot, route it through the documented delete or authorization policy.
Large datasets need partitioned snapshots and resumable transfer. Bind each partition to one snapshot generation so pages don’t mix two server moments. Verify checksums before activation. A low-storage device may not hold old and new copies simultaneously, which means the product needs a storage prerequisite, a streaming migration design, or a narrower offline dataset.
Handle Partial Failure as Data
A synchronization batch isn’t one success bit. Each operation can be accepted, replayed, conflicted, unauthorized, invalid, temporarily failed, or permanently rejected.
Return an outcome for every operation ID. Apply acknowledgments transactionally, move permanent failures to a visible blocked state, and retry only transient outcomes. Don’t let one invalid item hold the entire queue forever.
{
"results": [
{"operationId": "op-1", "status": "accepted", "revision": 8},
{"operationId": "op-2", "status": "conflict", "serverRevision": 11},
{"operationId": "op-3", "status": "rejected", "code": "POLICY_CHANGED"}
]
}
Authentication expiry is not a retryable network failure. Pause protected operations, refresh or reauthenticate, then revalidate them against current authorization. A queued approval created before the user’s role was revoked mustn’t execute merely because connectivity returned.
Use bounded exponential backoff with jitter. Respect platform scheduling and server retry hints. Infinite foreground loops burn battery and turn an outage into load amplification.
Respect Mobile Background Limits
Background schedulers offer opportunities, not delivery deadlines. Android WorkManager and Apple’s BGTaskScheduler can defer, coalesce, or stop work according to power, connectivity, and system policy.
Design sync to resume from durable state after expiration or process death. Keep units small. Save progress after each acknowledged page or batch. The foreground app should trigger the same engine rather than maintain a second synchronization implementation.
Android’s official offline-first data guidance describes local-source reads and queued work, while WorkManager handles deferrable background work. On iOS, use BackgroundTasks and register expiration handling.
Connectivity callbacks are hints. A network can exist without usable internet, authentication, or server reachability. Let actual requests determine reachability and avoid launching a sync storm on every brief transition.
Tell Users the Truth About State
Offline UX should distinguish local durability, synchronization progress, blocked work, and stale remote data. A single cloud icon can’t explain all four.
Show “saved on this device” immediately after the local transaction. Show pending count when it helps action. Surface rejected operations next to the affected entity with a concrete resolution. Display last successful refresh for information whose freshness affects decisions.
Don’t expose every retry. Transient network noise doesn’t require user labor. Escalate when the engine exhausts policy, needs reauthentication, detects a conflict requiring judgment, or lacks storage.
Provide export or copy options for valuable blocked content. If a server validation change rejects a long offline note, preserving the text is more important than producing a neat red banner.
Build a Reconciliation Ledger
A reconciliation ledger records each operation’s journey without storing sensitive payloads. It turns “my edit disappeared” from guesswork into an answer.
Track operation ID, entity type, local sequence, creation age, attempt count, latest outcome category, server revision, and terminal state. Correlate server logs with the same operation ID. Redact field values and secrets.
Monitor queue-depth distribution, age of oldest pending item, accepted and rejected rates, conflicts by entity type, cursor lag, resnapshot frequency, and recovery after process termination. Averages hide stranded devices, so use percentiles and maximum age.
Define a sync service-level objective
Availability alone is the wrong service measure because the API can be reachable while clients remain stale. Define objectives around outcomes: a percentage of eligible operations acknowledged within a chosen interval after connectivity returns, a maximum age for the oldest healthy pending item, and a bounded rate of unexplained permanent rejection. Segment by app version and platform.
Account for legitimate long-offline devices. An operation’s wall-clock age and its connected-age aren’t the same. Record connectivity opportunities without assuming every callback means usable service. A device that had no viable route for five days differs from one that contacted the server repeatedly yet made no cursor progress.
Alerts should point to a broken stage. Rising queue depth with normal request success may indicate acknowledgments aren’t committing locally. Stable uploads with growing cursor lag suggests pull failure. Conflicts concentrated after one backend release suggest schema or policy drift. This stage model shortens diagnosis because responders don’t start from a generic “sync is slow” symptom.
Run a periodic invariant check that compares selected server and client summaries. Checksums by partition, entity counts, and revision ranges can reveal divergence without uploading private content. Repair should be explicit and audited, not “clear the database and hope.”
Test the Failure Timeline
Sync testing should kill processes between durable steps, skew clocks, expire tokens, reorder responses, duplicate requests, fill storage, and keep devices offline past tombstone retention.
- Commit a local mutation, kill before network send, and verify recovery.
- Apply the request server-side, drop the response, and verify idempotent retry.
- Interrupt halfway through a pull page and verify cursor atomicity.
- Edit the same fields on two devices and assert the documented merge.
- Delete online while editing offline and preserve recoverable user work.
- Expire authorization before reconnect and block unauthorized replay.
- Upgrade schema with a non-empty outbox and retain every operation.
Property-based tests are useful for operation algebra. Generate sequences, reorder independent operations, duplicate delivery, and assert invariants. For CRDTs, test convergence after replicas receive the same operations in different orders.
Don’t benchmark only Wi-Fi. Test high latency, packet loss, metered links, low battery, background restriction, and storage pressure on physical devices.
Test schema evolution with work in flight
A mobile release can arrive with pending operations written by an older schema. The migration must preserve their meaning, not merely keep the rows readable. Version every operation payload. Decode old versions through explicit adapters, and retain adapters until no supported client or restored backup can produce them.
Server evolution needs the same care. Removing a field or changing validation can strand operations created offline before deployment. Accept an older command version for a documented window, transform it, or reject it with enough structured context for the client to preserve and repair the user’s work. A generic 400 response isn’t a migration strategy.
Run upgrade tests from every supported database version with pending, blocked, and in-flight rows. Simulate a crash during migration and confirm transactional recovery. Then downgrade where the platform allows it. If downgrade isn’t safe, stop the older binary from opening the migrated database rather than letting it reinterpret new state.
Frequently Asked Questions
What is offline-first mobile sync?
Offline-first mobile sync makes a durable local database the immediate source for reads and writes, then reconciles queued operations with a remote system. The interface remains useful without connectivity. Correctness depends on transactional local persistence, idempotent server handling, ordered pull cursors, domain-specific conflict rules, and honest user-visible synchronization state.
How do I prevent data loss during synchronization?
Commit each domain mutation and its outbox item in one local transaction. Keep a stable operation ID across retries, claim that ID transactionally on the server, and acknowledge locally only after a verified response. Apply pulled changes and their cursor atomically. Preserve rejected content and test process death at every boundary.
Why is last-write-wins dangerous?
Last-write-wins silently discards one concurrent value and often relies on unreliable device clocks. It can be acceptable for low-value preferences, but not collaborative text, inventory, approvals, or user work. Select a policy per invariant: operation merge, three-way merge, CRDT, server validation, or visible conflict resolution.
When should I use CRDTs?
Use CRDTs when replicas must accept concurrent edits and the data type has suitable merge semantics, such as collaborative text or sets. Avoid them when the real conflict is a business invariant, scarce resource, authorization decision, or simple single-writer record. CRDT metadata and debugging complexity must earn their place.
What is the difference between an outbox and a cache?
A cache stores data that can usually be fetched again. An outbox stores user intent that may exist nowhere else and therefore requires durable, transactional treatment. Evicting cache entries may reduce performance; losing an outbox item loses work. Keep retention, encryption, migration, backup, and observability policies separate.
How should deletes synchronize?
Represent deletion with a tombstone or equivalent ordered operation long enough for disconnected replicas to observe it. Define what happens when an offline edit meets that tombstone. For valuable content, preserve the edit for restoration or user resolution. Expired cursors should force a resnapshot instead of letting deleted records reappear.
Why can’t background sync be guaranteed?
Mobile operating systems schedule background work according to battery, usage, network, and resource policy. They may delay or stop it. Build a resumable engine backed by durable state, save progress in small units, handle expiration, and trigger the same engine in foreground. Never make correctness depend on a promised execution time.
When should an app not be offline-first?
A full offline mutation model is a poor fit when actions require current global truth, immediate authorization, or scarce-resource coordination. Payments, permission changes, and competitive reservations may need an online confirmation. The app can still cache read-only data and drafts while clearly marking actions that require a connection.
How do I monitor sync health?
Measure pending queue depth, oldest operation age, terminal rejection rate, conflicts by data type, cursor lag, resnapshots, and successful recovery after interruption. Correlate client and server events through operation IDs without logging payloads. Use percentiles because a healthy average can hide a small group of permanently stranded devices.
Is offline-first mobile sync worth the complexity?
It is worth it when users operate through unreliable connectivity or expect instant durable editing. It isn’t free reliability. The architecture adds distributed-state, storage, migration, conflict, and support costs. Narrow the offline scope to valuable journeys, define invariants first, and prefer online validation for actions whose correctness depends on current server state.
Make Every Mutation Accountable
Offline-first mobile sync is safe when every accepted user action enters durable storage, every retry has stable identity, every conflict follows a business rule, and every terminal outcome remains explainable.
- Commit the domain change and outbox entry together.
- Use server-side idempotency and ordered pull cursors.
- Choose conflict semantics per invariant, not per framework.
- Preserve rejected work and expose actionable state.
- Test interruption timelines, not only happy-path APIs.
Start with one vertical slice. Write its durability contract, conflict register, cursor protocol, and failure tests before expanding. The goal isn’t to synchronize faster. It’s to make silent loss structurally difficult and operationally visible.
