Durable Execution

Status: [Designed] -- Area 2 of the Grove language. Syntax and semantics are defined; implementation is in progress.

Grove workflows are durably executed: their progress is persisted so that execution survives process restarts, deployments, and infrastructure failures. This page covers the primitives that make durable execution practical for long-running workflows -- sleep, signals, suspend/resume, and timers.

Prerequisites: Workflows Overview, Activities. What you'll learn: How to pause workflows with sleep, communicate between workflows with signals, suspend and resume workflow execution, and use timers for deadline-based logic.

What Durable Execution Means

In a durably executed workflow:

Every completed activity result is persisted. If the process crashes, the workflow resumes from where it left off without re-executing completed activities.
Timers and sleep durations survive restarts. A workflow sleeping for 24 hours does not need the process to stay running for 24 hours.
In-flight state is checkpointed. The runtime reconstructs the workflow's position by replaying its event history.

This means workflows can span minutes, hours, days, or even months -- the runtime handles the bookkeeping.

Sleep

The sleep operation pauses a workflow for a specified duration. The workflow is suspended from the runtime's perspective -- it consumes no resources while sleeping:

activity wait_for_cooldown v1 {
  data pause {
    sleep 24h
  }

  apply(input: WaitInput) -> WaitResult {
    return { resumed: true }
  }
}

Duration Syntax

Sleep uses Grove's duration literal syntax:

Literal	Duration
`30s`	30 seconds
`5m`	5 minutes
`2h`	2 hours
`7d`	7 days

Use Cases

Rate limiting -- Wait between API calls to respect external rate limits.
Delayed follow-up -- Send a reminder email 3 days after signup.
Cooldown periods -- Wait before retrying a failed operation with a custom backoff.
Scheduled steps -- Pause until a specific relative time in a multi-day workflow.

workflow onboarding v1 {
  node welcome = send_welcome_email v1
  node wait_3d = wait_period v1 { duration: 3d }
  node followup = send_followup_email v1

  edge welcome -> wait_3d
  edge wait_3d -> followup
}

Signals

Signals are named messages sent to a running workflow execution. They enable inter-workflow communication and external-to-workflow communication. A workflow can wait for a signal, and another workflow, activity, or external system can send one.

Waiting for a Signal

An activity can suspend and wait for a named signal:

activity wait_for_approval v1 {
  data approval {
    signal "approval_decision"
  }

  apply(input: ApprovalInput) -> ApprovalResult {
    return {
      approved: approval.decision == "approved"
      approver: approval.approver_id
      comments: approval.comments
    }
  }
}

The activity blocks until a signal with the name "approval_decision" is delivered to the workflow execution. The signal's payload becomes the data block result.

Sending a Signal

Signals can be sent from another activity using the signal data operation:

activity approve_request v1 {
  data send {
    signal target_workflow {
      execution_id: input.target_execution_id
      signal_name: "approval_decision"
      payload: {
        decision: input.decision
        approver_id: input.approver_id
        comments: input.comments
      }
    }
  }

  apply(input: ApproveInput) -> ApproveResult {
    return { sent: true }
  }
}

Signals can also be sent from:

Routes (an HTTP endpoint triggers a signal)
External systems via the runtime API
Other workflows through activity data blocks

Signal Patterns

Human-in-the-loop approval:

workflow expense_approval v1 {
  node submit = record_expense v1
  node wait = wait_for_approval v1
  node process = branch {
    wait.approved -> process_approved_expense v1
    _ -> notify_rejection v1
  }

  edge submit -> wait
  edge wait -> process
}

External payment confirmation:

workflow payment_flow v1 {
  node initiate = initiate_payment v1
  node wait = wait_for_payment_confirmation v1 {
    timeout: 30m
  }
  node fulfill = fulfill_order v1

  edge initiate -> wait
  edge wait -> fulfill
}

Suspend and Resume

Workflows can be explicitly suspended, pausing all execution until they are resumed. This differs from sleep (which has a fixed duration) and signals (which wait for a specific message).

Suspend is useful when:

A workflow needs manual intervention.
An external prerequisite must be met before continuing.
An administrator needs to inspect the workflow state before allowing it to proceed.

The runtime exposes suspend and resume operations through its management API. From within a workflow, suspension typically happens via a signal wait with no timeout -- the workflow waits indefinitely until an external actor resumes it.

Timers and Deadlines

Timers combine sleep with branching to implement deadline-based logic. If a signal is not received within a time window, the workflow takes an alternative path:

activity wait_for_payment v1 {
  data payment {
    signal "payment_received"
    timeout: 48h
  }

  apply(input: PaymentWaitInput) -> PaymentWaitResult {
    if payment == null {
      return { timed_out: true, paid: false }
    }
    return {
      timed_out: false
      paid: true
      amount: payment.amount
    }
  }
}

When a timeout is specified on a signal wait, the data block resolves to null if the signal is not received within the window. The apply block can then branch based on whether the signal arrived.

Deadline Pattern

workflow order_with_deadline v1 {
  node place = place_order v1
  node wait_payment = wait_for_payment v1 { timeout: 48h }
  node outcome = branch {
    wait_payment.paid -> fulfill_order v1
    _ -> cancel_order v1 { reason: "payment_timeout" }
  }

  edge place -> wait_payment
  edge wait_payment -> outcome
}

Heartbeats

For long-running activities (e.g., processing a large file), heartbeats allow the activity to report progress to the runtime. If the runtime does not receive a heartbeat within the expected interval, it considers the activity failed and may retry it:

activity process_large_dataset v1 {
  data result {
    host process_records {
      dataset_id: input.dataset_id
      heartbeat_interval: 30s
    }
  }

  apply(input: DatasetInput) -> ProcessResult {
    return { records_processed: result.count }
  }
}

Heartbeats are configured on the data operation and managed by the host function implementation. They are a runtime concern rather than a language-level construct.

Execution Guarantees

Property	Guarantee
Activity completion	At-most-once. Completed activities are not re-executed on replay.
Sleep accuracy	Best-effort. The workflow resumes at or after the specified duration.
Signal delivery	At-least-once. Signals may be delivered more than once in edge cases.
Workflow completion	Exactly-once under normal operation. The runtime deduplicates.
State persistence	Every state transition is durably recorded before proceeding.