Fetch Module

Path: src/fetch/

Handles HTTP communication with the Spark REST API via the Databricks driver proxy.

Files

client.rs — SparkHttpClient

SparkHttpClient

#![allow(unused)]
fn main() {
pub struct SparkHttpClient {
    client: reqwest::Client,
    base_url: String,
    token: String,
}
}

FetchError

Error enum with user-friendly messages for common HTTP errors:

types.rs — API Types

Application & Job Types

Stage Types

SQL Types

Task Types

Executor Types

spark.rs — Endpoint Methods

Methods on SparkHttpClient:

databricks.rs — DatabricksClient

Thin client for Databricks REST API /api/2.0/* endpoints and workspace-level requests.

DatabricksClient

#![allow(unused)]
fn main() {
pub struct DatabricksClient {
    client: reqwest::Client,
    base_url: String,        // e.g. https://{host}/api/2.0
    workspace_root: String,  // e.g. https://{host}
    token: String,
    sparkui_cookie: Option<String>,
}
}

SparkuiProbeResult

Tri-state result from probing a Spark UI endpoint:

Key Methods

Warm-up Detection

The is_loading_page() helper detects HTML loading pages returned during Spark UI warm-up by checking for patterns like <title>Loading</title>, loading spark ui, please wait, or generic HTML responses.

eventlog/ — Event Log Parsing

The load_event_log() function orchestrates: discover log path → download from DBFS → decompress gzip → parse JSON lines → return structured data.

orchestrator.rs — Data Polling & Assembly

poll_once

Fetches all data and runs analysis in a single poll cycle:

1. Fetch jobs, stages, SQL executions, and executors concurrently (4-way tokio::join!)
2. Aggregate active executors into ClusterResources (total memory, cores, executor count)
3. Build cross-reference maps (job↔SQL, stage↔job)
4. Build ranked jobs (sorted by duration, running first)
5. Create a SuspectContext from the cross-reference maps
6. Run 10 stage-level detectors via function pointer table (detect_slow_stages, detect_spill, detect_cpu_efficiency, detect_record_explosion, detect_task_failures, detect_memory_pressure, detect_partition_count, detect_broadcast_join, detect_python_udf, detect_cache_opportunity)
7. Fetch task lists for top ~15 stages (selected by multiple heuristics)
8. Run skew detection on fetched tasks (duration, data-size, record-count, executor hotspot)
9. Aggregate and sort suspects (severity first, then estimated_savings_ms descending)
10. Build stage_sql_hints — maps stage_id to top SQL plan operations
11. Compute critical_stages — the longest wall-clock stage per job (critical path)
12. Compute HealthSummary (job/IO counts, critical/warning counts, top issues)
13. Return DataPayload

compute_health_summary

#![allow(unused)]
fn main() {
fn compute_health_summary(
    jobs: &[RankedJob],
    stages: &[SparkStage],
    suspects: &[Suspect],
) -> HealthSummary
}

Aggregates job counts, total I/O bytes, and suspect severity counts into a HealthSummary for the summary bar widget.

poller.rs — Background Poller

run_poller

#![allow(unused)]
fn main() {
pub async fn run_poller(
    client: Arc<SparkHttpClient>,
    databricks: Arc<DatabricksClient>,
    config: Arc<Config>,
    tx: mpsc::UnboundedSender<Action>,
    poll_interval: Duration,
)
}

1. Discovers the application ID (or detects cluster unreachable → historical fallback)
2. Enters a live poll loop: poll_once (from orchestrator.rs) → send result via channel → sleep
3. On connection loss during polling, falls back to historical data via try_load_historical

Historical Fallback Chain

When the cluster is unreachable, try_load_historical attempts these strategies in order:

The Spark UI strategy includes warm-up retry: if the endpoint returns an HTML loading page, it retries with backoff delays defined in SPARKUI_RETRY_DELAYS (3, 5, 10, 15, 20 seconds — ~53s total).

DataPayload

#![allow(unused)]
fn main() {
pub struct DataPayload {
    pub app_id: String,
    pub jobs: Vec<RankedJob>,
    pub stages: Vec<SparkStage>,
    pub sql_executions: Vec<SparkSqlExecution>,
    pub suspects: Vec<Suspect>,
    pub stage_tasks: Arc<HashMap<i64, Vec<SparkTask>>>,
    pub summary: HealthSummary,
    pub cluster_resources: ClusterResources,
    pub stage_sql_hints: Arc<HashMap<i64, String>>,
    pub critical_stages: Arc<HashSet<i64>>,
    pub last_updated: String,
    pub data_source: DataSourceMode,
    pub data_source_detail: Option<String>,
}
}

Note: DataPayload is defined in src/tui/mod.rs and contains all data needed to render the TUI.

Key fields:

• stage_sql_hints — maps stage_id → String with top SQL plan operations (e.g., “HashAggregate → Exchange → Scan parquet”), pre-computed for display in stage detail headers
• critical_stages — set of stage IDs that represent the critical path (longest wall-clock stage per job), used for “CP” annotations in the job detail view
• data_source — DataSourceMode::Live or DataSourceMode::Historical, displayed in the status line
• data_source_detail — optional label like "Spark UI", "history server", or "event logs", shown alongside the HISTORICAL badge