When a feature pipeline first goes into production, the batch recompute job feels cheap. You have a few months of event data, the Spark job reads it in a few minutes, and the resulting feature table gets written to the feature store without anyone noticing the cost. It's a line item, not a problem.
A year later, the same job takes 45 minutes. Two years in, it is running on a larger cluster for two hours, and the engineering team has started receiving questions about the data warehouse bill. Nothing changed about the job. What changed is the data volume it has to process to produce the same output.
This is the silent compounding problem with full-recompute feature pipelines. The output grows slowly (current feature values for each entity), but the input grows continuously (the complete history of events). The cost is a function of the input, not the output, and those two growth curves diverge over time.
Why Full Recompute Is Linear in History Size
To understand why the cost grows this way, consider what a batch feature pipeline actually does. You have a table of events. You want a table of current aggregate feature values per entity. The standard approach is a GROUP BY query: group by entity_id, compute aggregate functions over all rows.
The correct result for "SUM of all transactions per customer" requires reading every transaction for every customer. The result set is one row per customer, but deriving it requires touching every input row. If you have one million customers each with an average of 50 transactions, that is 50 million rows read to produce one million output rows. Six months later with 70 transactions per customer, the query reads 70 million rows to produce the same one million rows.
The output size is constant (or grows slowly as new customers are added). The compute cost grows with the average number of events per entity, which grows with calendar time. This is not a fixable problem in the batch recompute model; it is inherent to what full recompute means.
The Three Hidden Costs
The obvious cost is the cluster compute time that shows up on your bill. But there are two other costs that are harder to quantify and often overlooked until they become problems.
Latency floor inflation. As the recompute job takes longer to run, the batch interval often needs to expand to accommodate it. A job that was running every 15 minutes when it completed in 5 minutes now takes 12 minutes and is running close to the scheduling limit. Teams respond by increasing the interval to 30 minutes or an hour, which increases feature staleness. The data volume growth that caused the compute increase is the same root cause for the freshness degradation.
Failure blast radius. A batch job that runs for 2 hours has a larger window during which it can fail, and a larger blast radius when it does. If a nightly 2-hour feature recompute fails at minute 90, you have a choice between retrying from scratch (another 2 hours) or serving stale features. For a 5-minute job, a failure means 10 minutes of stale data during a retry. For a 2-hour job, it can mean an extended outage of fresh features.
Schema change cost. When a feature needs to be modified (a new column added, a window parameter changed), a full recompute is required to backfill the new value across the entire history. For a large dataset, this is a significant engineering coordination cost: you need to schedule the backfill, run the job, validate the output, swap the feature store pointer, and coordinate with any models consuming the old feature. The larger the history, the larger this cost.
Incremental Maintenance Changes the Cost Function
Incremental computation flips the cost structure. Instead of processing the full history on each run, the engine maintains the current state and applies only the delta from new events. When a new transaction arrives for a customer, only the aggregates for that customer are updated. The engine does not need to re-read the customer's full transaction history to do so; it already has the running state.
The computation cost per update is proportional to the size of the delta, not the size of the history. Whether a customer has 50 transactions or 5,000, the cost of incorporating their 5,001st transaction is the same: process one event, update the affected aggregates.
-- Feldera maintains this view incrementally: each new transaction
-- updates only the affected customer row. No full scan required.
CREATE MATERIALIZED VIEW customer_lifetime_features AS
SELECT
customer_id,
COUNT(*) AS total_transactions,
SUM(amount) AS lifetime_spend,
AVG(amount) AS avg_transaction_amount,
COUNT(DISTINCT merchant_category) AS category_diversity
FROM transactions
GROUP BY customer_id;For the running cost, the implication is that incremental computation costs are proportional to the event rate, not the total data volume. A system processing 10,000 transactions per second pays the same per-second compute cost whether the total transaction history contains 10 million or 10 billion records. The batch recompute approach pays a cost that grows with the history; the incremental approach pays a cost that grows with the throughput.
State Management: The Other Side of the Trade
Incremental maintenance does not eliminate history cost; it shifts it. Instead of re-reading history at compute time, the engine maintains state in memory (or on disk with memory-mapped access) that encodes the current aggregate for each entity. This state grows as the number of distinct entities grows.
For lifetime aggregates (total spend, lifetime transaction count), the state per entity is small: a few scalar values per key. For a customer table with 10 million customers, the state is measured in gigabytes, which is manageable on modern hardware. For time-windowed aggregates with large windows (30-day rolling counts), the state includes the individual events within the window, which can be significantly larger.
The practical tradeoff: incremental computation reduces compute costs and reduces latency, but it requires persistent state storage that grows with cardinality. Batch recompute has no persistent state (it reads from source each time) but pays compute costs that grow with history depth.
For most production feature pipelines, the cardinality of the entity set is much smaller than the total event count, which makes incremental maintenance the better trade. A platform with 10 million users and 10 billion transactions has a 1,000x ratio of events to entities. Incremental state stores 10 million rows; batch recompute reads 10 billion rows. The difference widens every day.
Where Full Recompute Still Has a Role
This is not an argument that full recompute should be abandoned entirely. There are specific cases where it remains the right approach.
Feature backfills for new model candidates require computing feature values as of specific historical timestamps, which incremental systems generally do not support (they maintain the current state, not the historical log of state values). Historical feature replay for model training needs the full history anyway.
Complex features that involve non-monotone aggregates (arbitrary rank calculations, nth percentile computations over an unbounded window, complex sequential pattern detection) may not be straightforwardly expressible as incrementally maintainable views. For these, a periodic batch recompute against a fixed historical window is still the practical approach.
The reasonable architecture for most growing ML platforms is to run both: incremental pipelines for the current-state features that feed real-time inference, and periodic batch jobs for the historical-window features and training data preparation. The incremental layer handles the freshness-sensitive features efficiently as data volume grows; the batch layer handles the features that require historical access and accepts the growing compute cost as part of the research and training workflow.