Most ML teams that have moved beyond batch processing now use Kafka. User clicks, transactions, sensor readings, and API calls move through Kafka topics in near real time. The data is current and available. The challenge is what comes next. Between Kafka and the feature store serving the model during inference is a computation layer, usually a scheduled batch job.
The irony is deploying a low-latency message bus, then adding a one-to-four hour staleness window by batching feature computation downstream. Kafka infrastructure is often not the bottleneck. Scheduled SQL or Python aggregation jobs usually are.
Why batch jobs remain despite Kafka
The straightforward answer is that correct stream processing code is harder to write than a SQL query or PySpark job. Kafka Streams and Flink are powerful, but safely running a multi-window aggregation requires knowledge of stateful operator semantics, watermark management, and checkpoint configuration. For an ML engineer who mainly works with DataFrames and SQL, that creates a high barrier for a routine feature engineering task.
Stream processing frameworks also have limited integration with the feature store ecosystem. A Flink job that computes a feature and writes it to a feature store needs custom sink implementations, schema management, and usually glue code that becomes technical debt as the feature library expands. Batch jobs that use a bulk load API to write to the feature store are easier to maintain, despite their freshness tradeoff.
The actual cost of the gap
Consider a fraud detection pipeline. For card-not-present fraud, a key behavioral signal is transaction velocity: how many transactions has a card made in the last 15 minutes, 1 hour, and 24 hours? If the batch pipeline runs every 30 minutes, velocity features can be up to 30 minutes stale during inference. A card crossing the fraud threshold after the last update remains invisible to the model until the next run. Fraudulent activity can exploit this 30-minute gap by keeping structured transactions just below threshold limits.
For recommendation systems, the cost is less severe but still measurable. Session-level behavior, category affinity in the current browsing session, recently viewed items, and current search context are among the strongest signals of near-term purchase intent. These features remain useful for minutes. An hourly batch pipeline may compute features describing a session that ended before those features became available.
Optimizing the batch job cannot solve either issue. The latency floor is architectural.
The architecture that closes the gap
Place an incremental computation engine between Kafka and the feature store. It subscribes to Kafka topics, maintains SQL materialized views over incoming events, and writes changed feature rows to the feature store. Each new event incrementally updates the affected aggregates. The feature store therefore reflects the latest events, limited by Kafka consumer group latency and the incremental compute step.
-- Source: Kafka topic subscribed as a streaming input table
CREATE TABLE transactions (
txn_id VARCHAR,
card_id VARCHAR,
amount DECIMAL(12, 2),
merchant VARCHAR,
ts TIMESTAMP
) WITH (
connector = 'kafka',
topic = 'transactions-v1',
format = 'json'
);
-- Velocity features: maintained incrementally per new event
CREATE MATERIALIZED VIEW card_velocity AS
SELECT
card_id,
COUNT(*) FILTER (WHERE ts >= NOW() - INTERVAL '15 minutes') AS count_15m,
COUNT(*) FILTER (WHERE ts >= NOW() - INTERVAL '1 hour') AS count_1h,
SUM(amount) FILTER (WHERE ts >= NOW() - INTERVAL '24 hours') AS amount_24h
FROM transactions
GROUP BY card_id;
-- Sink: write updated rows to feature store as they change
CREATE SINK feature_store_sink AS
SELECT * FROM card_velocity
INTO redis_sink(host = 'feature-store:6379');Here, Feldera subscribes to the Kafka topic and applies incremental delta processing to the card_velocity view. When a transaction arrives for a card, only that card's velocity aggregates change. The Redis sink receives the updated row within the incremental step's compute latency, typically under 100 milliseconds for simple aggregations. The feature store row then represents that card's latest transaction in near real time.
Handling Kafka's late arrivals
Kafka topics do not always deliver events in timestamp order. Network delays, producer retries, and partition rebalancing can make events arrive seconds or minutes after their event timestamps. In time-windowed aggregations, a late event may fall inside or outside the window that was current when it was produced.
Feldera handles late arrivals by tracking the current watermark for each input partition and applying a configurable late arrival tolerance. Events within the tolerance are processed normally. Events arriving after their window has fully closed are counted as late and may be written to a dead-letter queue. By default, events are processed up to a specified lag, while later arrivals are discarded. Teams where late data is common can tune the tolerance and monitor the late-arrival rate through pipeline metrics.
Waiting longer for late data raises end-to-end latency, while discarding more late data reduces feature accuracy. The appropriate tolerance depends on the source system's delivery behavior. For most Kafka-backed event streams with well-configured producers, a five-second late arrival tolerance catches nearly all out-of-order events without meaningfully raising latency.
Online-offline skew: the silent failure mode
Online-offline skew is a frequently overlooked production feature store problem: training and serving features are computed differently, so the model is evaluated against a distribution it did not see during training. It most often arises at the online-offline boundary, with batch pipelines producing training data and a separate online pipeline serving predictions.
Incremental computation does not remove online-offline skew risk, but it simplifies the architecture behind it. If the same SQL query handles training feature backfill and live serving, computed incrementally in both directions, the two pipelines cannot implement different aggregation semantics. SQL becomes the single source of truth for both paths. Feldera documentation emphasizes this design property because it addresses a production failure mode that is otherwise difficult to audit.
The tradeoff is operational complexity around state management, which batch pipelines avoid. Backfill and initial state population need a separate bootstrap step, and query changes require careful state migration. These issues are solvable, but batch pipelines do not have them. If hourly batch jobs satisfy your freshness requirements, their simplicity has real value. Incremental pipelines are the appropriate investment when the freshness gap measurably harms model prediction quality.