Spots 
Roughly 80% of the "exactly-once" guarantees I see implemented in production financial pipelines are actually "at-least-once" disguised by a fragile post-processing cleanup script. We obsess over the Spark checkpointing mechanism while ignoring the fact that our sink is a leaky bucket.
Why I chose this topic: I’ve spent the last six years cleaning up "exactly-once" messes that caused multi-million dollar reconciliation failures in healthcare billing systems. If you don't understand the physical limitations of your sink, your streaming job is just a very expensive way to generate duplicate records.
Most of my peers argue that Spark’s checkpointLocation is the gold standard for fault tolerance. They treat it like a holy relic. The reality is that Spark’s exactly-once guarantee is strictly confined to the boundary between the source offset management and the state store. Once your data hits an external database or a file system that doesn't participate in a distributed transaction with Spark, your "exactly-once" guarantee is effectively vaporware. Why the common approach falls short
The industry standard is to use foreachBatch to sink data into a relational database or a cloud object store. Developers assume that because Spark retries the batch upon failure, they are protected against duplicates. This is a dangerous fallacy.
If your Spark executor writes 10,000 rows to a Postgres table and crashes at row 9,999, the driver will restart the task. If you aren’t using idempotent writes—where the sink itself knows how to deduplicate based on a business key—you now have a partial write. If your sink logic involves an INSERT statement without an ON CONFLICT DO UPDATE or a similar upsert mechanism, you have just injected garbage into your downstream analytical layer. Take a look at a common, broken pattern I see in Spark 3.3.x:
This code is a lie. If the save() operation fails midway, Spark reruns the batch. If the database processed 5,000 rows before the crash, those 5,000 rows remain. The retry will attempt to insert all 10,000 rows again. You have just doubled your data. Unless your schema has a strict unique constraint and you’ve configured your JDBC driver to handle the resulting exception, you’ve corrupted your downstream data. The illusion of checkpointing
Spark’s checkpointing is brilliant for maintaining the state of your streaming query. It stores the metadata of processed offsets in HDFS or S3. But people confuse "I know which offsets I’ve processed" with "I have committed the data to the sink."
When a worker node dies, Spark rolls back the state to the last successful offset recorded in the checkpoint directory. It then re-executes the batch. This is where the physical reality of your sink takes over. If you are writing to S3 using the default file sink, Spark uses a commit protocol that involves renaming temporary files. If that rename operation isn't atomic—and on many S3-compatible object stores, it isn't—you are left with "ghost" files that weren't cleaned up by the failed task but aren't technically part of the final manifest.
In my experience, relying on checkpointLocation without a high-performance, atomic storage layer is like trying to build a skyscraper on a swamp. You aren't doing exactly-once; you are doing "at-least-once plus a prayer that the infrastructure handles the retry gracefully." The sink is the bottleneck
True exactly-once in a streaming system requires a two-phase commit protocol or an idempotent sink. Spark Structured Streaming implements this only when both the source and the sink support it natively. For example, if you are reading from Kafka and writing to Kafka, Spark can participate in the transaction.
Roughly 80% of the "exactly-once" guarantees I see implemented in production financial pipelines are actually "at-least-once" disguised by a fragile post-processing cleanup script.