Time-Series Storage: How to Evaluate Encoding and Compression for IoT Data
A developer-oriented guide to testing storage efficiency without losing the query and ingestion behavior that makes telemetry useful.
Spots 
When an IoT deployment keeps data for months
When an IoT deployment keeps data for months or years, storage settings become part of the application design. A database may receive millions of measurements, but the important question is what those measurements look like after they are stored: can the system retain the required history, keep accepting new data, and answer time-window queries predictably? This is where encoding and compression matter. They are connected, but they are not interchangeable terms. Encoding versus compression
Encoding chooses a representation for values before they
Encoding chooses a representation for values before they are stored. Time-series data often has useful structure: timestamps follow a time axis, neighbouring values may be similar, and each measurement has a stable type and device context. Compression then reduces redundancy in that representation. A simplified pipeline looks like this:
The goal is not merely to make a
The goal is not merely to make a file smaller. The stored representation still has to support continuous writes, historical reads, aggregation, recovery, and whatever operational processes depend on the data. Start with the data profile
Do not select a setting from a single
Do not select a setting from a single compression-ratio example. First describe the series you actually expect to store:
Different signals can behave very differently. A slowly
Different signals can behave very differently. A slowly changing temperature series may contain more repeated structure than a noisy vibration series. A counter has a different pattern again. A realistic sample should include the main types rather than replacing them with random values. Match the method to the signal
Apache IoTDB V2.0.x provides encoding methods for different
Apache IoTDB V2.0.x provides encoding methods for different data types and patterns. Three examples make the selection logic concrete: TS_2DIFF is suited to monotonically increasing or decreasing integer sequences. RLE is useful when values repeat consecutively.
GORILLA is designed for nearby consecutive numeric values
GORILLA is designed for nearby consecutive numeric values and is a better fit than differential or run-length approaches for some floating-point series.
Compression follows encoding and operates on the binary
Compression follows encoding and operates on the binary stream. V2.0.x supports several compression methods, including LZ4, SNAPPY, GZIP, ZSTD, and LZMA2. Treat them as candidates to test rather than a fixed best-to-worst list.
A useful experiment changes one major storage variable
A useful experiment changes one major storage variable at a time and records the surrounding conditions. At minimum, capture: stored size for the same input data; write acceptance and end-to-end availability; CPU and memory during writes and reads; recent-value and time-range query behavior; aggregation behavior over longer windows; compaction, recovery, backup, or retention effects relevant to the deployment.
News
Time-Series Storage: How to Evaluate Encoding and Compression for IoT Data
A developer-oriented guide to testing storage efficiency without losing the query and ingestion behavior that makes telemetry useful.
@spots #dev
Source: Dev.to