Data lakes have become the default storage layer for big data. They're cheap, scalable, and flexible. But they're not designed for low-latency online queries. If you need to look up a single record—a point query—scanning a data lake is painfully slow.
Indexing changes that. By building secondary indexes on data lake storage, you can enable fast, low-latency lookups without moving data into a separate database.
The challenge
Data lakes typically store data in columnar formats like Parquet or ORC. These formats are optimized for batch analytics—scanning large ranges of data—not for random access. To find a single record, you might have to read entire files or partitions.
The scale makes it worse. Data lakes can hold petabytes of data across billions of files. Even with partitioning, point queries can be slow and expensive.
Indexing strategies
There are several approaches to indexing data lakes for point queries:
External indexes: Build separate index structures (e.g., in a key-value store or a search engine) that map keys to file locations. When a query comes in, look up the key in the index to find the right file, then read only that file. This works well but adds operational complexity and consistency challenges.
Embedded indexes: Store index metadata within the data files themselves. Parquet, for example, supports row group statistics that can help skip data. With careful partitioning and sorting, you can achieve decent performance without external systems.
Materialized views: Pre-compute and store the results of common point queries in a separate table or storage layer. This is essentially caching, but it can be very effective for high-frequency queries.
Hybrid approaches: Combine multiple strategies. Use partitioning to narrow the search space, embedded indexes to skip data within files, and external caches for hot keys.
Trade-offs
Indexing adds overhead. Indexes need to be built and maintained as data changes. The more indexes you have, the more storage and compute you need. There's also a consistency trade-off—indexes can become stale if updates aren't applied atomically.
But for the right use case—high-frequency point queries on large datasets—the benefits far outweigh the costs. The key is to index selectively, focusing on the queries that matter most.
The future
As data lakes become more operational, we can expect to see better built-in indexing capabilities. New file formats, table formats (like Iceberg and Delta Lake), and query engines are making point queries more feasible. In the future, the line between data lakes and databases may blur, with data lakes offering both batch and low-latency access.
For now, indexing data lakes for point queries is a practical, cost-effective way to bridge the gap between analytics and operations.