Data lakes are the central repositories of choice for many modern organizations. They offer flexibility and scalability at a low cost. However, they are notoriously poor at serving fast, point lookup queries. Unlike a database with a B-tree or hash index, a data lake often relies on full table scans or partition pruning, which is insufficient for interactive applications.
The Problem
An "online point query" is a request for a specific record, such as "Get the user record for user_id = 12345." In a traditional database, this would be a microsecond operation. In a data lake, it can take seconds or even minutes because the system has to scan files.
To solve this, data engineers have historically had to copy data from the data lake to a specialized database (e.g., DynamoDB, Cassandra) for serving these queries. This introduces complexity, latency, and cost.
Indexing Strategies
The industry is developing new strategies to index data lakes directly. The goal is to avoid moving the data while achieving database-like query speeds.
1. Skipping Indexes: This approach builds indexes that map key values to file locations. When a query comes in, the system consults the index to skip irrelevant files and only read the ones containing the requested record. This is effective for sparse data but can be inefficient for high-cardinality lookups.
2. Z-Ordering: This technique orders data within files so that similar keys are stored near each other. It allows for efficient range queries and point lookups by limiting the data that needs to be read.
3. Lookup Tables: For extremely high-traffic queries, a separate, small index table can be stored. This table maps the key to the physical location (file path, offset). The query engine first looks in the index table (which is fast to scan) and then makes a specific read request to the data lake file.
The Future of Data Lakes
The ability to index data lakes for point queries is a game-changer. It allows organizations to keep all their data in a single, cost-effective storage layer while still supporting the performance requirements of operational applications. This brings us closer to the vision of the "lakehouse" architecture, where data warehouses and data lakes merge into a unified system.