← Research blog

RESEARCH / STATE

Two Views of State Inside a Cache

Cosmos SDK merges pending writes with stored keys; mapping those views comes before judging what a range operation can change.

Model the store before judging a range operation

The security question here starts with the storage model, not with a suspicious Delete call. In Cosmos SDK v0.53.0, a cachekv.Store sits over a parent KVStore. Its Get path checks the cache and then the parent; its Set and Delete paths change the cache. Write later sends dirty entries to the parent. A key can therefore be readable without appearing among the writes of the current cache layer.

RECONSTRUCTED SOURCE MAPRead view and pending write layer
Cosmos SDK v0.53.0

Scroll the diagram horizontally

Parent KVStore and current cache feed cachekv.Iterator for an effective range view. The current cache also feeds dirtyItems, which enumerates changed entries and writes them to the parent.
PINNED SOURCE

This map has two independent pieces of evidence: Iterator opens both parent and cache iterators, while dirtyItems walks pending entries. They are different sets by design. The cacheMergeIterator implementation also has to resolve duplicate keys and tombstones, so simply concatenating both iterators would not reproduce the effective read view.

Only after mapping these roles does a bulk deletion become an audit question: does its caller obtain keys from the effective iterator, or from a pending-write index? The existence of dirtyItems is not itself a defect. The answer depends on the operation’s promised scope and its actual call site.

A merged iterator in public code

Cosmos SDK v0.53.0 gives its cache store an iterator over two sources. In store/cachekv/store.go, lines 199–210, the ascending case opens one iterator on the parent store and one on the sorted cache:

parent = store.parent.Iterator(start, end)
cache, err = isoSortedCache.Iterator(start, end)

The function returns internal.NewCacheMergeIterator(parent, cache, ascending). The source establishes the inputs to this iterator: previously stored entries and pending cache entries both participate. It does not tell us which application operations call the iterator.

The scope lives in the view

“All keys in a range” is a claim about a view, not merely about a loop. A bulk operation that enumerates only pending writes and one that enumerates the merged view can have different results even when each loop is implemented correctly.

Take a synthetic state with A in the parent, B in the cache, and a deletion marker for C. An operation confined to the write layer cannot enumerate A; a merged view has to account for it. A test that creates and removes everything within one write layer cannot distinguish the two views.

Parent-only control

For a range-wide operation, identify its iterator call site and the meaning of its advertised scope. Put a parent-only key, a cache-only key, and a deletion marker into a small fixture; run the operation, commit, reopen, and inspect the effective state. The result answers whether the implementation matches its own contract. A security conclusion would require a separate downstream use of any unexpected survivor.

The merge rules, not just the inputs

The public implementation offers a more precise answer than “it merges two iterators.” Get checks the cache first and falls back to parent.Get on a miss; Delete records a nil value in the cache (store.go, lines 53–93). An entry read from the parent may therefore appear in the cache without being a pending write. The distinction matters when someone inspects the cache map and assumes that every entry represents a mutation.

For duplicate keys, cacheMergeIterator.Value returns the cache value. If the cache value is a deletion marker, skipUntilExistsOrInvalid advances the parent and cache iterators together, hiding the parent value. If the parent key sorts before the next cache key, that parent key stays visible. These rules determine which keys a range operation can act on.

Synthetic input, range [A,D):
parent: A=old, C=old
cache:  B=new, C=<deleted>
merged visible keys: A, B
overlay-only visible keys: B, C's deletion marker

An overlay-only sweep cannot discover A; the merged iterator can. Whether that difference is a bug depends on the caller’s promised scope. An API explicitly confined to pending writes should not be judged by the contract of a full-range clear.

Tests that exercise both layers

The upstream TestCacheKVMergeIteratorDeleteLast writes five keys to the parent, adds five more in the cache, asserts that ten are visible, and then deletes keys while checking the visible count. Another test places alternate deletion markers over written keys and compares the iterator with a reference database.

A range-clear test needs a further step: commit the operation, reopen the store, and inspect the effective view. Checking only the current write layer can miss a parent-only survivor. Use ascending and reverse iteration, plus keys exactly on the range boundaries, to separate a layer mistake from a range-endpoint mistake. Finally, trace a reachable reader that relies on the range being empty. Without that downstream reader, an unexpected key may be a correctness problem rather than demonstrated security impact.

Why an apparently complete test can mislead

There are at least three different experiments hidden in the phrase “delete every key.” If every key is created and deleted before Write(), all keys live in the same layer; even an overlay-only enumerator can appear complete. If the test writes the keys first and then deletes them by an explicit list of known names, it never exercises range enumeration. The discriminating experiment writes A to the parent, opens a new cache, and asks the operation under review to discover A without handing it the key. The visible post-operation state must then be checked through the same read API that later consumers use.

Cosmos SDK’s cache write path is a useful reminder that enumeration and persistence are separate stages. A correct iterator does not by itself prove the caller called Delete on every returned key, wrote the cache, or committed the surrounding transaction. Conversely, a write-back routine may work perfectly while its caller supplied an incomplete key set. Record both the enumerated key list and the reopened state; otherwise those two failure modes look identical.

There is also a performance trap that should not be confused with the security claim. dirtyItems, lines 295–320, sorts pending entries relevant to the range and even comments on the cost of interleaving iteration with writes. A range sweep can be expensive without being incomplete, and an optimized overlay scan can be fast while violating a full-view contract. The benchmark question and the semantic question need different evidence.

MORE RESEARCH

Explore more research.

Research blog