Efficiently sorting gigabytes: the Deferred Merge Sort
At Superluminal we’re dealing with massive files that may contain unsorted data. This post explains how we stream and sort these huge capture files in one pass.
At Superluminal we’re dealing with massive files that may contain unsorted data. This post explains how we stream and sort these huge capture files in one pass.
In Superluminal we need to store potentially millions of callstacks very efficiently. This article describes how we went from naive storage taking up gigabytes of memory in the worst case, to a very compact data structure that efficiently compresses and stores large amounts of stack data in a fraction of the memory.