The demo was flawless. Ten million rows, instant search, no waiting. You bought it. Six weeks later your real data is in, and the same search takes long enough that staff have started avoiding the screen.
Nothing was faked in that demo. That is the trap. The speed was real, on that machine, with that data, already warm. Demo speed and your-Tuesday speed are two different measurements, and almost nobody quotes both.
We ran a set of experiments on data reading inside our own internal tooling (research infrastructure of ours, not a client project, and said plainly). The lesson generalizes to any system you buy or build, and it is the reason we ask the questions most demos never answer.
The three ways software reads data
When software needs facts from a big file or table, there are three doors:
- Load it all. Pull the whole thing into memory (RAM, the fast working desk inside the machine) and search it there. Blazing fast while it fits. When the data outgrows the desk, the whole approach collapses at once.
- Stream it. Read the file in ordered pieces, front to back, using memory the size of one piece. Works on any amount of data. Punishes anything that needs scattered, jump-around access.
- Map it. Memory mapping (the technical name is mmap) asks the operating system to make the file pretend to be memory. The program touches it like an array, and the OS fetches pages from disk behind the scenes, only the pages actually touched. The desk never holds the whole file; the filing cabinet sends pages on demand.
The interesting engineering is rarely picking one. Our experiments ran a hybrid: map the parts worth jumping around in, stream the parts read in order, and keep a switch that turns the mapping off entirely so the same workload runs both ways. That switch (the plain word is an ablation: disable the trick, measure the trick) is what separates measurement from marketing.
The trap the experiment exists for
Here is the failure that eats buyers: run the search once, it crawls. Run it again, it flies. Conclusion: the mapping works. Conclusion is wrong.
The second run was fast because the operating system kept the pages in memory from the first run (that cache is called the page cache). The disk never got faster. The demo machine, warmed by rehearsal, was showing you its cache, not its speed. Our own working rule from these experiments is one line: never mistake a warm cache for a fast disk.
The honest version of the experiment runs cold, on data bigger than memory, with the trick switched on and off, and reports both numbers. Anything less is a rehearsal.
The failure path is the product
The other lesson came from testing, not speed. Part of this work used sealed files (read-only files the operating system itself refuses to let anyone modify) so a bug could not corrupt data. One of our tests started failing intermittently: 173 failures in 400 runs. The file was never actually leaking. A nearby thread in the test was nudging a global counter the test counted on. The test was measuring the neighborhood, not the house.
Rewritten to check the exact thing it claimed to check, the same test found zero leaks and stopped crying wolf. The business translation: a test that fails for the wrong reason trains everyone to ignore alarms, and an ignored alarm is how real failures ship. When you pay for a system, you are paying for the tests to be the kind that mean something on a bad day.
What to ask, whatever you buy
- “What happens when the data outgrows memory?” A system designed on load-it-all has a cliff. The answer names the design; the honest answer names the cliff.
- “How was the speed claim measured?” You want to hear: cold start, data larger than memory, trick on and off. Anything else is a warm machine telling you a story.
- “What does the system do when a file is missing halfway through?” Data work lives or dies on the failure path, and the failure path is testable in a demo if you ask before you sign.
Read our guide on what a passing test actually proves, and if a speed demo is sitting in your inbox waiting for your signature, discuss your project with us first. We will tell you which questions to put to it, and the answers will tell you whether you need us at all.
Common questions
What is memory mapping (mmap), in one sentence?
An arrangement where the operating system makes a file on disk pretend to be a section of memory, so the program reads it like a simple array while the OS quietly fetches only the pages actually touched. It trades the program’s memory budget for disk traffic, which is exactly why it shines for scattered reads on big files and disappoints when the workload does not match.
Why would anyone still load everything into memory?
Because while the data fits, nothing is faster: memory access beats every disk arrangement. Small reference datasets (a currency table, a product list) are better loaded. The design error is assuming this year’s data volume is the ceiling. Ask what the design does at ten times the data, and you will hear which door the system is actually built on.
What is a page cache?
Memory the operating system uses to keep recently read disk pages around, so the next read of the same page does not touch the disk at all. It is why the second run of anything is faster than the first. It is also why a rehearsed demo outruns a production Tuesday: the demo’s speed is partly the cache talking.
Was this experiment done on client work?
No. It is research on our own internal tooling, run to sharpen the judgment we use on client systems, and it is labeled that way on purpose. What client work proves is separate: Milyon Digital is our live client site, and our demonstrations are labeled demonstrations. We keep the two honest because a research claim dressed as a client result is the exact fraud this blog exists to prevent.