PB158 · Lecture 2 — I/O from Python to the operating system

<p class="kicker">PB158 · Lecture 2</p>

<h1>I/O from Python to the operating system</h1>

<p class="lede">The call you write is not the call the operating system runs. <code>fflush()</code> flushes only the C library's buffer [S3]. The kernel buffer needs a second call [S3]. Four boundaries sit below your code: app to library, library to kernel, kernel to device, and device to medium. A different tool watches each of the first three [V10] [S1] [V30]. No tool here watches the fourth [V5].</p>

<div class="author-block">
  <span class="name">Ondřej (Ondra) Krajíček</span>
  <a href="mailto:me@ondrejkrajicek.com">me@ondrejkrajicek.com</a> ·
  <a href="https://www.linkedin.com/in/OndrejKrajicek">www.linkedin.com/in/OndrejKrajicek</a>
  <span class="built">Built 2026-09-22 12:48 UTC</span>
  <span class="attribution">Generated by the agentic vault — an autonomous research and authoring harness. Every factual claim was gated against a cited source registry before this document was produced.</span>
</div>

<div class="footnotes">
[S1] strace(1) — Linux man-pages. https://man7.org/linux/man-pages/man1/strace.1.html
[S3] fflush(3) — Linux man-pages. https://man7.org/linux/man-pages/man3/fflush.3.html
</div>

Note:
Lecture one described how this course works. Lecture two is the first technical lecture. It runs for about an hour.

The frame is one sentence. You write Python and you write C. Your theory course told you what a system call is. Most of you have never watched one happen. This lecture closes that gap with a measurement rather than a definition.

Three programs run live on this machine. If a demonstration fails, every claim the argument relies on is also written on a slide. Nothing in the argument depends on the projector.

One housekeeping note. Everything written in this course is in English, including the slides and the code comments. The lecture is spoken in Czech. The written English is deliberately plain, so a student can read it and listen at the same time.

---

<p class="kicker">Last time · Lecture 1</p>

<h2>Where we stopped</h2>

<div class="rows">
<div class="row">
<div class="num">
1
</div>
<div class="term">
[P1] The course grades looking
</div>
<div class="desc">
Not reciting. Ondra built the assessment around what a student can show.
</div>
</div>
<div class="row">
<div class="num">
2
</div>
<div class="term">
[P2] The course works downward
</div>
<div class="desc">
From the code you write to the operating system under it.
</div>
</div>
</div>

<p class="closing">Ondra asserted both in Lecture 1. This hour is where he makes the first one good.</p>

Note:
Two claims came out of the orientation lecture, and both return in this hour.

The first is about grading: this course asks what a system does, not what a textbook says about it. Today that becomes one measurement and one trace.

The second is the direction of travel, and this lecture is the first descent.

---

<p class="kicker">Roadmap</p>

<h2>Roadmap</h2>

<div class="rows">
<div class="row">
<div class="num">
1
</div>
<div class="term">
What does one byte cost?
</div>
<div class="desc">
A stopwatch.
</div>
</div>
<div class="row">
<div class="num">
2
</div>
<div class="term">
Where does the difference come from?
</div>
<div class="desc">
A tracer.
</div>
</div>
<div class="row">
<div class="num">
3
</div>
<div class="term">
It is not one buffer — it is a stack
</div>
<div class="desc">
A diagram.
</div>
</div>
<div class="row">
<div class="num">
4
</div>
<div class="term">
The same write, at three levels
</div>
<div class="desc">
A program.
</div>
</div>
<div class="row">
<div class="num">
5
</div>
<div class="term">
Two operating systems, side by side
</div>
<div class="desc">
Two traces.
</div>
</div>
<div class="row">
<div class="num">
6
</div>
<div class="term">
What we have not looked at
</div>
<div class="desc">
One open question.
</div>
</div>
</div>

Note:
Here is the shape of the hour, so you know where the payoff sits.

Sections one and two are twenty minutes on a single question. Why do two programs that write the same bytes take very different times? The answer is a number you can count, and by the end of section two you will count it yourself.

Section three is the structural part. I draw one diagram there, and I point back at it in the durability lecture, the memory mapping lecture and the crash analysis lecture.

In sections four and five I spend the tracer, first on one program at three levels, then on two operating systems. In section six I deliberately leave one question open, and that question is the bridge into the rest of the course.

Assignment one measures what section one measures, so you will have the model to explain your own numbers.

---

<p class="kicker">Where we are going</p>

<h2>Takeaways</h2>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">[T1] Adjacency decides the call count</div><div class="desc">The saving is in calls, not work, and only for adjacent bytes. A seek empties the buffer.</div></div>
<div class="row"><div class="num">2</div><div class="term">[T2] A library call is not a system call</div><div class="desc">Several levels sit beneath yours [S3].</div></div>
<div class="row"><div class="num">3</div><div class="term">[T3] The boundary is observable</div><div class="desc">A tracer shows what your line became [S1].</div></div>
<div class="row"><div class="num">4</div><div class="term">[T4] The platforms differ by design</div><div class="desc">Each nominates a different level [V8].</div></div>
</div>

Note:
These four claims are the ones to hold in a week. The list comes first, so each section has a stated purpose.

Number one is the one you use on Monday, and it is what assignment one measures. Notice that it is not the rule "buffering is fast". It carries a condition, and the condition is the interesting part. Lampson put sequential access in the same sentence as his batching advice in 1983 [S12]. The seek half is the speaker's own, argued in part two.

Number two is the structural claim, and it is the reason for this lecture.

Number three is the promise about the instrument. By the end of section two you will read a trace line, and you will know which parts of it to believe.

Number four is the platform claim. Some of you run Windows. This lecture does not rank one system above the other. It shows that the two systems nominate different levels as stable.

The last section brings all four back, and each section earns one of them.

---

<p class="kicker">Scope</p>

<h2>What I cover, and what I leave out</h2>

<div class="compare">
<div class="side-a">
<p class="kicker">Today</p>
<div class="rows">
<div class="row"><div class="num">1</div><div class="desc">Why two programs differ</div></div>
<div class="row"><div class="num">2</div><div class="desc">When a buffer helps</div></div>
<div class="row"><div class="num">3</div><div class="desc">The four levels</div></div>
<div class="row"><div class="num">4</div><div class="desc">Reading a trace</div></div>
</div>
</div>
<div class="side-b">
<p class="kicker">Not today</p>
<div class="rows">
<div class="row"><div class="num">5</div><div class="desc">When a write is safe</div></div>
<div class="row"><div class="num">6</div><div class="desc">Memory mapping</div></div>
<div class="row"><div class="num">7</div><div class="desc">Operating the tools</div></div>
<div class="row"><div class="num">8</div><div class="desc">Asynchronous I/O</div></div>
</div>
</div>
</div>

Note:
The left column is what assignment one measures. Every item in the right column carries a lecture number.

Three of the exclusions are the ones you will most want to ask about.

The first is durability. Today's material shows that "a successful return from write() does not make any guarantee that data has been committed to disk" [S2]. It does not show what does make that guarantee, because today's instrument cannot answer it. Today's trace stops at the system call boundary, and no trace separates a device that flushed from one that only reported it [V5]. The durability lecture answers it, with a different method.

The second is running the tools. You have no toolchain installed yet, and lecture one told you to install it before the first lab. So today you read traces, and the demonstrations run at the front.

The third is asynchronous I/O. It changes the picture, and one sentence near the end covers it.

Everything here has a lecture number attached, and each number comes up as the lecture passes it.

---

<p class="lede">Two programs. The same mebibyte. Predict the ratio before we run them.</p>

Note:
Nine minutes, and this part contains no theory at all. It contains two programs and a stopwatch.

The order is deliberate. The usual way to teach this is to explain the buffer first and then show a benchmark that confirms it. That teaches you to trust the explanation. This part reverses the order. The number comes first, and the explanation answers a question the number has already raised.

Write your prediction down. Do not keep it in your head.

---

<p class="lede">Two programs write the same mebibyte to a file. That is 1,048,576 bytes. One uses a buffer. One does not.</p>

```python
# A: unbuffered — one call per byte
with open("a.out", "wb", buffering=0) as f:
    for i in range(1024 * 1024):
        f.write(b"x")

# B: buffered — the default
with open("b.out", "wb") as f:
    for i in range(1024 * 1024):
        f.write(b"x")
```

One argument differs. Python documents it: "Pass 0 to switch buffering off (only allowed in binary mode)" [S6].

<div class="footnotes">
[S6] Built-in Functions, open() — Python documentation. https://docs.python.org/3/library/functions.html#open
</div>

Note:
Read both programs now. The next twenty minutes depend on you believing they are the same program apart from one argument [S6].

Same loop. Same iteration count. Same interpreter. Same bytes, in the same order, to the same kind of file.

Notice also that you have to ask for the unbuffered version. It is not the default. You cannot write program A by accident, and that will matter when we decide what the practical lesson is.

One word about the word buffer, because I use it in exactly one sense today. A buffer is memory that collects bytes for consecutive positions in the file. The next level then takes them in one call instead of many. It saves calls. It does not save work.

---

<h2>Two programs, two access patterns</h2>

<div class="kv">
  <div class="pair"><div class="k">Sequential access</div><div class="v">Each write continues where the last one ended.</div></div>
  <div class="pair"><div class="k">Random access</div><div class="v">Each write follows a move to a new offset.</div></div>
  <div class="pair"><div class="k">Why it matters</div><div class="v">Lampson advised batch processing in 1983. He set sequential access aside from that advice, with the words "even aside from" [S12].</div></div>
</div>

<div class="footnotes">
[S12] Hints for Computer System Design — Lampson (1983), p. 19. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/acrobat-17.pdf
</div>

Note:
Two terms, and I hold them to these meanings for the rest of the course.

Sequential access means each write continues where the last one ended. Random access means each write follows a move to another offset. Do not say scattered and do not say non-linear. Say random access, because that is what the assignment says.

Both programs on the last slide use sequential access. Assignment one asks you to run the comparison a second time with random access. It also asks you to explain the difference. Listen to section two with that in mind.

The vocabulary is not new. Lampson wrote both terms down in 1983, in a paper of design hints [S12]. Use batch processing if you can, he says, because doing things incrementally almost always costs more. That reason ends with a clause — even aside from the fact that disks and tapes work much better when accessed sequentially [S12]. Those three words set the clause apart, and he does not rest the advice on it.

That sentence is forty-three years old, and it names the two access patterns you measure next week.

---

<h2>Predict before we run</h2>

<p class="lede">Write down a ratio. How much slower is the unbuffered program?</p>

<div class="stats">
  <div><div class="stat">1×</div><div class="stat-cap">nothing worth measuring</div></div>
  <div><div class="stat">10×</div><div class="stat-cap">an optimisation</div></div>
  <div><div class="stat">100×</div><div class="stat-cap">a different kind of thing</div></div>
</div>

Writing a number down before the run changes what you do with the result. Taber calls the method predict-observe-explain, and reports that "making a prediction will give students some sense of ownership in the following activity" [S14].

<div class="footnotes">
[S14] Predict-Observe-Explain — Taber. https://science-education-research.com/teaching-science/constructivist-pedagogy/predict-observe-explain/
</div>

Note:
Pick one of the three numbers. Write it on paper. Take thirty seconds [S14].

I will ask for a show of hands, and I want you to decide before you see the answer. This is not a game. The method has a name, and Taber describes it for exactly this case, where a room holds a strong intuition the machine does not match [S14].

Hands up for one times. Hands up for ten times. Hands up for one hundred.

Now I run both programs on this machine and we see which hand was closest.

Whatever number appears is a fact about this machine, this Python version and this filesystem. Your number will differ, and that turns out to be the lesson rather than a nuisance.

---

<h2>The measurement runs here, not on a slide</h2>

<p class="lede">I print no ratio here, on purpose. A benchmark you did not watch me produce is the kind of claim this course teaches you to distrust.</p>

<div class="kv">
  <div class="pair"><div class="k">Held fixed</div><div class="v">Loop, interpreter, iteration count, bytes, filesystem.</div></div>
  <div class="pair"><div class="k">Varied</div><div class="v">One argument, <code>buffering=0</code> [S6].</div></div>
  <div class="pair"><div class="k">Write down</div><div class="v">Both times, the Python version, the buffer size.</div></div>
  <div class="pair"><div class="k">What it includes</div><div class="v">The operating system's own memory for file data, in both runs. Name the cache layer your number includes, because you cannot remove all of them [V19].</div></div>
</div>

Note:
Both programs run now, untraced, with a plain wall-clock timer.

The measurement has a limit. One mebibyte does not reach a disk in any way you could detect. It lands in the operating system's own memory and stays there. So this measures the cost of asking the operating system to do something, a million times against a few hundred. Name the cache layer your number includes, because you cannot remove all of them [V19].

Copy both times into your notebook. You need them for assignment one, and the review lecture on the sixth of October uses your numbers rather than the ones on this machine.

Now look at the ratio. Then look at the hands that went up. The intended reaction is that this stopped being an optimisation. Part one exists to produce that reaction.

---

<h2>Your laptop will not agree with this projector</h2>

<p class="lede">The buffer size is not a constant. It grew sixteen-fold in Python 3.14.</p>

<div class="kv">
  <div class="pair"><div class="k">Python 3.13</div><div class="v">The default "was set to 8192 since 16 years ago" [S7].</div></div>
  <div class="pair"><div class="k">Python 3.14</div><div class="v">"the buffer will typically be <span class="neu">128 kilobytes</span> long" [S6].</div></div>
  <div class="pair"><div class="k">Why</div><div class="v">Write performance improved "3 to 5 times" over the 8192-byte default [S7]. The reporter measured it. No independent reproduction.</div></div>
  <div class="pair"><div class="k">So</div><div class="v">Report your version and buffer size with every number.</div></div>
</div>

<div class="footnotes">
[S6] Python documentation, open(). [S7] CPython issue 117151. https://github.com/python/cpython/issues/117151
</div>

Note:
This deck prints no benchmark figure. This slide gives the reason.

Python's default buffer was eight kilobytes for sixteen years, and is now typically one hundred and twenty-eight [S6] [S7]. The change landed in Python 3.14. The project's own issue explains the old value [S7]. The maintainers chose eight kilobytes in 2008. The machine they chose it for ran a thirty-two bit operating system with under two gigabytes of memory [S7].

One caution on that multiplier. The issue author measured it himself, on Python 3.11, writing a file he had downloaded over HTTP [S7]. A core developer confirmed the direction in the same thread, and posted no numbers [S7]. So nobody reproduced the figure. The slide carries it as the maintainers' stated reason for the change, not as a figure you can rely on.

Think what that does to your homework. Say some of you run 3.13 and others run 3.14. Your crossing counts then differ by a factor of sixteen. That reason has nothing to do with what the assignment measures. So report the version and the buffer size beside every number. One line of Python prints it.

---

<h1>Where does the difference come from?</h1>

<p class="lede">One answer, stated immediately, and then the instrument that proves it.</p>

Note:
Twelve minutes. This part explains the number you just watched, and it introduces the instrument the rest of the course depends on.

I will tell you the answer immediately rather than build up to it. The answer is short and the interesting work is in the consequences.

First one elimination. It is not the disk. One mebibyte never reached a disk in either run. It is not the interpreter either, because both runs share it. So whatever the difference is, it happens between your loop and the operating system.

---

<h2>Where does the difference come from?</h2>

<p class="lede">The difference is the number of times each program crosses into the operating system.</p>

Your theory course defined a system call. Today I count them: each one makes a **crossing** out of your program, "a privilege transition, not a jump" [V15].

<div class="rows">
<div class="row"><div class="num">A</div><div class="term">Unbuffered</div><div class="desc">One call per byte, so <span class="neg">1,048,576</span> calls.</div></div>
<div class="row"><div class="num">B</div><div class="term">Buffered</div><div class="desc">One mebibyte divided by the buffer size.</div></div>
<div class="row"><div class="num">=</div><div class="term">So measure first</div><div class="desc">At 8 KiB, <span class="pos">128</span> calls. At 128 KiB, 8 calls.</div></div>
</div>

Note:
Here is the answer. The two programs differ in the number of crossings they make into the operating system.

Program A asks once per byte. A mebibyte is one million and forty-eight thousand five hundred and seventy-six bytes, so that many requests.

Program B asks once per buffer fill. Divide one mebibyte by the buffer this machine allocated. Print the size before you divide, because I am not going to guess it for you.

Be precise about the vocabulary now, because I use it for the rest of the course. You already know what a system call is, so I am not going to define it. I am going to name the thing I count. The passage out of your program and back is one crossing, and every crossing is one system call.

Do not call it a kernel call. Do not call it an API call. An API call may or may not become one, and that distinction is what I am here to draw.

---

<h2>What one crossing costs</h2>

<div class="stats">
  <div><div class="stat">4,972</div><div class="stat-cap">instructions, one <code>stat</code> call</div></div>
  <div><div class="stat">13,585</div><div class="stat-cap">cycles for the same call</div></div>
  <div><div class="stat">2,559</div><div class="stat-cap">L3 lines evicted — your program was running</div></div>
</div>

Soares and Stumm measured this on 2010 hardware [S16]. The counts are not current [V44]. What carries over is the regime distinction, not the numbers. That distinction is a call rate. The lost cache state dominates at one call per 2,000 to 100,000 instructions. The transition cost takes over above that rate [S16].

<div class="footnotes">
[S16] <span class="src-title">FlexSC</span> — Soares and Stumm, OSDI 2010. https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Soares.pdf
</div>

Note:
Why is a crossing expensive? Your theory course probably said the processor changes privilege level. That is true, and it is not the main cost [S16].

Soares and Stumm measured what one system call does to the processor. Retiring an instruction means finishing it, and one `stat` call retires 0.37 instructions per cycle [V2]. That is roughly one finished instruction every three cycles. Straight-line code on the same processor finishes more than one per cycle, so this is slow work rather than ordinary work.

Now the third number. That one call evicted two and a half thousand lines from the level three cache, the processor's own last cache before main memory [V2]. The paper counts the lines. It does not say whose data they held, and your program was the code running. Every one of those lines your code touches again is a miss it pays for after the call returns.

Two caveats. The measurement is from 2010 hardware, so do not quote the counts as current. And the paper separates two regimes by call rate [S16]. In the medium band, one call per two thousand to one hundred thousand instructions, the lost cache state dominates. Above that rate the transition itself takes over [S16].

Program A makes one system call per loop iteration, so it probably sits above that band. That placement is mine, and I have no count of the interpreter's instructions per call.

Either way the cost is largely fixed per call. Doubling the bytes in one call is nearly free. Doubling the calls is not.

---

<h2><code>strace</code>: the tool that watches the boundary</h2>

<p class="lede">A tracer runs your program and writes down its system calls [S1] [T3].</p>

<div class="kv">
  <div class="pair"><div class="k">What it does</div><div class="v">It "intercepts and records the system calls made by a process" [S1]. It never samples.</div></div>
  <div class="pair"><div class="k">Where it comes from</div><div class="v">Paul Kranenburg wrote the original for SunOS. Branko Lankester ported it [S1]. The licence is the Lesser GPL [S30].</div></div>
</div>

```text
strace ./iodemo                 # one line per system call [S1]
strace -c ./iodemo              # counts, errors, and times not to trust [S1] [V13]
strace -e trace=write ./iodemo  # only the writes [S1]
```

<span class="neu">It is a Linux tool</span> [S30]. It costs speed [S1]. I use NtTrace in Part V [S10].

<div class="footnotes">
[S1] strace(1) man page. https://man7.org/linux/man-pages/man1/strace.1.html
[S30] strace. https://strace.io/
[S10] NtTrace. https://rogerorr.github.io/NtTrace/
</div>

Note:
You cannot read a trace until you know what produced it.

What it does. Its manual page says it intercepts and records the system calls a process makes, and the signals it receives [S1]. So it sits at the boundary and records the crossings. It records rather than samples, which is why you may believe its counts.

The project page calls it "a diagnostic, debugging and instructional userspace utility for Linux" [S30]. It runs on Linux only. The manual's history says that in 2012 "unmaintained and apparently broken support for non-Linux operating systems was removed" [S1].

Where it comes from, because somebody built this tool. Paul Kranenburg wrote the original for SunOS, and Branko Lankester ported it to Linux [S1]. The licence is the Lesser GPL, and the repositories are public [S30]. It is already in the course dev container, a prepared Linux environment VS Code starts for you. In the lab you run it yourselves.

What it costs. The manual page lists this under BUGS: a traced process runs more slowly than a non-traced one [S1]. Figures come later. Never benchmark under the tracer.

How you use it. `iodemo` in those commands is the small C program that part four compiles. The same forms work on today's Python pair. Plain `strace` gives one line per call. The next slide shows one. `-c` gives a table instead, and `-e trace=write` keeps only the writes [S1].

---

<h2>What one trace line says</h2>

<p class="lede">A trace records the system calls a program made, in order, with arguments and return values.</p>

```text
open("/dev/null", O_RDONLY) = 3
```

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">The name</div><div class="desc">Which system call ran [S1].</div></div>
<div class="row"><div class="num">2</div><div class="term">The arguments</div><div class="desc">In parentheses, decoded [S1].</div></div>
<div class="row"><div class="num">3</div><div class="term">The return value</div><div class="desc">After the equals sign [S1].</div></div>
</div>

A trace is not a log. A log carries what the program's author chose to print. A trace carries what the program did.

<div class="footnotes">
[S1] strace(1) — Linux man-pages, which documents this example line. https://man7.org/linux/man-pages/man1/strace.1.html
</div>

Note:
This is one line of its output, and it is the example the manual page itself uses [S1].

Read it left to right. The name of the system call. Then the arguments in parentheses, decoded into names rather than raw numbers. Then an equals sign and the return value.

That is the whole format. Every line looks like this. Once you can read one line you can read a thousand, and the rest of this course assumes you can.

One word about vocabulary. A trace is not a log. A log contains what the author decided to print. A trace contains every call the program made, including the ones the author never printed. So a tracer answers questions the author did not anticipate.

Read that line again, slowly.

---

<h2>Count the crossings</h2>

<p class="lede">The same two programs, run again under a counting tracer.</p>

<div class="kv">
  <div class="pair"><div class="k">Command</div><div class="v"><code>strace -c -e trace=write python3 a.py</code> [S1]</div></div>
  <div class="pair"><div class="k">Program A</div><div class="v">One <code>write</code> per byte.</div></div>
  <div class="pair"><div class="k">Program B</div><div class="v">One <code>write</code> per buffer fill.</div></div>
  <div class="pair"><div class="k">First</div><div class="v">Print <code>io.DEFAULT_BUFFER_SIZE</code>, then divide [S6].</div></div>
  <div class="pair"><div class="k">Two runs</div><div class="v">A traced run cannot be timed [V17].</div></div>
</div>

<div class="footnotes">
[S1] strace(1) — Linux man-pages. https://man7.org/linux/man-pages/man1/strace.1.html
</div>

Note:
I am running the counting version now. The filter matters. Before your code starts, Python itself opens and reads a large number of files. Without the filter you get every call that work makes, and the one interesting line disappears into the noise [S1].

Look at the two summaries side by side. First I print the buffer size this machine uses. Then I divide one mebibyte by it. Then I compare my result with the count the tool reported. If those two disagree, the arithmetic used the wrong divisor. Finding that out is the exercise working rather than failing.

This is what I mean when I say the boundary is observable. You run one command and you get the exact number of requests your program made. Not an estimate, and not a model. Program A was slow because it made a million of them.

---

<h2>The count ratio over-predicts the time</h2>

<p class="lede">Illustration, not a measurement from this room: a count ratio of 100,000 beside a time ratio of 40 is not a mistake.</p>

<p class="model"><code>time ≈ crossings × fixed cost  +  bytes × marginal cost</code> [V1]</p>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">Two terms</div><div class="desc">Not a function of bytes alone [V1].</div></div>
<div class="row"><div class="num">2</div><div class="term">One term is shared</div><div class="desc">Both programs write the same mebibyte through the same loop.</div></div>
<div class="row"><div class="num">3</div><div class="term">So counts over-predict</div><div class="desc">Under this model the count ratio exceeds the time ratio.</div></div>
</div>

Note:
I added this slide because of a sentence I was tempted to say and will not. The tempting sentence is that the ratio of the counts is the ratio of the times. It is false, and the run you just watched shows it false [V1].

Now apply it. Both programs write one mebibyte through the same million-iteration loop. So the second term, and all the interpreter work, is identical. It is a floor neither program can get under.

Under that model the count ratio over-predicts the time ratio, and the shared floor is why. If your counts differ by a hundred thousand and your times by forty, you measured the floor rather than making a mistake. Those two figures are an illustration, not a measurement from this room.

One caveat on the model itself. My note grades it confident rather than verified [V45]. And the per-crossing cost is not one number. One reported measurement spans 76 to 668 cycles for the same calls, across fifteen hosts [V46]. Treat the model as a shape, not as a formula to solve. You will meet this gap again in assignment one.

---

<h2>What a buffer actually does</h2>

<p class="lede">A buffer collects adjacent bytes and hands them down in one call.</p>

<figure class="diagram"><svg width="592pt" height="44pt"
 viewBox="0.00 0.00 592.00 44.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 40)">
<title>buffering</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-40 588,-40 588,4 -4,4"/>

<g id="node1" class="node">
<title>loop</title>
<polygon fill="none" stroke="black" points="109,-36 0,-36 0,0 109,0 109,-36"/>
<text text-anchor="middle" x="54.5" y="-21.2" font-family="Helvetica,sans-Serif" font-size="11.00">your loop</text>
<text text-anchor="middle" x="54.5" y="-9.2" font-family="Helvetica,sans-Serif" font-size="11.00">1,048,576 writes</text>
</g>

<g id="node2" class="node">
<title>buf</title>
<polygon fill="#eeeeee" stroke="black" points="347,-36 203,-36 203,0 347,0 347,-36"/>
<text text-anchor="middle" x="275" y="-21.2" font-family="Helvetica,sans-Serif" font-size="11.00">buffer</text>
<text text-anchor="middle" x="275" y="-9.2" font-family="Helvetica,sans-Serif" font-size="11.00">collects adjacent bytes</text>
</g>

<g id="node3" class="node">
<title>os</title>
<polygon fill="none" stroke="black" points="584,-36 459,-36 459,0 584,0 584,-36"/>
<text text-anchor="middle" x="521.5" y="-21.2" font-family="Helvetica,sans-Serif" font-size="11.00">operating system</text>
<text text-anchor="middle" x="521.5" y="-9.2" font-family="Helvetica,sans-Serif" font-size="11.00">one crossing per fill</text>
</g>

<g id="edge2" class="edge">
<title>buf->os</title>
<path fill="none" stroke="black" d="M347.05,-18C378.98,-18 416.6,-18 448.63,-18"/>
<polygon fill="black" stroke="black" points="448.82,-21.5 458.82,-18 448.82,-14.5 448.82,-21.5"/>
<text text-anchor="middle" x="403" y="-21" font-family="Helvetica,sans-Serif" font-size="10.00">one buffer each</text>
</g>
</g>
</svg></figure>
Cost has two terms: the number of crossings, and the number of bytes [V1]. A buffer cuts the first and leaves the second alone, so the saving sits entirely in the call count.

Note:
Look at the picture. Your loop still runs a million times. Every one of those writes still happens, and every byte is still copied.

What changed is the right-hand arrow. Instead of a million crossings there are a few hundred, because the buffer holds bytes until it is full and then hands over a block.

I want to be precise about the word adjacent, because the next slide turns on it. A buffer holds bytes destined for consecutive positions in the file. It can do that because a block write to the operating system carries one offset and one length. Bytes going to one place travel together. Bytes going to two places cannot.

Say it back to yourselves in one sentence. A buffer saves calls, not work.

That is takeaway one, half stated. The other half is next, and it is the half the assignment measures.

---

<h2>Why a seek gives the saving back</h2>

<p class="lede">A buffer holds bytes for consecutive positions. Move elsewhere and those bytes cannot stay in it.</p>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">Constraint</div><div class="desc">Buffered bytes belong at the old offset.</div></div>
<div class="row"><div class="num">2</div><div class="term">Consequence</div><div class="desc">Every seek forces the buffer out.</div></div>
<div class="row"><div class="num">3</div><div class="term">Result</div><div class="desc">The buffered run converges on the unbuffered.</div></div>
</div>

The rule is <span class="neu">buffering is fast when writes are adjacent, and buys little when they are not</span> [T1]. Lampson gave the batching half in 1983 [S12].

Note:
If you keep one slide from today, keep this one. Assignment one is graded on the rule it states.

A buffer collects bytes for consecutive positions. Now suppose you write one byte, move to a different offset, and write another. The first byte belongs at the old offset. It cannot sit in the buffer waiting, because the buffer has one destination. So the library must hand it over before it can move.

Every seek forces the buffer out. Seek between every write and the buffer never holds more than one byte. The buffered program then makes the same number of crossings as the unbuffered one, and the trace shows it.

So the rule has a condition. Buffering is fast when writes are adjacent, and buys little when they are not. In 1983 Lampson wrote the batching half [S12]. The part about a seek emptying a userspace buffer is mine. It follows from the mechanism above rather than from him.

One honest limit. Adjacency is necessary and not sufficient. A program already writing a mebibyte in one call has nothing left to collect.

---

<h2>Does the seek itself cost a crossing?</h2>

<p class="lede">Yes. And a pair of system calls exists to remove it.</p>

<div class="kv">
  <div class="pair"><div class="k">Naive</div><div class="v"><code>lseek</code> then <code>write</code> — two crossings for one operation.</div></div>
  <div class="pair"><div class="k">Merged</div><div class="v"><code>pread</code> and <code>pwrite</code> "combine[s] file access and seek operations into a single command" [V3].</div></div>
  <div class="pair"><div class="k">Measured</div><div class="v">The merged pair cut the call count [V3]. For the same data volume the study also reports "a relatively reduced load in terms of overall duration" [V27].</div></div>
</div>

Note:
Somebody always asks this, so I answer it before you do.

Yes. Moving to a new offset is itself a request to the operating system. So a random-access run does not make one crossing per byte. It makes two.

That is worth knowing for your assignment, because the random pattern is worse than the sequential unbuffered pattern rather than merely equal to it.

And here is the evidence that the crossing count is the real variable. Linux offers a pair of calls that merge the move and the access into one call. A study replaced the two-call pattern with the merged pair. The call count fell [V3]. For the same data volume the study reports a reduced overall duration [V27].

I am not claiming that the crossing count was why those calls were added. I do not know that, and the source does not say it. My claim is narrower. Merging two crossings into one gave a relatively reduced overall duration for that workload [V27].

You will not need these calls for assignment one. I mention them as evidence.

---

<h2>Trust the counts. Do not trust the times.</h2>

<p class="lede">The tracer stops your program twice per call. That destroys the timing and leaves the counting intact.</p>

| evidence | what it says | take it as |
|---|---|---|
| Every call stops | so "the counts are trustworthy and the absolute times are not" [V13] | counts only |
| Gregg's `dd` loop | one byte per call, "runs 442× slower under strace" [V17] | his machine |
| Re-runs | "roughly 37× to 136×", other hardware [V31] | not a bound |

The range does not contain Gregg's figure. Each number measures one machine and one workload.

<p class="closing">What survives these three rows is the count — the only number on this slide you may use [V13].</p>

Note:
You have no reason yet to believe this tracer. This slide gives one. It also names which of the tracer's numbers you must not trust.

Because it stops every call that reaches the kernel, it counts every one of them. Nothing is sampled. That is why you may believe the counts.

One program is allowed to freeze another at each call, read its state, and let it continue. That is the whole mechanism here.

There is one gap worth knowing. Linux serves a few calls without a crossing at all, from a page it shares with your process. The tracer is silent on those, because there is nothing to intercept [V26].

Because it stops every call, it also makes your program much slower. Brendan Gregg's worst case is a dd loop copying one byte per call, which does almost nothing but system calls. It ran four hundred and forty-two times slower under the tracer [V17]. Later runs of that same loop report thirty-seven to one hundred and thirty-six times, on other hardware and with different processor security mitigations [V31]. Notice that the later range does not contain Gregg's figure.

None of the numbers is wrong, because the overhead follows the call rate of one program on one machine. A single overhead number is therefore not a fact about the tool.

So the rule for this course is: count under the tracer, time without it. That is why part one and part two used two different runs.

If you benchmark under a tracer in the lab, you will report nonsense.

---

<h2>Windows counts too — differently</h2>

<p class="lede">NtTrace, this course's Windows tracer, prints no counting summary. Both systems count anyway.</p>

| platform | where the count lives | the definition |
|---|---|---|
| Windows | `GetProcessIoCounters` fills `IO_COUNTERS` [S24] | `WriteOperationCount` — "The number of write operations performed" [S23] |
| Linux | the file `/proc/pid/io` | `syscw` — the "file write" system calls [S25] |

One source file, `--count` on both: Windows counted 1 write operation [S37], the Linux container counted 7. Take assignment one's counts in the <span class="neu">course dev container</span>.

<div class="footnotes">
[S23] IO_COUNTERS — Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-io_counters
[S24] GetProcessIoCounters — Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getprocessiocounters
[S25] proc_pid_io(5) — Linux man-pages. https://man7.org/linux/man-pages/man5/proc_pid_io.5.html
[S37] Both iodemo runs, 2026-09-15 — demo/iodemo-c/README.md
</div>

Note:
Some of you run Windows. Twenty minutes of Linux came first, and this slide gives the accurate reason why.

The counting summary is a feature of `strace`. NtTrace, this course's Windows tracer, is a different design and prints no such summary. Counting was the job here, so the demonstration used the tool that counts.

Neither system needed a tracer for this. Both were already counting, and reading the count costs one file read on Linux and one call on Windows.

Read the two definitions in the table again, because they are not the same sentence. The Linux one names system calls [S25]. The Microsoft one says "write operations performed" and never says what makes one [S23]. The next slide returns to that.

Both demo programs read their own count. Pass `--count`, and the program prints it at the end. On Windows the Python one calls `GetProcessIoCounters` through `ctypes`, the standard library's module for calling C functions directly. On Linux it reads `/proc/self/io`, the running process's own directory.

You do not have to write code at all. Windows lets one process read another process's count, given the right query permission [S24]. That is why a ready-made process viewer can show you the same numbers. Task Manager can add I/O columns on its Details tab — check it on your own machine rather than taking the claim on trust.

One measurement makes that concrete. On Windows `iodemo.exe --count` printed "write operations counted by the kernel: 1" [S37]. The same program in the Linux container printed 7 [S37]. The one write is the same, and the other six belong to the Linux loader [S37].

So the answer to "what about Windows" is not "wait for Part V". Your system already counted, and here is the field.

---

<h2>What the counters will not tell you</h2>

<p class="lede">Three limits, none visible in the count.</p>

| the limit | what it does to your count |
|---|---|
| Not one measurement | Linux counts the `write(2)` family of system calls [S25]. Microsoft says "write operations performed" [S23]. |
| Operations, not crossings | `syscw` counts only write operations [S25]. It misses a seek. |
| Start-up writes count too | The count covers the process, not your loop. Compare two runs. |

Part V reads a Windows trace I captured [S34]. Equal footing is a course promise, not a per-lecture one.

<div class="footnotes">
[S23] IO_COUNTERS (winnt.h) — Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-io_counters
[S25] proc_pid_io(5) — Linux man-pages. https://man7.org/linux/man-pages/man5/proc_pid_io.5.html
[S34] Our NtTrace64 cmd.exe capture.
</div>

Note:
Take each limit in turn, because each one changes what you may write in the report.

First, the two fields are not one measurement. Linux names the "file write" system calls, and counts them "including when invoked by the kernel as part of other syscalls" [S25]. So the Linux number is close to a crossing count and is not identical to one. Microsoft names "write operations performed" and stops there [S23], so the Windows number I cannot characterise at all.

Second, and this follows from the first: a seek is a crossing, and it is not a write. The Linux field counts calls from the write family, so it misses a seek. The random-access program seeks once per write, so its count is about half its crossings. The sequential program never seeks, so for that one the two agree. Microsoft has a separate member for the "I/O operations performed, other than read and write operations" [S23]. The page does not say a seek increments it, so neither will I.

Third, the count covers the whole process — on Linux, "the process and its waited-for children" [S25]. So take the count for each program. Then report the ratio, not either absolute number.

All three limits still leave the count useful. It answers the benchmark's question, and it needs no tracer to do it.

---

<h1>It is not one buffer — it is a stack</h1>

<p class="lede">We counted the crossings. Now we ask where the bytes actually are.</p>

Note:
Fifteen minutes, and this is the structural part of the lecture.

Section two left you with a good model and one obvious question. If the library holds my bytes and hands them down, where are they afterwards? The answer is not "in a buffer". It is "in one of several, and you cannot tell which without looking".

I draw one diagram in this section. I refer back to it in the durability lecture, in the memory mapping lecture and in the crash analysis lecture. If you photograph one slide today, photograph the next one.

---

<h2>It is not one buffer — it is a stack</h2>

<figure class="diagram"><svg width="540pt" height="227pt"
 viewBox="0.00 0.00 540.00 227.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 223)">
<title>stack</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-223 536,-223 536,4 -4,4"/>

<g id="node1" class="node">
<title>app</title>
<polygon fill="none" stroke="black" points="259,-219 0,-219 0,-189 259,-189 259,-219"/>
<text text-anchor="middle" x="129.5" y="-201.2" font-family="Helvetica,sans-Serif" font-size="11.00">1  Application buffer (Python)</text>
</g>

<g id="node3" class="node">
<title>ker</title>
<polygon fill="none" stroke="black" points="395,-156 136,-156 136,-126 395,-126 395,-156"/>
<text text-anchor="middle" x="265.5" y="-138.2" font-family="Helvetica,sans-Serif" font-size="11.00">3  Kernel page cache</text>
</g>

<g id="edge1" class="edge">
<title>app->ker</title>
<path fill="none" stroke="black" d="M161.04,-188.85C180.01,-180.34 204.35,-169.43 224.76,-160.27"/>
<polygon fill="black" stroke="black" points="226.44,-163.36 234.13,-156.07 223.57,-156.97 226.44,-163.36"/>
<text text-anchor="middle" x="229" y="-170" font-family="Helvetica,sans-Serif" font-size="10.00">f.flush()</text>
</g>

<g id="node2" class="node">
<title>lib</title>
<polygon fill="none" stroke="black" points="532,-219 273,-219 273,-189 532,-189 532,-219"/>
<text text-anchor="middle" x="402.5" y="-201.2" font-family="Helvetica,sans-Serif" font-size="11.00">2  C library buffer (C)</text>
</g>

<g id="edge2" class="edge">
<title>lib->ker</title>
<path fill="none" stroke="black" d="M370.73,-188.85C351.53,-180.31 326.88,-169.33 306.26,-160.15"/>
<polygon fill="black" stroke="black" points="307.66,-156.94 297.1,-156.07 304.81,-163.34 307.66,-156.94"/>
<text text-anchor="middle" x="364" y="-170" font-family="Helvetica,sans-Serif" font-size="10.00">fflush()</text>
</g>

<g id="node4" class="node">
<title>dev</title>
<polygon fill="none" stroke="black" points="395,-93 136,-93 136,-63 395,-63 395,-93"/>
<text text-anchor="middle" x="265.5" y="-75.2" font-family="Helvetica,sans-Serif" font-size="11.00">4  Device write cache</text>
</g>

<g id="edge3" class="edge">
<title>ker->dev</title>
<path fill="none" stroke="black" d="M265.5,-125.85C265.5,-119.15 265.5,-110.96 265.5,-103.31"/>
<polygon fill="black" stroke="black" points="269,-103.07 265.5,-93.07 262,-103.07 269,-103.07"/>
<text text-anchor="middle" x="326" y="-107" font-family="Helvetica,sans-Serif" font-size="10.00">fsync() / FlushFileBuffers</text>
</g>

<g id="node5" class="node">
<title>med</title>
<ellipse fill="#eeeeee" stroke="black" cx="265.5" cy="-15" rx="129.5" ry="15"/>
<text text-anchor="middle" x="265.5" y="-12.2" font-family="Helvetica,sans-Serif" font-size="11.00">the medium</text>
</g>

<g id="edge4" class="edge">
<title>dev->med</title>
<path fill="none" stroke="black" d="M265.5,-62.85C265.5,-56.15 265.5,-47.96 265.5,-40.31"/>
<polygon fill="black" stroke="black" points="269,-40.07 265.5,-30.07 262,-40.07 269,-40.07"/>
<text text-anchor="middle" x="324" y="-44" font-family="Helvetica,sans-Serif" font-size="10.00">a cache-flush command</text>
</g>
</g>
</svg></figure>
Levels 1 and 2 are alternative routes into level 3 [V6]. Each arrow is one call [S17] [S3] [S13] [V37] [V33].

<div class="footnotes">
[S17] io.IOBase.flush() — Python documentation. [S3] fflush(3) — Linux man-pages. [S13] fsync — The Open Group.
</div>

Note:
Four levels. Read them from the top.

Level one is your program's own buffer, owned by the language runtime. Level two is the C library's buffer. Level three is the operating system's own memory for file data. Level four is the cache inside the storage device. Then, at the bottom, the medium.

Notice that levels one and two sit side by side. They are alternative routes, not a pile. A Python program uses the buffer at level one and calls the kernel directly. A C program uses the buffer at level two. I come back to this in four slides, because it is the detail people get wrong [V6].

Now read the arrows. Each one is a different call, and each one moves your bytes down exactly one step. No single call takes data from the top to the bottom.

Look at the bottom arrow. I come back to it, and I am not going to skip it.

---

<h2>Four levels, and who empties each one</h2>

| level | who owns it | emptied by |
|---|---|---|
| 1 Application | the runtime's buffer [S8] | `f.flush()` [S17] |
| 2 C library | the `stdio` buffer [S3] | `fflush()` [S3] |
| 3 Kernel | the page cache [V32] | `fsync()`, Windows `FlushFileBuffers()` [V37] |
| 4 Device | the drive's write cache [V20] | a cache-flush command [V33] |

The page cache lives in the machine's memory [V32]. It is not the disk's cache — the disk has its own, one level further down [V20].

The kernel also moves data upward. It fetches pages ahead of a sequential reader, so "a sequential `read()` just happens to return faster because the data was already fetched" [V34]. The same levels work in both directions.

Note:
The same four levels as a table, because you will want to copy this one.

Rows one and two are the two alternatives, and the Windows half of the room should know the C runtime adds no third row. Microsoft says of the layer Python enters there: "Low-level input and output calls don't buffer or format data" [S28]. I walk that path in full three slides from now [S26] [S27].

Level three deserves its own name. The page cache is the operating system's own buffer for file data, and it lives in the machine's memory. A write that reaches it has left your program, and it has not left the machine.

Two things people get wrong. First, it is not the disk's cache. The disk has its own, and that is level four, one step further down. Two caches, two owners, two failure modes.

Second, a busy machine shows very little free memory because the page cache has grown into it. That is correct behaviour and not a leak. The memory is reclaimable the moment something else needs it.

One sentence about reading, because every demonstration today is a write. The same four levels carry data upward. Level three fetches pages ahead of a sequential reader, so the read returns from memory rather than from the device [V34].

Now look at the last row. The device has a cache of its own, and emptying that one is the least reliable step in the diagram. Hold that thought.

---

<h2>To flush is to cross one boundary</h2>

<p class="lede">The verb means one thing here: to hand a buffer's contents to the next level down.</p>

The C library manual says it plainly: "Note that fflush() flushes only the user-space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too" [S3].

<div class="rows">
<div class="row"><div class="num">✗</div><div class="term">Do not say</div><div class="desc">Save, commit, persist — all claim durability.</div></div>
<div class="row"><div class="num">✓</div><div class="term">Say</div><div class="desc">Flush. It names one boundary.</div></div>
</div>

<div class="footnotes">
[S3] fflush(3) — Linux man-pages. https://man7.org/linux/man-pages/man3/fflush.3.html
</div>

Note:
This misconception is the most important one to remove today, and it gets a slide of its own.

The common belief is that the verb flush means "written to disk". It does not. It means "handed to the next level down", and the manual page says so in its own words [S3].

Read the quote carefully. It flushes only the user-space buffers provided by the C library. To get data physically onto a disk, the kernel buffers must be flushed too [S3]. That is two calls, not one.

So in this course, flush is a verb with a precise meaning. It crosses one boundary. If you write "save" or "persist" in your assignment when you mean flush, the question back is which boundary you crossed.

A student who leaves believing that a program flushes all the way to the platter has learned something false. Ordinary use will not correct it, because only a power cut falsifies the belief.

---

<h2>Wholesale down, retail up</h2>

<p class="lede">Every level buys a large block from the level below. It sells small pieces to the level above.</p>

That asymmetry is why every level in this diagram holds a buffer. Each level's "unit of purchase is larger than its unit of sale" [V18].

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">Buys in blocks</div><div class="desc">4096 bytes on an ext4 file [V7].</div></div>
<div class="row"><div class="num">2</div><div class="term">Sells in bytes</div><div class="desc">Your loop writes one byte.</div></div>
<div class="row"><div class="num">3</div><div class="term">Same shape elsewhere</div><div class="desc">Allocators, caches, network stacks.</div></div>
</div>

Note:
Four levels is four facts, and four facts are hard to keep. One rule is easier, so here is the rule that generates all four [V18].

Each level buys wholesale from the level below and sells retail to the level above. Its unit of purchase is larger than its unit of sale. That asymmetry is why every level in the diagram holds a buffer.

Apply it to level two. Take an ext4 file, one of the common Linux filesystems. There the C library takes a buffer of four thousand and ninety-six bytes [V7]. It sells single bytes to your loop. The gap between those two units is the saving you measured in part one.

I give you the rule rather than the four facts because the rule travels. You will meet it in memory allocation, in processor caches and in network buffers.

I avoid one thing deliberately. I am not giving you a warehouse or a bucket for the page cache. Physical memory is a cache rather than a container, and a container image installs the wrong intuition.

---

<h2>One program, one buffer — C and Python take different paths</h2>

<p class="lede">Each program uses at most one buffer, and not the same one.</p>

```text
iodemo.c    fwrite()   →  stdio buffer (level 2)    →  write()  →  page cache
iodemo.py   f.write()  →  BufferedWriter (level 1)  →  write()  →  page cache
```

The C program's buffer is the C library's `stdio` [S3] [V7]. Python's is a `BufferedWriter` [S8]. `FileIO` calls `write()` itself, so no `stdio` buffer sits under Python on either platform [S26] [S27] [S28] [V6].

The four levels are the union of both paths: a program uses one of levels 1 and 2, never both. Python documents <span class="neu">128 kilobytes</span> as typical [S6]. On one ext4 file the C library took 4096 bytes, its block size [V7].

<div class="footnotes">
[S6] open() — Python documentation. https://docs.python.org/3/library/functions.html#open
</div>

Note:
I flagged this on the diagram and I am returning to it, because it is the detail people get wrong.

If you read the four levels as a stack, you expect a Python write to fall through level 1 into level 2. It does not. CPython reimplements the buffering policy above the C library rather than inheriting it [V6]. So follow the two programs instead of the diagram.

The C program calls `fwrite`. The C library collects the bytes in its own buffer, and that is level 2. The manual calls that level "the user-space buffers provided by the C library" [S3]. When the C library hands that buffer down, the bytes cross into the kernel — the manual calls the level below "the kernel buffers" [S3].

The Python program calls `f.write`. The object `open()` handed you is a `BufferedWriter`, and its buffer is level 1. Under it sits `FileIO`, which talks to the operating system. PEP 3116 named both and specified the layers in 2007 [S8]. CPython's `FileIO.write` "Only makes one system call" and calls `_Py_write` [S26], which calls `write()` [S27]. Nothing from `stdio` sits in between [V6].

Windows users, one sentence, because I closed this two slides ago. Microsoft says of the C runtime layer Python enters on your machine: "Low-level input and output calls don't buffer or format data" [S28]. The picture is the same on both platforms.

So the diagram's four levels are the union of the two paths. Each program uses at most one of levels 1 and 2, and never both. Both paths then meet in the kernel. A program opened with `buffering=0` uses neither [S6].

Here is evidence you can check yourself. Python documents a typical buffer of one hundred and twenty-eight kilobytes [S6]. On one ext4 file, a Linux filesystem, the block size is four thousand and ninety-six bytes, and the C library buffer measured the same [V7]. Two sizes, a factor of thirty-two apart, and neither appears in your source code. That is one measured machine. Do not carry the number to a machine you have not measured.

---

<h2>The bottom level acknowledges early too</h2>

<p class="lede">Level four is the uncomfortable one, and skipping it would make the diagram dishonest.</p>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">The device holds a buffer</div><div class="desc">"most HDDs and many SSDs acknowledge a write once it lands in an onboard DRAM cache" [V20].</div></div>
<div class="row"><div class="num">2</div><div class="term">The standard allows a no-op</div><div class="desc">On <code>fsync</code>: "a null implementation is permitted" [S13].</div></div>
<div class="row"><div class="num">3</div><div class="term">Software cannot check</div><div class="desc">No trace separates a device that flushed from one that only reported it [V5].</div></div>
</div>

<div class="footnotes">
[S13] fsync — The Open Group. https://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html
</div>

Note:
I could stop the diagram at three levels. I am not going to, because the fourth is where data is actually lost, and no tool in this lecture can watch it [V20] [V5].

The device has its own memory. Most spinning disks and many solid state drives report a write as complete when it lands in that memory [V20]. At that moment the bytes are still short of the platter or the flash, and a power cut in that window loses them.

And here is the limit on today's thesis, which I would rather state than have you find. A trace cannot separate a device that flushed from a device that only reported that it flushed [V5]. Everything above this level is observable with a tool you can run. This level is not.

That is why a durability lecture exists, why databases are written the way they are, and why people test storage with a power switch.

---

<h2>Three lines that print nothing</h2>

<p class="lede">A demonstration. You have the stack now, so predict this one.</p>

```python
import os
print("hello")
os._exit(0)
```

<div class="kv">
  <div class="pair"><div class="k">To a terminal</div><div class="v">Does <code>hello</code> appear?</div></div>
  <div class="pair"><div class="k">Redirected to a file</div><div class="v">What does the file contain?</div></div>
</div>

Write down both answers before I run either one [S14].

<div class="footnotes">
[S14] Predict-Observe-Explain — Taber. https://science-education-research.com/teaching-science/constructivist-pedagogy/predict-observe-explain/
</div>

Note:
Three lines. Import, print, exit. Nothing unusual anywhere.

Same method as part one: predict first, out loud. But this time I expect most of you to be right, and that is why this sits here rather than at the beginning.

Two runs. First in a terminal. Then redirected into a file, and I show you the file.

Take thirty seconds. Write two answers. Does hello appear on the terminal, and what is in the file?

Hands up if you think the terminal shows hello. Hands up if you think the file contains hello.

Now watch. Terminal first. And now redirected.

The file is empty. Zero bytes. If you predicted that, you did not guess. You used the diagram from the start of this part, and that is exactly what this demonstration tests.

---

<h2>Why the file is empty</h2>

<p class="lede">The rule sits in the C standard, not in the kernel.</p>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">Mode follows destination</div><div class="desc">Output is "fully buffered if and only if the stream can be determined not to refer to an interactive device" [S5].</div></div>
<div class="row"><div class="num">2</div><div class="term">A terminal is interactive</div><div class="desc">Bytes go out per line.</div></div>
<div class="row"><div class="num">3</div><div class="term">A file is not</div><div class="desc">Bytes wait.</div></div>
<div class="row"><div class="num">4</div><div class="term">The exit skips cleanup</div><div class="desc">No <code>write</code> reaches the kernel.</div></div>
</div>

<div class="footnotes">
[S5] C99 draft §7.19.3. https://busybox.net/~landley/c99-draft.html
</div>

Note:
Here is the explanation, and notice where it comes from [S5].

The rule is in the C language standard. It is not an operating system behaviour and it is not a Python quirk. The standard says that standard output is fully buffered if and only if the stream can be determined not to refer to an interactive device.

Then the third line ends the process without running any cleanup. The standard explicitly permits termination paths that do not close files properly. The buffer is process memory. The process is gone. The bytes went with it.

Now the sentence that connects this back to section two. A trace of this program holds no write call at all. Not a failed one. Not a short one. None. The bytes never got as far as the operating system.

That is the top of the stack, proved to exist by removing it.

---

<h2>What this demonstration does not prove</h2>

<p class="lede">It proves level one is real. It proves nothing about levels three and four.</p>

"the honest classroom demo is a power-cut, not a `kill -9`. Killing the process loses only the *runtime's* buffer; the kernel still writes back." [V14]

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">What died</div><div class="desc">The process, and the level-one buffer with it.</div></div>
<div class="row"><div class="num">2</div><div class="term">What survived</div><div class="desc">Anything already in the page cache [V14].</div></div>
<div class="row"><div class="num">3</div><div class="term">Wrong conclusion</div><div class="desc">"So <span class="neg">flush() was enough</span>." It was enough against this test.</div></div>
</div>

Note:
I have to bound that demonstration immediately, because it is easy to over-read.

Killing a process destroys that process's memory, and that is level one. Bytes the program gave to the operating system earlier sit in the page cache, and the operating system writes them back in its own time [V14]. Your process does not need to exist.

So this proves that level one exists and can lose your data. It proves nothing about levels three and four.

The honest test for the lower levels is a power cut, not a kill [V14]. Pull the plug and you find out what the page cache was holding. Nobody is doing that in this room today.

Why so careful? Because students test with a kill, watch the file survive, and conclude that the program flushed everything it needed to. That conclusion holds against a kill and not against a power failure. Same demonstration, correct conclusion — the difference is which level you claim it tested.

---

<h2><code>ltrace</code>: the same idea, one layer up</h2>

<p class="lede"><code>strace</code> stops at system calls. <code>ltrace</code> stops at calls into shared libraries.</p>

<div class="kv">
  <div class="pair"><div class="k">What it does</div><div class="v">It "intercepts and records the dynamic library calls" a process makes [S31]. It can also print "the system calls executed by the program" [S31].</div></div>
  <div class="pair"><div class="k">Where it comes from</div><div class="v">Juan Cespedes wrote it originally [S32]. The manual page names Petr Machata too [S31]. On Debian and Ubuntu you run <code>apt-get install ltrace</code> [S32].</div></div>
</div>

```text
ltrace ./iodemo       # the C library calls: fopen, fwrite, fclose
ltrace -S ./iodemo    # "system calls as well as library calls" [S31]
ltrace -c ./iodemo    # a count for each library call [S31]
```

<span class="neu">It runs on Linux only</span>.

<div class="footnotes">
[S31] ltrace(1). https://man7.org/linux/man-pages/man1/ltrace.1.html
[S32] ltrace. https://ltrace.org/
</div>

Note:
Here is the second tracer. It makes the next slide checkable.

What it does. Its manual page says it intercepts and records the dynamic library calls the process makes, and that it can also print the system calls underneath [S31]. The page says its use is very similar to `strace` [S31]. So this is the idea you already have, one layer up.

Where it comes from, and how to install it, is on the slide [S31] [S32].

How you use it. With `-S` you get the system calls as well as the library calls, which is the option that matters here [S31]. In the lab you run both tracers on one program and watch the outputs disagree.

---

<h2>Which levels can you actually watch?</h2>

<p class="lede">I promised three of four boundaries. Here is the tool each needs.</p>

| boundary | what watches it | in this course |
|---|---|---|
| App → library | `ltrace`, not `strace` [V10] | above |
| Library → kernel | `strace` attaches here [S1] | today |
| Kernel → device | block-layer tracer [V30] | elsewhere |
| Device → medium | no system-call trace shows whether the device flushed [V5] | none |

Note:
On the first slide I said three of the four boundaries are observable. This is the slide where I show you which three, and why the fourth is not one of them [V5].

There are four boundaries in the diagram. Here is what each costs to see.

The top one needs the tool from the slide before this. My source states the pair directly: printf is visible to ltrace and write to strace [V10]. In our demo program the library call is fwrite instead. That is how you test takeaway two rather than believe it.

The second boundary is the one we have watched all hour.

The fourth is not observable from software at all.

So: three of these four you can watch, one of them with a tool you now know. The one you cannot watch is where data actually gets lost. I come back to that asymmetry in section six.

---

<h1>The same write, at three levels</h1>

<p class="lede">Now that the levels have a reason to exist, walk down them with one small program.</p>

Note:
Eight minutes. In section three I said the levels exist, and why. This section walks down them with a program you can compile yourself.

You write C in this programme, so I am not going to apologise for the C or detour through pointers. We read the core of it. I compile it in front of you. I run it under the tracer. Then we read its trace.

---

<h2>The same write, at three levels</h2>

| level | Python | C | buffered? |
|---|---|---|---|
| Application | `f.write(b"x")` [S6] | — | yes [S6] |
| C library | — | `fwrite(...)` [S3] | yes [S3] |
| System call | `os.write(...)` [S20] | `write(...)` [S2] | no |

The bottom row takes a **file descriptor** — the small integer naming an open file. You saw one already: the `= 3` in the trace line was a file descriptor [S1].

<div class="footnotes">
[S1] strace(1) — Linux man-pages. https://man7.org/linux/man-pages/man1/strace.1.html
[S2] write(2) — Linux man-pages. https://man7.org/linux/man-pages/man2/write.2.html
[S3] fflush(3) — Linux man-pages. [S6] Python documentation, open().
[S20] os — Miscellaneous operating system interfaces, os.write(). https://docs.python.org/3/library/os.html#os.write
</div>

Note:
Read the table by column first, then by row.

The Python column and the C column are two routes to the same bottom row. Python's file objects hold a buffer at level one. C's fwrite fills a buffer at level two. Both call write in the end, and write is the system call.

Now read the last column, because it carries the point. Each level holds a buffer, or it passes the bytes straight down. A Python file object holds one. The os.write function does not, because it wraps the system call directly [S20]. The levels are not decoration. They are the mechanism you measured in part one.

The bottom row introduces a term I deferred until now. A file descriptor is the small integer a Linux program uses to name an open file in system calls. You have seen one: the open call returned three [S1]. It appears as the first argument of every write to that file.

Remember the word. In section five it is the first thing that differs.

---

<h2>Meet the program: <code>iodemo.c</code></h2>

<p class="lede">Nine lines of C. The first open fails on purpose, so the trace shows an error path.</p>

```c
f = fopen("no-such-file.txt", "rb");   /* 1. deliberate failure */
if (f != NULL) { fclose(f); return 2; }

f = fopen(path, "wb");                 /* 2. the real file */
if (f == NULL) { return 1; }

fwrite("one\n",   1, 4, f);            /* 3. three writes */
fwrite("two\n",   1, 4, f);            /*    fourteen bytes */
fwrite("three\n", 1, 6, f);

if (fclose(f) != 0) { return 3; }      /* 4. flush, then close */
```

It prints nothing, on purpose. A program that prints nothing adds no `write` lines of its own to the trace.

Note:
Nine lines on the screen. The full file adds comments and error returns.

Step one opens a file that does not exist, and that failure is deliberate. If every call succeeds, the trace is success all the way down, and the most interesting difference between the two platforms never appears. I show you that difference in section five.

Step two opens the file we mean to write.

Step three writes fourteen bytes in three calls. The C library collects them in its own user-space buffer before any of them crosses into the kernel [S3] [S5]. Watch that number.

Step four closes the file and checks the return value of the close. Almost none of the C I have read does that. I come back to why in section six.

The program prints nothing at all. My reason is the plain one. Every `write` line in the trace then comes from the three `fwrite` calls, and not from a print statement. A silent program also makes a trace that is easier to read.

I am compiling it now, with warnings on, and running it.

---

<h2>Three calls in the source. One in the trace.</h2>

<div class="stats">
  <div><div class="stat">3</div><div class="stat-cap"><code>fwrite</code> calls in the source</div></div>
  <div><div class="stat">14</div><div class="stat-cap">bytes written in total</div></div>
  <div><div class="stat">1</div><div class="stat-cap"><span class="pos">write</span> call in the trace</div></div>
</div>

The three `fwrite` calls go into one C library buffer. The manual calls that level "the user-space buffers provided by the C library" [S3]. The line to look for in the live trace, predicted from the source:

```text
write(3, "one\ntwo\nthree\n", 14) = 14
```

The `3` is the file descriptor. The `14` is all three `fwrite` calls, in one crossing [T2].

<div class="footnotes">
[S3] fflush(3) — Linux man-pages. https://man7.org/linux/man-pages/man3/fflush.3.html
</div>

Note:
This is where takeaway two stops being a claim and becomes something on a screen.

Three fwrite calls in the source. Fourteen bytes. One write in the trace.

Point at the source. Point at the trace. There is no line-by-line correspondence between them, and there never was. What you write is a library call. What the operating system sees is a system call. Your source did not say how many system calls to make — the buffering policy in the library fixed that number.

The line on the slide is a prediction. I wrote it from the source before I ran anything. Now watch the real trace. Check it against the prediction. If they differ, the prediction was wrong and the machine was not.

And notice when the write happens. Not at the third fwrite. It happens when fclose runs, because fclose hands the buffer down to the kernel. The call in your source and the call in the trace are not even at the same point in time.

---

<h2><code>fwrite</code> takes a size and a count</h2>

<p class="lede">Four arguments, and two of them multiply.</p>

```c
fwrite("three\n", 1, 6, f);
/*      data      ^  ^   stream
                  |  +-- number of items
                  +----- size of one item, in bytes  */
```

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">It returns a count</div><div class="desc">Items written, not bytes. The count equals bytes only when the item size is 1 [S21].</div></div>
<div class="row"><div class="num">2</div><div class="term">Fewer is legal</div><div class="desc">Below it, "a successful write() may transfer fewer than count bytes" [S2].</div></div>
</div>

<div class="footnotes">
[S2] write(2) — Linux man-pages. https://man7.org/linux/man-pages/man2/write.2.html
[S21] fread(3), which documents fwrite(3) — Linux man-pages. https://man7.org/linux/man-pages/man3/fwrite.3.html
</div>

Note:
A small thing that trips people. It costs thirty seconds now or twenty minutes in the lab.

The call takes four arguments: a pointer to the data, the size of one item in bytes, the number of items, and the stream. In our program I passed one and six, meaning six items of one byte each. Six and one would have written the same bytes.

Why does it matter? Two reasons.

First, the return value counts items, not bytes. The manual says the count equals the number of bytes only when the item size is one [S21]. Write six items of one byte. You get four back, and you wrote four bytes. Write one item of six bytes. You get zero back, and you wrote nothing.

Second, fewer is legal. The system call underneath may transfer fewer bytes than you asked for [S2], and the library call may report fewer items than you asked for [S21]. Two different contracts, one habit. Read the return value.

---

<h1>Two operating systems, side by side</h1>

<p class="lede">Read as a comparison. Running the tools is the first lab's job.</p>

Note:
Twelve minutes, and some of you waited for this section.

A warning about what this is not. It is not a demonstration that one platform is better. It is not a tutorial on installing a Windows tracer. It is a reading exercise on two pieces of output. The lesson is a structural difference between the systems, not a difference in formatting.

The evidence needs care as well. Each piece of output comes with its origin, and with a statement of what it cannot do.

---

<h2>Process Monitor: files and registry, not system calls</h2>

<p class="lede">It reports filesystem and registry activity [S18], which is a different cut than the system-call boundary.</p>

<div class="kv">
  <div class="pair"><div class="k">What it does</div><div class="v">It shows "real-time file system, Registry and process/thread activity" [S18]. That is a filesystem-and-registry view, not a list of system calls.</div></div>
  <div class="pair"><div class="k">Where it comes from</div><div class="v">Mark Russinovich wrote the page, in Microsoft's Sysinternals documentation [S18]. The tool "combines the features of two legacy Sysinternals utilities, Filemon and Regmon" [S18]. You download it. You can also run it from Sysinternals Live [S18].</div></div>
</div>

```text
Procmon.exe                          # it starts capturing at once
Filter → Process Name is iodemo.exe  # keep one process
Operation column                     # expect CreateFile and WriteFile rows
```

<div class="footnotes">
[S18] Process Monitor. https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
</div>

Note:
Two Windows tools, one after the other.

This is the first. Microsoft describes it as showing real-time file system, Registry and process activity [S18]. That is a filesystem-and-registry view of the machine, and it is not a list of the system calls a program made.

Where it comes from is on the slide [S18]. The same page also links a Procmon for Linux [S18].

How you use it. It captures everything at once, so you filter by process name first. Then you read the Operation column. I expect `CreateFile` and `WriteFile` rows for our program. Check me on the lab machine.

---

<h2>NtTrace: a tracer for the Windows Native API</h2>

<p class="lede">It watches the native API, "the interface between the application space and the OS kernel" [S10].</p>

<div class="kv">
  <div class="pair"><div class="k">What it does</div><div class="v">It "provides a simple trace facility for the Windows Native API", "roughly equivalent to strace on Linux" [S10]. It attaches the way a debugger does, so it "needs no driver and no administrator rights" [V21].</div></div>
  <div class="pair"><div class="k">Where it comes from</div><div class="v">Roger Orr wrote it. The source is MIT-licensed, and the release ships pre-built executables [S10]. You run it directly.</div></div>
</div>

```text
NtTrace iodemo.exe                # one line per native call
NtTrace -filter File iodemo.exe   # keep calls whose names contain "File" [S10]
NtTrace -only iodemo.exe          # not its child processes [S10]
```

<div class="footnotes">
[S10] NtTrace — Roger Orr. https://rogerorr.github.io/NtTrace/
[S33] NtTrace source. https://github.com/rogerorr/NtTrace
</div>

Note:
And here is the second tool, the one we read today.

What it does. Its project page says it provides a simple trace facility for the Windows Native API, and that it is roughly equivalent to strace on Linux [S10]. That comparison is the tool author's own, so you should know whose sentence it is. The same page says the tool uses the Windows debug interface to intercept the returns from the native API [S10]. The consequence, from my reading note: it needs no driver and no administrator rights [V21]. That one choice is why it runs in a teaching lab at all.

Where it comes from. Roger Orr wrote it, and the project page lists releases from 2011 to February 2026 [S10]. The source is MIT-licensed [S33].

How you use it. The page's own example filters on the substring `File`, which is the form we want [S10]. One honest word: this tool is one person's work, and Microsoft calls those entry points "subject to change from one release of Windows to another" [S9].

---

<h2>Why NtTrace and not Process Monitor</h2>

<p class="lede">Two Windows tools, two layers. One answers this lecture's question.</p>

<div class="compare">
<div class="side-a">
<p class="kicker">Process Monitor</p>
<div class="rows">
<div class="row"><div class="num">1</div><div class="desc">Shows "real-time file system, Registry and process/thread activity" [S18]</div></div>
<div class="row"><div class="num">2</div><div class="desc">The usual Windows choice</div></div>
<div class="row"><div class="num">3</div><div class="desc">The wrong layer for us</div></div>
</div>
</div>
<div class="side-b">
<p class="kicker">NtTrace</p>
<div class="rows">
<div class="row"><div class="num">4</div><div class="desc">Reports native API calls, in user mode [S10]</div></div>
<div class="row"><div class="num">5</div><div class="desc">Runs on the debugger interface, so it "needs no driver and no administrator rights" [V21]</div></div>
</div>
</div>
</div>

<div class="footnotes">
[S18] Process Monitor — Sysinternals, Microsoft Learn. https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
</div>

Note:
You have seen both tools. The choice between them is a choice about layers.

Process Monitor reports filesystem events rather than system calls, so it watches the wrong layer here. So we use NtTrace.

Be precise, because I could overstate it. NtTrace is not at the same attachment point as the Linux tracer. The Linux tracer sits at a documented kernel boundary. NtTrace sits one level above the kernel, in a user-mode library. That gap is the lesson I want you to take from this section.

Needing no administrator rights is what makes NtTrace usable here at all. A kernel-mode tracer needs a signed driver and a boot-time change, and nobody installs one in a teaching lab.

Neither of today's tools is what people run in production, and I come back to that at the end.

---

<h2>Three Windows words you will need</h2>

<div class="kv">
<div class="pair">
<div class="k">
Win32
</div>
<div class="v">
The documented interface a Windows program calls. The 32 is history, not a width. Microsoft renamed it the Windows API. The new name "more accurately reflects its roots in 16-bit Windows and its support on 64-bit Windows" [S22].
</div>
</div>
<div class="pair">
<div class="k">
The native API
</div>
<div class="v">
The layer below Win32, inside <code>ntdll.dll</code>. Microsoft calls it "internal to the operating system" [S9].
</div>
</div>
<div class="pair">
<div class="k">
The object manager
</div>
<div class="v">
The kernel component that names open objects. Its paths begin <code>\??\</code> [V22].
</div>
</div>
</div>

<div class="footnotes">
[S9] winternl.h — Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntquerysysteminformation
[S22] Windows API index — Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/apiindex/windows-api-list
</div>

Note:
Three words, thirty seconds, and then the Windows trace makes sense.

Win32 is the documented interface a Windows program calls, and the Windows equivalent of the C library plus the POSIX calls. The name is historical. Microsoft renamed the same thing the Windows API. The new name "more accurately reflects its roots in 16-bit Windows and its support on 64-bit Windows" [S22].

So the 32 in the older name tells you nothing about the width of a program or of a trace. The trace you see next came from a sixty-four bit process [S36], and the name Win32 still applies to it.

The native API is the layer below Win32. It lives in a library called ntdll, and it is the last user-mode code before the kernel. Microsoft's own documentation calls it internal to the operating system.

The object manager is the kernel component that names open things — files, processes, events. Its paths look different from the paths you type. You see one later in this section, in a trace of the Windows command shell [S34].

---

<h2>A real Windows trace</h2>

<p class="lede">You can already read this. The anatomy matches the Linux line, and the vocabulary differs.</p>

```text
NtCreateFile(… ObjectAttributes=0x58:"no-such-file.txt", …)
  => 0xc0000034 [2 'The system cannot find the file specified.']
NtCreateFile(FileHandle=… [0x124], … ObjectAttributes=0x58:"iodemo.out", …) => 0
NtWriteFile(FileHandle=0x124, …, Length=0xe, …) => 0
NtClose(Handle=0x124) => 0
```

`iodemo.exe` under NtTrace, 2026-09-15 — the same four steps as the Linux trace [S36] [S37]. Ignore every argument except the handle, the file name and the return value.

<div class="footnotes">
[S36] Our NtTrace64 capture of iodemo.exe, 2026-09-15 — demo/iodemo-c/NtTrace64-IODEMO-Output.txt
[S37] The container capture of iodemo, 2026-09-15 — demo/iodemo-c/README.md
</div>

Note:
The trace shows this course's own program on Windows. `NtTrace64.exe` captured it on 15 September 2026 [S36]. The Linux trace of the same program comes from the same day [S37].

Read them as the four steps from part four [S36].

Name, arguments in parentheses, return value after the arrow. This anatomy matches the single Linux trace line on the earlier slide.

One thing does differ. Here `=> 0` means success, and a return value starting `0xc` is a failure code [S36].

The three dots mark arguments that this slide omits [S36].

The capture runs to 45 lines [S36]. These are the last four traced calls. Every call above them is the loader's, opening its own files before `main` ran [S36].

The first call wraps over two printed lines, so count calls and not lines. Call one is the deliberate failure, and its return value carries two error codes [S36]. Call two opens `iodemo.out` and returns handle `0x124` [S36]. Call three writes `Length=0xe` bytes, which is fourteen — three `fwrite` calls in one crossing [S36]. Call four closes the handle [S36].

One argument looks unfamiliar. `ObjectAttributes=0x58:"iodemo.out"` means the program named the file relative to an open directory [S36]. So neither of the program's two opens carries the object-manager prefix [S36]. The loader's calls above do carry it [S36]. The example in this deck comes from the shell trace [S34].

---

<h2>A handle is not a file descriptor</h2>

<p class="lede">First difference, visible in the first argument of almost every line.</p>

<div class="boxes">
<div class="box">
<p class="kicker">Linux</p>
A file descriptor. A small integer indexing a per-process table. Our trace shows <code>= 3</code> [S37].
</div>
<div class="box filled">
<p class="kicker">Windows</p>
A handle. An opaque value naming an open object. Our trace shows <code>FileHandle=0x124</code> [S36].
</div>
</div>

Same job in a trace, different mechanism in the system. Do not use one word for the other.

Note:
First difference, and it sits in the first argument of nearly every line on both systems [S36] [S37].

One program, one file, two systems. Linux answered `3`, and Windows answered `0x124` [S36] [S37].

On Linux you get a file descriptor. A small integer indexing a table that belongs to your process. Three, four, five. You can reason about the numbers.

On Windows you get a handle. An opaque value that names an open object, and the object need not be a file. It might be a process, an event or a registry key.

In a trace they occupy the same position and do the same job, which is why it is tempting to use one word for both. Do not. In this course I hold the two words apart. If you write handle in a Linux assignment, I will ask which system you meant.

This is a small thing. I spend a slide on it because it is the cheapest instance of today's platform lesson: the same job, two mechanisms, visible in one argument.

---

<h2>Two error namespaces in one return value</h2>

<p class="lede">Second difference, and it is the layer boundary showing up where nobody expects it.</p>

```text
=> 0xc0000034 [2 'The system cannot find the file specified.']
```

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">First number</div><div class="desc">An <code>NTSTATUS</code>, the native layer's code [S36].</div></div>
<div class="row"><div class="num">2</div><div class="term">Second number</div><div class="desc">The Win32 error it translates into [S36].</div></div>
<div class="row"><div class="num">3</div><div class="term">Linux</div><div class="desc">One <code>errno</code>: <code>-1 ENOENT</code> [S37].</div></div>
</div>

Note:
Second difference, and this is the one I find genuinely interesting.

That return value is our own program's first line, the `fopen` I made fail on purpose [S36].

Look at it. It carries two error codes at once. The first is the native layer's own code. The second is the Win32 error the first translates into, with the message text you would actually see.

Two namespaces, in one return value, from one call.

Linux has one. The same failed open returns `-1` and the name `ENOENT` [S37]. That number is the error, and there is nothing to translate, because only one layer names errors.

Why does Windows have two? Because there genuinely are two layers there, and both are real interfaces with their own error vocabularies. You are looking at the seam between them, visible because the tracer sits below the translation and reports both sides.

This is why the demo program opens a file that does not exist. Without a failure this line never appears, and a clean run hides the seam between the two layers.

---

<h2>The stable boundary sits at a different level</h2>

<p class="lede">Third difference, and the other two are symptoms of it.</p>

In our trace of the Windows command shell a path reads `"\??\C:\WINDOWS\system32\c_852.nls"` [S34]. That prefix "is the object-manager namespace, which a Win32 program never types" [V22].

<div class="matrix">
<div class="m-row"><div class="src">Linux</div><div class="ref">[V8]</div><div class="claim">The system call interface is the public contract.</div><div class="kind">Syscall</div></div>
<div class="m-row"><div class="src">Windows</div><div class="ref">[S9]</div><div class="claim">Native entry points are "subject to change from one release of Windows to another".</div><div class="kind">Win32</div></div>
</div>

<div class="footnotes">
[S9] <span class="src-title">winternl.h</span> — Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntquerysysteminformation
</div>

Note:
Third difference, and the other two are symptoms of this one.

Look at the path on this slide. It comes from the shell's trace [S34]. Our own program's two opens name their file relative to an open directory, so neither of them carries the prefix [S36].

The path starts with a backslash, two question marks, a backslash. No Win32 program ever types that string. It is the name of the file in the object manager's namespace, which sits below the interface a Windows programmer uses. The tracer is showing you a layer programs do not normally see.

Microsoft says so in its own documentation. The native entry points are internal to the operating system and subject to change between releases [S9]. That is a first-party statement.

On Linux the opposite holds. The system call interface is the public contract. You may write a system call number down and rely on it.

Now the careful sentence, and it is the takeaway. By nominate I mean one thing: the vendor documents that layer as the one it will not break. Both systems are stable at the layer they nominate, and the disagreement is about which layer that is. Everything else follows from that single placement, including where a tracer can attach.

---

<h2>Linux is the outlier, not Windows</h2>

<p class="lede">The tempting conclusion is backwards, and the evidence is a list.</p>

| system | what it promises about system calls |
|---|---|
| Linux | a public contract [V9] |
| FreeBSD, OpenBSD, NetBSD, Solaris/illumos, macOS | nothing [V43] |
| Windows | nothing either [V9], and NT's reason is contested [S19] [S15] |

Row two is five systems that publish nothing, not five unstable ones [V36]. Windows and macOS control their own C library. Linux historically did not, so it had to promise the kernel boundary [V38].

<p class="closing">Say the platforms are different [T4]. Never say one is worse.</p>

<div class="footnotes">
[S15] The Microsoft POSIX subsystem — Reiter (2010). https://brianreiter.org/2010/08/24/the-sad-history-of-the-microsoft-posix-subsystem/
[S19] Inside Native Applications — Russinovich (2006). https://learn.microsoft.com/en-us/sysinternals/resources/inside-native-applications
</div>

Note:
Now the correction to the conclusion you were probably forming.

After the last slide it is tempting to think Windows is the odd one out. It is not. My source names the list: FreeBSD, OpenBSD, NetBSD, Solaris/illumos and macOS all behave like Windows here [V43]. None of them publishes a Linux-style promise about the raw system call interface [V9].

Do not read that list as five equally unstable systems [V36]. FreeBSD compiles in compatibility layers, so in practice it holds the binary interface that compiled programs depend on, while declining to promise it formally [V36]. Read the list as "nobody else promises it in the same way", not as "everybody else breaks it".

Row one is Linux, and here is why it is the exception. It was a kernel built against a separately developed GNU userland it did not control [V38]. It could not update the C library in lockstep, so it had to make the kernel boundary itself the stable one [V38].

Row three is the part I will not settle for you. Reading A is a design choice: NT's operating environments included POSIX, OS/2 and Win32, and each one reached NT through the native API [S19]. Reading B is a vestige: one practitioner account says the POSIX subsystem was built for 1980s DoD Orange Book and FIPS specifications [S15]. The same account says it was "never really completed" [S15].

The mechanism is settled. The intent is not. The handout sets out both readings.

In this course you say the platforms are different. You do not say one is worse. If you write that, I will ask which property you are ranking them on.

---

<h2>One program, two kernels</h2>

<p class="lede">One program, two systems, traced on one day.</p>

| step | Linux [S37] | Windows [S36] |
|---|---|---|
| failed open | `openat(… "no-such-file.txt" …) = -1 ENOENT` | `NtCreateFile(… "no-such-file.txt" …) => 0xc0000034 [2 '…']` |
| open | `openat(… "iodemo.out" …) = 3` | `NtCreateFile(… [0x124] … "iodemo.out" …) => 0` |
| write | `write(3, "one\ntwo\nthree\n", 14) = 14` | `NtWriteFile(FileHandle=0x124, … Length=0xe …) => 0` |
| close | `close(3) = 0` | `NtClose(Handle=0x124) => 0` |

Same program, same fourteen bytes, <span class="pos">one write on each side</span>. Hold the program fixed, and what is left is the two systems [T4].

Note:
This is the slide the section was built for.

Read across the rows, not down. Each row is the same step of the same source file [S36] [S37].

Row one is the deliberate failure. The Linux call is `openat`, which is `open` with a starting directory. It returns `-1` and one error name [S37]. Windows returns a native code and the Win32 error it translates into [S36].

Row two is the open, and the two columns are not symmetric. Linux returns the file descriptor `3` as the return value [S37]. Windows returns the status `0` and hands the handle `0x124` back through an argument [S36].

Row three is the write. Fourteen bytes on Linux, `Length=0xe` on Windows [S36] [S37]. Three `fwrite` calls in the source, one call in each trace.

One number to reconcile, from part two. Seven write operations there, one here. The other six were the Linux loader's [S37].

Now the caveat, and it is about the Windows tool rather than about this pair. NtTrace is one author's work [S10]. Microsoft documents those native entry points as subject to change between releases [S9]. So read the Windows column as the shape of the boundary, not as an interface you may depend on.

My own note on the sample I taught from says "It is not a matched pair." [V23] This one is. Both captures ship in this deck, and you rerun them in the first lab.

---

<h1>What we have not looked at</h1>

<p class="lede">One question I will not answer, and the work that follows from it.</p>

Note:
Six minutes left, and I am not going to spend them listing topics.

I will put one question back on the screen. I will not answer it. Then I show you two consequences of the stack. You already have everything you need to see them.

---

<h2><code>write()</code> returned success. The bytes are not on a disk.</h2>

<p class="lede">I built the whole hour towards this sentence. I leave it unresolved.</p>

The manual says it plainly: "A successful return from write() does not make any guarantee that data has been committed to disk" [S2].

Saltzer, Reed and Clark gave the general form in 1984. A lower level may offer "an incomplete version of the function… useful as a performance enhancement" [S11].

<span class="neu">So when is it actually safe?</span> That is the durability lecture's question. I leave it open today.

<div class="footnotes">
[S2] write(2) — Linux man-pages. [S11] End-to-End Arguments in System Design — Saltzer, Reed and Clark (1984). https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf
</div>

Note:
Read the first quote again. It comes from the manual page for the most basic write call there is. It says a successful return guarantees nothing about a disk [S2].

That should now be obvious rather than alarming, because you have the stack. A successful write means the bytes reached the next level down. Which level depends on which call you made.

The second quote is there so you do not think this is a bug someone will fix. Saltzer, Reed and Clark wrote it in 1984. A lower level may offer an incomplete version of a function as a performance improvement. The complete version needs the application at the end points. Every level in our diagram is an instance of that.

So: when is it actually safe? I am not answering today. I want you to leave with the question open, because a programmer who believes it is already answered writes no durability step at all.

---

<h2>Buffering moves your errors</h2>

<p class="lede">The error arrives somewhere other than where you caused it.</p>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">Why it moves</div><div class="desc">The bytes only reached a buffer.</div></div>
<div class="row"><div class="num">2</div><div class="term">Where it arrives</div><div class="desc">Where the library flushes, so at <code>fclose</code>.</div></div>
<div class="row"><div class="num">3</div><div class="term">The rule</div><div class="desc">Check the return value of the call that flushes.</div></div>
<div class="row"><div class="num">4</div><div class="term">The idiom hides it</div><div class="desc">"the most common durability idiom in every mainstream language… contains no durability step anywhere" [V4].</div></div>
</div>

Note:
Here is the practical consequence, and it will cost you a debugging afternoon at some point.

Your fwrite returned success because the bytes reached a buffer. A full disk, an unplugged device, an exceeded quota — no error from any of those has reached the library yet, and none can.

That is why the demo program checks the return value of fclose. Almost none of the C I have read does. That single check is the difference between having written fourteen bytes and believing you did.

The last row is worth saying slowly. The most common way to write a file in every mainstream language contains no durability step anywhere [V4].

Python differs here. A Python with block does raise if the close fails. The two languages differ in what they force you to notice, which is the property I am ranking them on.

This is error reporting, not error recovery. Recovery belongs to the durability lecture.

---

<h2>Never retry a failed close</h2>

<p class="lede">The obvious repair is wrong. Three storage engines dropped the retry in 2018 [V40].</p>

"Retrying the close() after a failure return is the wrong thing to do, since this may cause a reused file descriptor from another thread to be closed" [S4].

The maintainers of three databases reached the same conclusion one level down, for `fsync`: PostgreSQL, MySQL's InnoDB and MongoDB's WiredTiger [V40]. WiredTiger wrote the reason into its own source comment: "we don't retry fsync-style calls and panic if a flush operation fails" [V24]. Not three separate discoveries, though: that comment cites the LWN report on PostgreSQL [V24].

<div class="footnotes">
[S4] close(2) — Linux man-pages. https://man7.org/linux/man-pages/man2/close.2.html
</div>

Note:
Now you check the return value of close. You will want to do the obvious thing with it, and the obvious thing is wrong [S4].

Do not retry. The manual page says why, and the reason is specific. After a failed close the descriptor may already be gone. If another thread has opened a file meanwhile it may have been given that same number. Your retry closes somebody else's file.

That convergence is strong, and it has a limit. The three engines have three different architectures, and their maintainers reached the same answer within months [V41]. All three replaced the retry with a deliberate crash when `fsync` fails [V40]. None of them calls again.

But they did not reach the answer separately. WiredTiger's own comment quotes the LWN report on the PostgreSQL case as its justification [V24]. So read this as three architectures agreeing, not as three independent discoveries.

The transferable rule: when a failure leaves you unable to tell what state you are in, stop the program and report it. Continuing hides the damage and does not repair it.

---

<h2>Networking and IPC are this same mechanism</h2>

<p class="lede">Not new subjects. The same objects and the same calls, wearing different names.</p>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">A socket on Linux</div><div class="desc">It is a file descriptor, and Linux makes that contract binding [V28].</div></div>
<div class="row"><div class="num">2</div><div class="term">A socket on Windows</div><div class="desc">An opaque <code>SOCKET</code>, and Windows makes Winsock the contract [V28].</div></div>
<div class="row"><div class="num">3</div><div class="term">A pipe</div><div class="desc">A file descriptor too, so the same calls reach it [V29].</div></div>
</div>

Depth belongs to the next lecture. I want to stop you filing networking under "something else".

Note:
Thirty seconds, and then I stop naming things.

When you meet networking in this course it will not be a new mechanism. On Linux a socket is a file descriptor. The same read and write calls work on it. Sockets, pipes and terminals are all file descriptors, which is why one program can wait on all of them at once [V29].

On Windows a socket is an opaque type rather than a file descriptor. The binding contract is Winsock, the Windows sockets interface, rather than the file-descriptor contract [V28]. That is section five's pattern showing up again in a place you did not expect it [V8]. Same job, different mechanism.

---

<h2>Honest limits</h2>

<p class="lede">What today's tools cannot see.</p>

<div class="gaps">
<div class="gap"><div class="dot">•</div><div class="gap-term">The device level</div><div class="ref">[V5]</div><div class="gap-desc">Not observable from software: no trace can check it.</div></div>
<div class="gap"><div class="dot">•</div><div class="gap-term">Asynchronous I/O</div><div class="ref">[V11]</div><div class="gap-desc">The tracer sees the setup, then no I/O.</div></div>
</div>

Note:
Here is what I did not establish today.

The device level is not observable from software [V5]. I said it in section three and I repeat it here. It is the limit on today's thesis, and it is where data is actually lost [V5].

Asynchronous I/O breaks the model you have just learned, and it breaks it interestingly. A modern Linux interface moves submission into shared memory rings [V11]. One option starts a kernel thread. That thread polls the rings.

Steady-state submission then makes no system calls at all. The tracer sees the setup and then nothing. The boundary you just learned to watch is already moving.

---

<h2>Honest limits, continued</h2>

<p class="lede">And what today's tools cost.</p>

<div class="gaps">
<div class="gap"><div class="dot">•</div><div class="gap-term"><code>strace</code> in production</div><div class="ref">[V25]</div><div class="gap-desc">Even the cheaper <code>--seccomp-bpf</code> mode [V39] "should therefore not be used on production systems!"</div></div>
<div class="gap"><div class="dot">•</div><div class="gap-term">Mapping, crash dumps</div><div class="ref">—</div><div class="gap-desc">Each has its own lecture.</div></div>
</div>

Note:
And the tools. The tracer has a second and cheaper mode, selected with the option `--seccomp-bpf`. It hands the kernel a filter, so the tracer stops only for the calls you asked about. On the workload of rebuilding strace itself, plain strace costs 3.06 times the untraced run and that mode costs 1.44 times [V39].

A strace contributor disqualifies it anyway. The mode "still stops at each syscall of interest and should therefore not be used on production systems!" [V25]. Production uses a different mechanism, in-kernel programs rather than a stop at every call: eBPF on Linux, and event tracing on Windows.

I have no per-call figure for either, so I am not giving you one. Today's tools watch the layer I am teaching, which is why I use them.

Finally, memory mapping and crash dumps. Each has a lecture.

---

<h2>Assignment 1, and what next week opens on</h2>

<div class="rows">
<div class="row"><div class="num">1</div><div class="term">Assignment 1 measures takeaway 1</div><div class="desc">Two settings, two patterns, one mebibyte.</div></div>
<div class="row"><div class="num">2</div><div class="term">Start from section one</div><div class="desc">Add the random-access variant.</div></div>
<div class="row"><div class="num">3</div><div class="term">Report your buffer size</div><div class="desc">Beside every number.</div></div>
<div class="row"><div class="num">4</div><div class="term">Do not benchmark under the tracer</div><div class="desc">Count traced, time untraced [V17].</div></div>
</div>

<p class="closing">A successful <code>write()</code> is no guarantee that the data has been committed to disk [S2]. The next lecture starts there, on 22 September.</p>

Note:
Two things close the hour: assignment one, and the question for next week.

Assignment one measures takeaway one. Buffered against unbuffered, sequential against random, one mebibyte, and an explanation in Markdown. Four numbers and a paragraph.

The two programs from part one are your starting point. Add the random access variant and you have the whole experiment. You now have the model to write the explanation. I mark that part.

One thing for the Windows half of the room, because I owe you it after part two. The counting tool is Linux-only, and the assignment asks for counts. The course ships a dev container for this. It holds the same tools on a Windows machine and on a Linux one. Lecture one told you to install it. I ask nobody to produce a count their platform cannot produce.

Three rules from today that will earn you marks. First, report your version and buffer size beside every number. The default moved from eight kilobytes [S7] to a typical one hundred and twenty-eight [S6]. Second, count under the tracer [V17]. Time the program without it, in a second run [V17].

Say in your write-up that you did both. Third, your count ratio and time ratio may disagree by orders of magnitude. Explain the gap rather than assuming a mistake. The explanation carries more marks than the numbers do.

The last line is the question for next week. The manual says a successful return from `write()` makes no guarantee that the data has been committed to disk [S2]. Next week asks what does.

---

<p class="kicker">Closing</p>

<h2>Summary — what follows</h2>

<div class="rows">
<div class="row">
<div class="num">
1
</div>
<div class="term">
[T1] Buffer for adjacent writes
</div>
<div class="desc">
A seek hands the saving back, so the access pattern sets the call count.
</div>
</div>
<div class="row">
<div class="num">
2
</div>
<div class="term">
[T2] Ask: accepted, or done?
</div>
<div class="desc">
A successful return reports acceptance, not completion [V12].
</div>
</div>
<div class="row">
<div class="num">
3
</div>
<div class="term">
[T3] Count, do not reason
</div>
<div class="desc">
No level reports which one holds your data, so make the crossings visible [V16].
</div>
</div>
<div class="row">
<div class="num">
4
</div>
<div class="term">
[T4] Say different, never worse
</div>
<div class="desc">
Each system nominates its own level as the stable one [V8].
</div>
</div>
</div>

<p class="closing">Then name the level that holds your bytes, and measure the machine in front of you [S7].</p>

Note:
The same four takeaways that went up at the start, in the form each one takes on Monday.

One is the graded one. Buffer when the writes are adjacent, and expect nothing from a buffer when a seek sits between every pair of writes.

Two, as one question: did that call report acceptance, or completion? Success means accepted, not done [V12]. The same question covers a message broker acknowledging a message.

Three is the method of this course. No level reports which one holds your data [V16]. You have to make the crossings visible and count them.

Four is the platform claim. Say the two systems are different [V8], and if you say more, say which property you are ranking them on.

Then the line at the bottom. Name the level that holds the bytes. The one constant that was safe to quote for sixteen years moved by a factor of sixteen in Python 3.14 [S7].

The handout carries all of this in prose with every source. Questions come now. A terminal is available afterwards for anything that needs one.

---