← All posts

Strip the repeated header from extracted text

You extracted the text of a 300-page report and got the text of a 300-page report, plus ANNUAL REPORT 2025 — CONFIDENTIAL three hundred times and Page 47 of 312 three hundred times, and half of those have landed in the middle of a sentence.

So you searched for how to remove headers and footers from extracted text, and every result explained how to edit the header out of the PDF itself — a different job, on a different file, in a tool you probably do not have. The one page that was actually on the subject was a discussion thread in a Python library's issue tracker.

There is no button for this, here or anywhere, and there is a reason for that. But there is a reliable method, and the first half of it is deciding how to extract the text in the first place.

Why the header is in your text at all

A PDF has no concept of a running title. When Word or InDesign or a report generator writes a header, it does not record "this document has a header"; it writes the words onto the page at coordinates, once per page, exactly as it writes the body. Nothing in the file distinguishes them afterwards. The table extraction problem is the same one wearing different clothes: the structure you can see was never stored.

There is one exception, and it is worth knowing so you can stop hoping for it. A properly tagged PDF marks page furniture as an artifact — a bracket in the content stream saying "this is pagination, not content" — and a tagged export from Word usually does this correctly. Almost no text extractor reads it. Ours does not, poppler's pdftotext does not, and a plain copy-and-paste in Acrobat does not. They walk the content stream and hand you every glyph they find, artifact or not. Our PDF inspector will tell you whether the document is tagged at all, along with the page count you will need in a minute, but even a "yes" does not get the header out of your text file.

Why it lands in the middle of a sentence

This is the part that makes the problem feel worse than it is, and nobody explains it.

Unless you ask otherwise, an extractor gives you the text in the order the glyphs were written into the file, which is whatever order the producing program happened to emit them. Plenty of producers draw the body first and stamp the header and footer on last. When that happens, the extracted page reads: body, body, body, Page 47 of 312, ANNUAL REPORT 2025 — CONFIDENTIAL — and because the next page's body follows immediately, the furniture ends up wedged between the end of one paragraph and the start of its continuation.

Ask instead for the text in the order it sits on the paper, and the same page comes out with the header as the first line and the footer as the last, every time. That single change turns a scattering problem into a trimming problem.

Extract it in a shape you can clean

Run PDF to text with two settings deliberately set:

  1. Keep the layout — on. This reads by position rather than by writing order. The header becomes the first line of every page and the footer the last. See the warning below before you do this on a two-column document.
  2. Mark where each page ends — on. You get a form feed, U+000C, between pages and nowhere else. Every editor, every language and every command-line tool already knows what that character is, and it means text.split("\f") gives you exactly one block per page. Without it you get blank lines, which are indistinguishable from the blank lines inside the document.
  3. Set the page range if you only want part of the file. Reading pages 40–60 leaves you twenty headers to deal with rather than three hundred.

Then read the note it gives back. "Four pages have no text at all" means those four pages are pictures — a scan stapled into a born-digital report — and no cleanup pass will help, because nothing was extracted from them to clean.

Pass 1: find the furniture by counting it

Do not start by writing patterns. Start by asking the text which of its lines repeat, because a line that appears on nearly every page is page furniture by definition.

The trick that makes this work is normalising the numbers before you count. Page 47 of 312 and Page 48 of 312 are two different strings and one piece of furniture, so replace every run of digits with a # first and they collapse into one.

import re, collections

pages = open("report.txt", encoding="utf-8").read().split("\f")
shape = lambda s: re.sub(r"\d+", "#", s.strip())

counts = collections.Counter(shape(l) for p in pages for l in p.splitlines() if l.strip())
for line, n in counts.most_common(20):
    print(f"{n:4d}/{len(pages)}  {line}")

Twenty lines of output and you know what you are dealing with. Anything on more than about 60% of the pages is furniture; anything appearing three times is a sentence somebody reused. Then the deletion is two more lines:

furniture = {s for s, n in counts.items() if n > 0.6 * len(pages)}
clean = "\n".join(l for p in pages for l in p.splitlines() if shape(l) not in furniture)

The threshold matters more than it looks. Chapter opening pages usually suppress the running head, and front matter is numbered differently or not at all, so a header that is genuinely on every body page appears on perhaps 85% of the file. A test of "on every page" finds nothing.

If you would rather not run code: paste the text into a spreadsheet one line per row, add a column with =COUNTIF(A:A,A1), sort by it descending, and the furniture is sitting at the top.

Pass 2: the patterns worth deleting by hand

Counting finds repeated lines. It misses furniture that varies — a date stamp, a reference with a section number in it, a header naming the current chapter. Those need patterns, and there are only four worth having:

Delete to a copy, keep the raw extraction, and count what you removed. Three hundred deletions from a 312-page file is right. Nine hundred is a pattern that is eating your text.

Pass 3: rejoin what the furniture broke

Removing the lines leaves the damage they did. Two fixes cover nearly all of it.

Hyphenated words split across a line break. Typeset text breaks words at the margin, and the extraction preserves the hyphen and the newline:

clean = re.sub(r"(\w)-\n(\w)", r"\1\2", clean)

Run it before you rejoin the paragraphs, and be aware that it will also close up a genuine compound that happened to break at a hyphen — well-\nknown becomes wellknown. If that matters, check the list of joins before applying them.

Sentences broken by the page boundary. Where a block ends without a full stop and the next begins with a lower-case letter, it is one sentence:

clean = re.sub(r"([^.!?:\n])\n(?=[a-z])", r"\1 ", clean)

Do this last. If you do it before removing the furniture, you will glue the header to the sentence instead of to itself.

The trap: cropping the page does not crop the text

The advice you will be offered, if you are offered any, is to crop the margins off first. It does not work, and it is worth knowing why before you spend an hour on it.

Cropping a PDF moves the crop box — the window a reader displays and a printer prints. It deletes nothing. Our Crop PDF says so on the page, and it is the right behaviour: it means cropping is instant on a 500-page file and reversible. But the header is still in the content stream, at the same coordinates it always was, and a text extractor walks the content stream. We checked: crop 30 mm off the top and bottom of a page and both PDFBox and pdftotext still hand you the header and the footer. Crop when you want the printed page changed. Not for this.

What does work is cropping inside the extractor, which is a different operation — it discards glyphs whose coordinates fall outside a region. Poppler's pdftotext takes a region directly, in points measured from the top left of the page as you see it:

pdftotext -x 0 -y 60 -W 595 -H 660 report.pdf clean.txt

That is A4 with the top 60 points and the bottom 122 points thrown away, and it removes the header and footer before they ever reach your text file. Add -layout for the positional read, and -f 40 -l 60 for a page range. If the header sits at a different height on left- and right-hand pages, run it twice with different regions and interleave the output.

The same idea in Python is pdfplumber, where page.crop((0, 60, 595, 782)).extract_text() does it per page — which is what that lone useful discussion thread was about, and the right answer if this is a job you will do every month rather than once.

When it is a table you are after, this solves itself

If the reason you extracted the text was to get figures into a spreadsheet, do not clean prose at all. PDF to Excel reads the page as rows and cells and then throws away every row with fewer cells than the minimum you set — which is two by default. A running header is one cell. A page number is one cell. Both are dropped before they reach the CSV, and the result tells you how many rows were dropped so you can see it happened. Raise the minimum to three and stray one- and two-cell lines go too. The full route into a spreadsheet has the rest of it.

What we cannot do for you

There is no "strip headers" option on our text tool, and adding one would be a lie. An extractor cannot tell a running head from a heading. Both are short, both sit at the top of a page, both are title-cased; Notes to the financial statements is a running head on forty pages and a real section heading on one of them. The only signal that distinguishes them is frequency across the whole document, which is why the counting pass above is the honest method and a checkbox would not be.

Footnotes are a separate problem and worse. They sit at the foot of the page, so a positional trim removes them — which is right if you want the prose and wrong if you want the citations. They also leave their reference numbers glued to words in the body text, and a bare superscript 12 is indistinguishable from a real number once the type size is gone. If footnotes matter, extract them deliberately with a region of their own rather than cleaning them out by accident.

Nothing here recovers what was never extracted. A scanned page contributes no text and therefore no header either, which can be confusing when the frequency count says a header appears on 240 of 312 pages and you cannot see why the other 72 are missing. Check what has a text layer before you conclude your pattern is wrong.

And if you only ever wanted one section of the document, all of this shrinks: pull the pages out first and extract from the extract. Getting the text of just pages 10 to 20 is a two-step job, and twenty headers can be deleted by hand faster than you can write a regular expression for them.

Tools this post uses: