pdfalto

pdfalto is a command line executable for parsing PDF files and producing structured XML representations of the PDF content in ALTO format, capturing in particular all the layout and style information of the PDF.

pdfalto is initially a fork of pdf2xml, developed at XRCE, with modifications for robustness, addition of features, improved layout element detections, and output enhanced format in ALTO (including in particular space information, useful for instance for further machine learning processing). It is based on the Xpdf library.

Versions are provided in the CHANGELOG.md and in the Github release panel on the right side of the main page of this repository.

An Archlinux package for pdfalto is available here, thanks to @andreasbaumann. The build process described below will create a portable standalone pdfalto executable that can be packaged with other tools without further installation requirements for the end-user.

Requirements

• compilers : clang > 5 or gcc > 7, c++17 required
• makefile generator : cmake >= 3.10.0
• fetching dependencies : wget
• git : the Xpdf source is a git submodule, so the repository must be cloned (not downloaded as a zip) and the submodule initialised with git submodule update --init --recursive — see Build

Usage

General usage is as follows:

Usage: pdfalto [options] <PDF-file> [<xml-file>]
  -f <int>                      : first page to convert
  -l <int>                      : last page to convert
  -verbose                      : display pdf attributes
  -noImage                      : deprecated, use -onlyGraphsCoord instead
  -onlyGraphsCoord              : only extract image coordinates, do not dump image files
  -skipGraphs                   : skip all graphics processing (bitmap and vectorial)  
  -vectorCoordsOnly             : for vector graphics, dump only each path's bounding-box rectangle instead of full curve geometry (smaller .svg, same coordinates)
  -vectorLimit <int>            : max vector paths emitted per page (0 = unlimited); guards against pathological files
  -vectorBoxes                  : emit one bounding box per vector group in the ALTO (instead of a single per-page union box), so vector coordinates can be read without the .svg files
  -outline                      : create an outline file xml
  -annotation                   : create an annotations file xml
  -noLineNumbers                : do not output line numbers added in manuscript-style textual documents
  -readingOrder                 : blocks follow the reading order
  -noText                       : do not extract textual objects (might be useful, but non-valid ALTO)
  -charReadingOrderAttr         : include TYPE attribute to String elements to indicate right-to-left reading order (might be useful, but non-valid ALTO)
  -fullFontName                 : fonts names are not normalized
  -nsURI <string>               : add the specified namespace URI
  -opw <string>                 : owner password (for encrypted files)
  -upw <string>                 : user password (for encrypted files)
  -filesLimit <int>             : limit of asset files be extracted
  -q                            : don't print any messages or errors
  -v                            : print version info
  -h                            : print usage information
  -help                         : print usage information
  --help                        : print usage information
  -?                            : print usage information

In addition to the ALTO file describing the PDF content, the following files are generated:

• _metadata.xml file containing a pdf file metadata (generate metadata information in a separate XML file as ALTO schema does not support that).

• _annot.xml file containing a description of the annotations in the PDF (e.g. GOTO, external http links, ...) obtained with -annotation option

• _outline.xml file containing a possible PDF-embedded table of content (aka outline) obtained with -outline option

• .xml_data/ subdirectory containing the vectorial (.svg) and bitmap images (.png) embedded in the PDF, this is generated by default. This extraction slows down the process very significantly, so if no image files are required, use -onlyGraphsCoord (or the deprecated alias -noImage) to keep extracting image coordinates without dumping the image files. To skip all graphics processing (bitmap and vectorial), use -skipGraphs.

Runtime resources

pdfalto reads xpdfrc and the languages/ tree at startup; the paths inside xpdfrc are resolved relative to the directory holding it. It is looked for in this order:

1. $PDFALTO_DATA_DIR, if set.
2. Beside the executable — how the release archives and a local cmake ./ && make build are laid out.
3. ../share/pdfalto relative to the executable, so pdfalto can be installed the way any Unix program is, with the binary in bin/ and its data under share/. This is what the Python wheel does.

None of them being present is not an error: xpdf falls back to its built-in defaults, and only documents needing the non-Latin encoding tables are affected.

Memory footprint on large or pathological PDFs

Some PDFs (very long documents, or figures built from millions of vector paths) can push peak memory well beyond several GB. Peak memory has three independent causes, each with its own lever:

• Text/DOM accumulation (cost grows with the number of text elements): bounded by the per-page DOM streaming that is always on — peak no longer grows with page count.
• Bitmap decoding (width × height × 3 bytes per image): avoided with -onlyGraphsCoord, which emits image coordinates without decoding/dumping the pixels.
• Vector graphics explosion (one libxml node per path; pathological figures can hold millions): when the .svg file is not going to be written (-onlyGraphsCoord / -noImage), the full vector geometry is no longer built in memory at all — only the per-page bounding box that downstream consumers read is kept.

For batch/server processing (e.g. GROBID), -onlyGraphsCoord bounds all three causes at once while still emitting every coordinate downstream tools consume, and is the recommended invocation.

If you do need the vector .svg files written but want them small and bounded, combine -vectorCoordsOnly (reduces each path to its bounding-box rectangle — the same box a consumer derives from the full geometry) with -vectorLimit <N> (caps the number of vector paths emitted per page; the per-page bounding box is still emitted).

Note: internal per-page buffers are written under $TMPDIR. Point $TMPDIR at real on-disk storage (not a tmpfs/RAM disk), otherwise the streaming buffers count against RAM and defeat the memory bound.

Reading vector coordinates without the .svg files

By default the ALTO carries a single <Illustration TYPE="svg"> per page, whose box is the union of every vector path drawn on that page — so a consumer that wants individual figure regions has to parse the .svg. With -vectorBoxes, pdfalto instead emits one <Illustration TYPE="svg"> per vector group, each carrying that group's own bounding box, and writes no .svg at all:

<Illustration ID="p1_s1" HPOS="168.548" VPOS="245.197" WIDTH="406.431" HEIGHT="2.268" ROTATION="0.000000" TYPE="svg"/>

Each box is clipped to the page rectangle. Fully transparent groups render nothing and are omitted, so what appears in the ALTO is the visible vector content; partially transparent groups are visible and are kept. Pages with more than 5000 vector groups (overridable with -vectorLimit <N>) fall back to a single union box for that page, so both memory and ALTO size stay bounded on pathological files.

Extra script to get only text content

The goal of pdfalto is to extract all the content of a PDF, not just text, but also layout, style, font, vector graphics, embedded bitmap, annotation, metadata, and outline information. For convenience and debugging, we provide a simple XSLT to extract only the text content from the produced ALTO XML file. For instance, using xsltproc command line, the following outputs the text content only:

xsltproc schema/alto2txt.xsl alto_file.xml

Python

pdfalto is published on PyPI as pdfalto. The wheels bundle the compiled executable, so there is nothing else to install:

pip install pdfalto

import pdfalto

result = pdfalto.convert("paper.pdf", "paper.xml", outline=True)
print(result.alto, result.metadata, result.outline, result.data_dir)

alto_xml = pdfalto.convert_to_string("paper.pdf")   # no files left behind

Every command line option is a keyword argument of convert(). Installing the package also puts the pdfalto executable itself on PATH — the real binary, not a Python wrapper, so it starts just as fast:

pdfalto -outline paper.pdf paper.xml

See python/README.md for the full API, the list of platforms with wheels, and the differences between the wheel binaries and the ones on the releases page.

Dependencies

Nothing to do here for a normal build. The static libraries pdfalto links against (libxml2, freetype, libpng, zlib, ICU) are committed to this repository under libs/, prebuilt for each supported platform (linux/64, linux/arm64, mac/64, mac/arm64, and windows). The xpdf language support packages are likewise committed under languages/. Cloning the repository is enough — go straight to Build.

These libraries are produced by the ci-build-libs.yml workflow, which builds every platform in a matrix and is run manually (workflow_dispatch) when a dependency needs updating. The library versions it pins are declared at the top of that file. Rebuilding them is a maintainer task, not a prerequisite for building pdfalto.

If you do need to rebuild them locally — porting to a new platform, or bisecting a dependency — ./install_deps.sh downloads and builds them into libs/ and languages/; see compiling dependencies procedures for details.

Known issues

(issue 41) might occur while building, in this case you'll need to compile the dependencies before building pdfalto.

Build

• NOTE for windows : it's recommended to use Cygwin and install standard libraries (either for cland or gcc)

git clone https://github.com/kermitt2/pdfalto.git && cd pdfalto

• Xpdf-4.05 is shipped as git submodule, to download it:

git submodule update --init --recursive

• Build pdfalto:

cmake .

make

The executable pdfalto is generated in the root directory. Additionally, this will create a static library for xpdf-4.05 at the following path xpdf-4.05/build/xpdf/lib/libxpdf.a and all the libraries and their respective subdirectory.

To use the additional xpdf language support packages, the executable pdfalto comes with a config file xpdfrc and language resources installed under languages/. Both xpdfrc and languages/ must be alongside the executable pdfalto to be used. To add pdfalto with these additional resources to a third party application (e.g. GROBID), move the executation together with these files:

lopez@work:~$ ls my_pdfalto/
languages  pdfalto  xpdfrc

Known issues

(issue #135) on macOS "fontconfig.h file not found" might occur while building, see described workaround.

Future work

• Text like containing block element characters (https://unicode.org/charts/PDF/U2B00.pdf) are used as placeholders for unknown character unicodes, instead of what would be expected when visually inspecting the text. The reason for these unsolved character unicode values is that the actual characters are glyphs that are embedded in the PDF document which use free unicode range for embedded fonts, not the right unicode. The only way to extract the valid text for those special characters is to use OCR at glyph level . This is our targeted main future enhancement, relying on a custom Deep Learning approach.

• map special characters in secondary fonts to their expected unicode

• reduce the memory still retained per page in the text extraction path: peak memory no longer grows with the number of pages, but some per-page allocations are released later than they could be, so very long documents (tens of thousands of pages) still hold more than they need to

• see the issue tracker for further tasks

Changes

All changes are in the CHANGELOG.md

Release

To release pdfalto you need bump-my-version.

Create and activate a virtual environment and install the tool:

python3 -m venv venv
source venv/bin/activate
pip install bump-my-version

Then you can run show-bump to see the plan of updating

bump-my-version show-bump
0.6.0 ── bump ─┬─ major ─ 1.0.0
               ├─ minor ─ 0.7.0
               ╰─ patch ─ 0.6.1

and make the new release by:

bump-my-version bump patch|minor|major

and git push --tags

Pushing the tag runs both release workflows: ci-build.yml builds the binaries and publishes the GitHub Release, and ci-python.yml builds the wheels and the source distribution and uploads them to PyPI. The Python package takes its version from project(pdfalto VERSION ...) in CMakeLists.txt, which bump-my-version already updates, so there is nothing extra to bump.

PyPI upload uses trusted publishing rather than a stored API token. It has to be configured once, as a pending publisher while the project does not exist yet, with owner kermitt2, repository pdfalto, workflow ci-python.yml and environment pypi.

Rehearsing a release on TestPyPI

Before tagging, the whole publishing path can be exercised against TestPyPI: run the Python wheels workflow from the Actions tab with Upload the result to TestPyPI ticked and a pre-release version such as 0.6.3.dev1. It builds the same wheels and sdist and uploads them through the same trusted-publishing path, so an index that rejects the metadata or the wheel tags says so before the tag exists rather than after.

The version must be given, and must be one TestPyPI has never seen: an index refuses a version it already holds, and does not release it even if the file is deleted. Bump the suffix (.dev2, .dev3, ...) on each attempt. It is only used for the rehearsal — a tag push always takes its version from CMakeLists.txt.

TestPyPI needs its own pending publisher, configured at test.pypi.org/manage/account/publishing with the same owner, repository and workflow but environment testpypi.

Installing the result needs the real index for the dependencies, since TestPyPI does not mirror them:

pip install --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple/ \
            pdfalto==0.6.3.dev1

Contributors

Contact: Patrice Lopez (patrice.lopez@science-miner.com)

pdfalto is developed by Patrice Lopez (patrice.lopez@science-miner.com) and Achraf Azhar (achraf.azhar@inria.fr).

pdf2xml is orignally written by Hervé Déjean, Sophie Andrieu, Jean-Yves Vion-Dury and Emmanuel Giguet (XRCE) under GPL2 license.

Xpdf is developed by Glyph & Cog, LLC (1996-2017) and distributed under GPL2 or GPL3 license.

The windows version has been built originally by @pboumenot and ported on windows 7 for 64 bit, then for windows (native and cygwin) by @lfoppiano and @flydutch.

License

As the original pdf2xml and main dependency Xpdf, pdfalto is distributed under GPL2 license.

Useful links

Some tools for converting ALTO into other formats:

• https://github.com/filak/hOCR-to-ALTO
• https://github.com/UB-Mannheim/ocr-fileformat