Processing

Overview

These are the main pipeline settings. Experiment to find the best values for your use case.

MATTER_CONFIG

The core pipeline object which determines which matter pages to process. What are matter pages?

Default configuration

MATTER_CONFIG = {
    "front": {
        "max_pages": 8
    },
    "back": {
        "mode": "never",
        "max_pages": 5,
        "fields": {
            "publisher": false,
            "year": false,
            "edition": false,
            "isbn": false,
            "doi": false,
            "loc": false
        }
    }
}

Why are title and author not in the fields section?

Title and at least one author are always required and are implicitly included. In other words, we need some baseline fields to name a file.

This will be made configurable in the next update.

Front Matter

front · object · required

Hide Properties

max_pages · number · default: "8" · required

Number of pages to process from the start of document. Begins counting from PAGE_NUM_OFFSET

Front Matter Examples

Default

MATTER_CONFIG = {
    "front": {
        # Safe default for most documents
        "max_pages": 8,
    }
}

Back matter

Back matter generally has less metadata we need and is designed primarily as a fallback if the title and at least one author name was not found in the front matter.

There are certain situations where you may want to process back matter differently.

back · object · required

Hide Properties

mode · string · default: "never"

When to process back matter pages:

• never: Skip back matter entirely, regardless of what’s found in front matter. Fastest and most cost-effective option. Best for fiction and short stories.

• always: Process back matter regardless of what fields are found in front matter. Best for documents where back matter contains detailed appendices, like reports, technical papers, and research papers.

• fallback: Only process back matter if front matter doesn’t have a title, at least one author name, and all of the fields set to true in MATTER_CONFIG.back.fields.

  Best balance of speed and completeness. Will check back matter only if key metadata is missing from front matter. This is the recommended setting for most document collections with mixed formats and structures.

max_pages · number · default: "5" · required

Number of last pages of the document to process.

fields · object · required

Which metadata fields to look for if not found in front matter, and back matter processing is enabled with MATTER_CONFIG.back.mode set to fallback or always.

These settings are ignored if MATTER_CONFIG.back.mode is never.

Author and Title are the minimum required fields for metadata validation and filename generation and are not configurable. In other words, the filename must have something to name the document.

This API shape is a work in progress and a bit unclear. Will refactor to be more intuitive in a future update.

Look at the examples below for clarification.

Hide Properties

publisher · boolean · default: "false" · required

year · boolean · default: "false" · required

Publication year

edition · boolean · default: "false" · required

isbn · boolean · default: "false" · required

ISBN identifier

doi · boolean · default: "false" · required

Digital Object Identifier

loc · boolean · default: "false" · required

Library of Congress number

Back Matter Examples

Never

# Never process back matter, regardless of what's found in front matter.
# Note that even though ISBN is set to `true`, it won't trigger a
# back matter search because back matter processing is disabled.
MATTER_CONFIG = {
    "back": {
        "mode": "never",
        "max_pages": 5, # no effect

        # no effect
        "fields": {
            "publisher": false,
            "year": false,
            "edition": false,
            "isbn": true,
            "doi": false,
            "loc": false
        }
    }
}

Counting Pages

PAGE_NUM_OFFSET · number · default: "1" · required

Determines how to express page numbering when naming diagnostic files. 1-based makes it easier to cross-reference page numbers to the source PDF page numbers.

1-based

PAGE_NUM_OFFSET = 1
MATTER_CONFIG = {
    "front": {
        "max_pages": 8, 
    }
}
Pages will be numbered 1-8. Front cover is page 1.

Writing Changes

Critical setting that controls whether generated metadata and filenames are written to PDFs in [project-root]/data/ which are not in a RUNTIME_IGNORE_DIR_NAME.

Only set this to True once you have run the script and confirmed suggestions are acceptable. Review Evaluation for a full guide.

Back up your files before setting this to True. Many edge cases may cause unexpected results. See Test Cases for cases we test for, and Known Issues for those we know about.

WRITE_PDF_CHANGES · boolean · default: "false" · required

Cap the filename character length for cross-platform compatibility. This has only been tested on a macOS APFS volume.

MAX_FILENAME_LENGTH · number · default: "255" · required

Data folder

Drop your PDFs in the [project-root]/data folder. Read about ignoring directories, a helpful feature for troubleshooting.

The [project-root]/data folder location is currently hardcoded. While this makes it easy to ignore with git, it could be made configurable.

This is planned for a future update. For now, you can create a symlink from your preferred location to [project-root]/data.