> ## Documentation Index
> Fetch the complete documentation index at: https://cstreams.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Processing

## Overview

These are the main pipeline settings. Experiment to find the best values for your use
case.

### `MATTER_CONFIG`

The core pipeline object which determines which matter pages to process.
[What are matter pages?](/key-concepts/matter-pages)

#### Default configuration

```json theme={null}
MATTER_CONFIG = {
    "front": {
        "max_pages": 8
    },
    "back": {
        "mode": "never",
        "max_pages": 5,
        "fields": {
            "publisher": false,
            "year": false,
            "edition": false,
            "isbn": false,
            "doi": false,
            "loc": false
        }
    }
}
```

<Note>
  Why are title and author not in the fields section?

  Title and at least one author are always required and are implicitly included.
  In other words, we need some baseline fields to name a file.

  This will be made configurable in the next update.
</Note>

### Front Matter

<ResponseField name="front" type="object" required>
  <Expandable title="Properties" defaultOpen>
    <ResponseField name="max_pages" type="number" default="8" required>
      Number of pages to process from the start of document. Begins counting from{' '}
      <a href="/config/processing#param-page-num-offset" class="param-field-link">PAGE\_NUM\_OFFSET</a>
    </ResponseField>
  </Expandable>
</ResponseField>

#### Front Matter Examples

<CodeGroup>
  ```python Default theme={null}
  MATTER_CONFIG = {
      "front": {
          # Safe default for most documents
          "max_pages": 8,
      }
  }
  ```

  ```python Minimal theme={null}
  # Search the first 3 pages for title and at least one author.
  # Any other fields found are picked up but not required.
  MATTER_CONFIG = {
      "front": {
          # Usually best for novels, short stories, research papers
          "max_pages": 3,
      }
  }
  ```

  ```python Extended theme={null}
  # Search the first 15 pages for title and at least one author.
  # Any other fields found are picked up but not required.
  MATTER_CONFIG = {
      "front": {
          # Best for non-fiction books, like academic texts, textbooks
          "max_pages": 15,
      }
  }
  ```

  ```python Entire document theme={null}
  # Search the entire document for title and at least one author.
  # Any other fields found are picked up but not required.

  # A better way to accomplish this is planned.
  MATTER_CONFIG = {
      "front": {
          # Safe value, will stop processing at the last page
          "max_pages": 10000,
      }
  }

  ```
</CodeGroup>

### Back matter

Back matter generally has less metadata we need and is designed primarily as a fallback if
the title and at least one author name was not found in the front matter.

There are certain situations where you may want to process back matter differently.

<ResponseField name="back" type="object" required default={{ mode: "never", max_pages: 5, fields: {} }}>
  <Expandable title="Properties" defaultOpen>
    <ResponseField name="mode" type="string" default="never">
      When to process back matter pages:

      * `never`: Skip back matter entirely, regardless of what's found in front matter.
        Fastest and most cost-effective option. Best for fiction and short stories.

      * `always`:
        Process back matter regardless of what fields are found in front matter.
        Best for documents where back matter contains detailed appendices,
        like reports, technical papers, and research papers.

      * `fallback`:
        Only process back matter if front matter doesn't have a title, at least one
        author name, and all of the fields set to `true`
        in `MATTER_CONFIG.back.fields`.

        Best balance of speed and completeness. Will check back matter only if
        key metadata is missing from front matter. This is the recommended setting
        for most document collections with mixed formats and structures.
    </ResponseField>

    <ResponseField name="max_pages" type="number" default="5" required>
      Number of last pages of the document to process.
    </ResponseField>

    <ResponseField name="fields" type="object" required>
      Which metadata fields to look for if not found in front matter,
      and back matter processing is enabled with `MATTER_CONFIG.back.mode`
      set to `fallback` or `always`.

      These settings are ignored if `MATTER_CONFIG.back.mode` is `never`.

      Author and Title are the minimum required fields for metadata validation
      and filename generation and are not configurable. In other words,
      the filename must have something to name the document.

      This API shape is a work in progress and a bit unclear.
      Will refactor to be more intuitive in a future update.

      Look at the examples below for clarification.

      <Expandable title="Properties" defaultOpen>
        <ResponseField name="publisher" type="boolean" default="false" required />

        <ResponseField name="year" type="boolean" default="false" required>
          Publication year
        </ResponseField>

        <ResponseField name="edition" type="boolean" default="false" required />

        <ResponseField name="isbn" type="boolean" default="false" required>
          ISBN identifier
        </ResponseField>

        <ResponseField name="doi" type="boolean" default="false" required>
          Digital Object Identifier
        </ResponseField>

        <ResponseField name="loc" type="boolean" default="false" required>
          Library of Congress number
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

#### Back Matter Examples

<CodeGroup>
  ```python Never theme={null}
  # Never process back matter, regardless of what's found in front matter.
  # Note that even though ISBN is set to `true`, it won't trigger a
  # back matter search because back matter processing is disabled.
  MATTER_CONFIG = {
      "back": {
          "mode": "never",
          "max_pages": 5, # no effect

          # no effect
          "fields": {
              "publisher": false,
              "year": false,
              "edition": false,
              "isbn": true,
              "doi": false,
              "loc": false
          }
      }
  }
  ```

  ```python No DOI theme={null}
  # If every field we're looking for (title, author, publisher, year, edition,
  # ISBN, and LOC) are found in the front matter, don't go looking for the DOI
  # in the back matter
  MATTER_CONFIG = {
      "back": {
          "mode": "fallback",
          "max_pages": 5,
          "fields": {
              "publisher": true,
              "year": true,
              "edition": true,
              "isbn": true,
              "doi": false,
              "loc": true
          }
      }
  }
  ```

  ```python Only Edition theme={null}
  # If we can't find at least one edition in the front matter,
  # process the last 20 pages for it.
  MATTER_CONFIG = {
      "back": {
          "mode": "fallback",
          "max_pages": 20,
          "fields": {
              "publisher": false,
              "year": false,
              "edition": true,
              "isbn": false,
              "doi": false,
              "loc": false
          }
      }
  }
  ```

  ```python Always theme={null}
  MATTER_CONFIG = {
      "back": {
          "mode": "always",
          "max_pages": 5,
          "fields": {
              "author": true,
              "title": true,
              "publisher": true,
              "year": true,
              "edition": true,
              "isbn": true
          }
      }
  }
  ```
</CodeGroup>

### Counting Pages

<ParamField query="PAGE_NUM_OFFSET" type="number" default="1" required>
  Determines how to express page numbering when naming diagnostic files. 1-based makes
  it easier to cross-reference page numbers to the source PDF page numbers.
</ParamField>

<CodeGroup>
  ```python 1-based theme={null}
  PAGE_NUM_OFFSET = 1
  MATTER_CONFIG = {
      "front": {
          "max_pages": 8, 
      }
  }
  Pages will be numbered 1-8. Front cover is page 1.
  ```

  ```python 0-based theme={null}
  PAGE_NUM_OFFSET = 0
  MATTER_CONFIG = {
      "front": {
          "max_pages": 8,
      }
  }
  Pages will be numbered 0-7. Front cover is page 0.
  ```
</CodeGroup>

### Writing Changes

Critical setting that controls whether generated metadata and filenames are written to
PDFs in `[project-root]/data/` which are not in a
<a href="/config/diagnostic#param-runtime-ignore-dir-name" class="param-field-link">RUNTIME\_IGNORE\_DIR\_NAME</a>.

Only set this to `True` once you have run the script and confirmed suggestions are
acceptable. Review [Evaluation](/analysis-and-iteration/evaluation) for a full guide.

<Warning>
  Back up your files before setting this to True. Many edge cases may cause unexpected
  results. See [Test Cases](/analysis-and-iteration/test-cases) for cases we test for,
  and [Known Issues](/analysis-and-iteration/known-issues) for those we know about.
</Warning>

<ParamField query="WRITE_PDF_CHANGES" type="boolean" default="false" required />

Cap the filename character length for cross-platform compatibility. This has only been
tested on a macOS APFS volume. We reserve five characters so the annotation sidecar suffix
`--ann` fits for both the markdown and JSON exports.

<ParamField query="MAX_FILENAME_LENGTH" type="number" default="250" required />

### Data folder

Drop your PDFs in the `[project-root]/data` folder. Read about
[ignoring directories](/config/diagnostic#ignoring-directories), a helpful feature for
troubleshooting.

<Note>
  The `[project-root]/data` folder location is currently hardcoded. While this makes it
  easy to ignore with git, it could be made configurable.

  This is planned for a future update. For now, you can create a symlink from your preferred
  location to `[project-root]/data`.
</Note>