What Is a TDM Exception? – Definition and How It Works

Learn what text and data mining exceptions permit, how the machine-readable opt-out works, and where it applies.

A text and data mining exception is a copyright provision permitting automated analysis of protected works without the rightsholder's permission, subject to conditions. It is the legal mechanism most directly relevant to lawful large-scale collection in the EU. This page is a general explanation, not legal advice.

1. The Problem It Solves

Analysing a corpus of text requires copying it — downloading, storing, and processing. Copying is a restricted act under copyright, so without an exception, large-scale analysis of protected material would need permission from every rightsholder.

TDM exceptions permit that copying for analytical purposes.

2. The Two EU Provisions

The EU's Digital Single Market directive created two distinct exceptions, and the difference between them is the whole practical story:

The research exception is robust — a rightsholder cannot opt out of it. The general exception, which is the one commercial actors rely on, only applies if the rightsholder has not reserved their rights.

3. The Machine-Readable Opt-Out

For the general exception, rights reservation must be expressed in an appropriate machine-readable manner for content made publicly available online.

In practice that has meant a variety of mechanisms:

• robots.txt directives, including AI-specific user agents.
• X-Robots-Tag and noai / noimageai style meta directives.
• Terms of service statements, whose machine-readability is contested.
• Emerging standard vocabularies for expressing reservations.

The honest position is that the mechanism is not fully settled, which is precisely why respecting the signals that do exist is the defensible course. A collector that ignores robots.txt cannot claim it did not see a reservation expressed there.

4. Lawful Access Is a Precondition

Both exceptions require lawful access to the content. That has concrete consequences:

• Content behind a paywall requires a subscription.
• Content behind authentication requires legitimate credentials.
• Circumventing a technical access control defeats the exception.
• Content obtained in breach of contract is not lawfully accessed.

So TDM exceptions do not license getting in. They license analysing what you were lawfully able to read.

5. Other Jurisdictions Differ

The US position is structurally different: rather than an exception, the question runs through fair use, which is a case-by-case balancing rather than a rule.

6. What This Means in Practice

1. Honour robots.txt, including AI-specific agents. It is the clearest place a reservation appears.
2. Do not circumvent access controls — doing so removes the lawful access precondition.
3. Record what access rules applied at fetch time, per record. That is the evidence that you respected reservations.
4. Distinguish research from commercial purpose honestly, since the exceptions differ sharply.
5. Retain only what the purpose needs under the general exception.
6. Take advice for anything at scale, because this area is actively developing.

Point 3 is the operationally important one and the easiest to omit: recording the robots.txt state observed at fetch time turns compliance from an assertion into a record.

7. Why It Matters More Every Year

TDM exceptions were drafted with research analysis in mind and now sit at the centre of disputes about AI training corpora. The direction of travel is toward more explicit reservation mechanisms and more litigation about what counts as machine-readable.

For anyone building corpora, that argues for over-recording rather than under-recording: provenance, access rules, and dates, captured during collection.

8. Summary

A TDM exception permits copying protected works for automated analysis, and the EU has two — a robust research exception that cannot be opted out of, and a general one that applies only where the rightsholder has not reserved rights in a machine-readable way.

Both require lawful access, so circumventing a paywall or access control removes the exception entirely — and because the machine-readability standard is unsettled, honouring robots.txt and recording the access rules observed at fetch time is what makes compliance demonstrable rather than merely claimed.

Potential Related Glossary

• Web Crawling
• Unstructured Data
• Parsing
• Data Fusion