Parse
- Text, tables and headings pulled from PDF, HTML, Office files and wiki pages
- Scanned pages sent through OCR with a quality flag
- Repeated headers, footers and navigation removed
AI data pipelines
Data pipelines for AI integration
A model is only as current and as safe as the data it is given. Agnotic Technologies builds the pipelines that feed AI systems: document ingestion for retrieval, versioned training and evaluation sets, de-identification before data reaches a model, and checks that stop stale or exposed data before it is used.
Retrieval ingestionAccess labels keptDeletion syncDataset lineage
Trusted by global innovators
AI data pipelines
What are data pipelines for AI integration?
Data pipelines for AI integration move source data into the form an AI system uses, and keep it correct afterwards. Agnotic Technologies builds retrieval ingestion that parses, chunks, embeds and indexes documents with their access labels, propagates edits and deletions, versions training and evaluation datasets with lineage, and de-identifies data before any model sees it.
These pipelines differ from analytics pipelines. A dashboard can live with a late row; an assistant that answers from a withdrawn policy, or shows a document to the wrong person, cannot.
Analytics ETL, streaming warehouses and testing a RAG system after launch have their own pages, linked at the end. This page covers the data path into the model.
Retrieval ingestion
What happens between a source document and the index?
Five stages, each recording what it did, so any chunk in the index can be traced to the source version it came from.
Chunk
- Split on headings and sections, not fixed character counts alone
- Document and section titles kept with every chunk
- A content hash per chunk to detect real changes
Embed
- Embedding model and version stored with each vector
- Batch re-embedding when the model changes
- Retries and a dead-letter queue for failed calls
Label
- Source permissions copied onto every chunk
- Tenant, team or role filters the retriever must apply
- A sensitivity tag for data that needs extra handling
Index and sync
- Upserts keyed by source document ID and chunk number
- Tombstones for deleted sources
- Counts checked between sources, chunks and vectors
Updates and deletions
How do edits and deletions reach the index?
Many retrieval systems are built for the first load only. These steps keep the index matching its sources after that.
1.
Detect the change
Change events, webhooks or modified timestamps from each source tell the pipeline which documents changed, moved or disappeared.
Detect2.
Rebuild only what changed
Changed documents are parsed and chunked again; chunks whose hash is unchanged keep their vectors, so embedding cost follows real edits.
Rebuild3.
Remove by source ID
A deleted or unshared document removes every chunk carrying its source ID, and the pipeline confirms the count reached zero.
Remove4.
Prove it
A scheduled check searches for recently deleted documents and fails the run if any can still be retrieved.
VerifyFailure points
Where do AI data pipelines fail?
The failures we design against, each with the pipeline check that prevents it.
Challenge
Stale embeddings: a source document is edited, but its old chunks keep answering questions.
Agnotic approach
Content hashes per chunk, change detection at the source and a freshness report of documents whose index copy is older than the original.
Challenge
Deleted documents can still be retrieved after they were removed at the source.
Agnotic approach
Deletion by source ID with tombstones, and a scheduled test that fails if a deleted document is returned.
Challenge
Access labels are dropped at chunking, so a restricted file becomes searchable by everyone.
Agnotic approach
Labels are required fields on every chunk, and ingestion rejects any chunk that arrives without them.
Challenge
Training rows have no lineage, so nobody can say where an example came from or remove it on request.
Agnotic approach
Every row carries its source, extraction run and dataset version, written to a provenance log.
Challenge
Identifiers are left in free text, such as names in notes or account numbers in email bodies.
Agnotic approach
Structured identifiers are removed or masked, free text runs through entity detection, and a person reviews a sample before release.
Challenge
Evaluation examples leak into training data, so test scores look better than real use.
Agnotic approach
Splits are fixed by source document and checked for overlap before each training run.
Challenge
Vectors from a new embedding model are mixed with vectors from the old one.
Agnotic approach
Model version stored per vector, a full re-embed into a new index, and a switch only after retrieval checks pass.
Training data
How do we version training and evaluation datasets?
A model can only be explained, retrained or corrected if the data behind it can be rebuilt exactly.
Versions
- Immutable snapshots with a version ID
- Each training run records the dataset version it used
- Old versions kept for the retention period you set
Lineage
- Source system, query and extraction run for every row
- Provenance logged as entities, activities and agents, following W3C PROV
- Labels traced to the labeler and the guideline version
Splits
- Train, validation and test fixed by source, not by random row
- Overlap checks before every run
- A held-out set nobody tunes against
Removal
- Every dataset version that holds a record can be found
- Rebuilt without it, with the change logged
- Retraining triggered when your policy requires it
Where does de-identification sit in the pipeline?
De-identification runs before data reaches a model, an embedding service or a labeling team, not after. Once raw identifiers have been sent to a third-party model API, removing them later does not undo the disclosure.
For protected health information, 45 CFR 164.514(b) gives two methods: a documented expert determination, or safe harbor, which removes 18 listed identifiers and requires no actual knowledge that the remaining data could identify someone.
Free text is the hard part. Names, dates and record numbers hide in notes and messages, so we combine pattern rules, entity detection and a reviewed sample, and keep any re-identification key in a separate, restricted store.
For other personal data, NIST SP 800-188 sets out de-identification methods and their disclosure risks, which helps us choose between masking, generalization and synthetic data for each field.
Quality gates
Quality gates on every pipeline load
- Freshness per source: the run fails if a source has not delivered within its agreed window
- Counts reconciled between source documents, chunks and stored vectors
- Schema and format changes in a source flagged before they reach parsing
- Distribution checks on training features, so input drift is caught before retraining
- Null, duplicate and empty-text rates compared with the previous run
- Label coverage: no chunk reaches the index without its access labels
Which teams need AI data pipelines?
Teams launching an internal assistant
Policies, wikis and tickets indexed with the same permissions people have in the source tools.
SaaS products adding AI features
Customer data embedded per tenant, with tenant filters on every retrieval and full deletion when a customer leaves.
Healthcare and life sciences teams
Clinical text de-identified before model training or vendor APIs, with lineage for every record.
ML teams retraining models
Versioned datasets and fixed splits, so a new model can be compared fairly with the last one.
Startups proving an AI feature
Data pipelines for AI integration for startups begin with one source and a small index, built so more sources can join without a rebuild.
References
Standards and frameworks behind our AI pipelines
45 CFR 164.514NIST SP 800-188NIST AI RMF 1.0W3C PROV
Facts and sources
Facts behind AI data pipeline design
Rules and standards that shape how we de-identify, trace and govern data before it reaches a model.
45 CFR 164.514(b) allows health information to be treated as de-identified in two ways: a documented expert determination, or the safe harbor method of removing 18 listed identifiers with no actual knowledge that the rest could identify the person.
Source: 45 CFR 164.514 (eCFR)NIST SP 800-188, published in September 2023, describes de-identification as removing the association between identifying data and the data subject, with the goal of limiting disclosure risk while the data stays useful.
Source: NIST CSRC: SP 800-188, De-Identifying Government DatasetsThe W3C PROV family of documents defines a model and serializations for provenance: information about the entities, activities and people involved in producing a piece of data.
Source: W3C: PROV-OverviewThe NIST AI Risk Management Framework, released on January 26, 2023, is intended for voluntary use to build trustworthiness into the design, development, use and evaluation of AI systems.
Source: NIST: AI Risk Management Framework
Scope
What drives the cost of an AI data pipeline?
We do not publish prices. These factors set the size of the build.
Number and type of sources
Each wiki, drive, database or ticket system needs its own connector, change detection and permission mapping.
Document formats
Clean HTML is quick to parse; scanned PDFs, tables and slides need OCR and layout handling.
Permission model
Mapping source permissions to chunk labels is simple for one tenant and harder with nested groups and shared links.
Freshness target
Near real-time updates need change feeds and incremental processing; a weekly rebuild does not.
Sensitive data
PHI or other personal data adds de-identification, review and restricted environments.
Training data needs
Versioned datasets, labeling workflows and lineage records add work beyond retrieval.
Frequently Asked Questions
We start from the use: which questions the AI answers or which model it trains. Then we list the sources and their permissions, build ingestion for one source end to end with its checks, and add sources once that path is proven.
It depends mostly on the number of sources, their formats and the permission model. We agree a first deliverable on the scoping call, usually one source indexed end to end with its checks running.
We choose the store with you: a vector extension such as pgvector in PostgreSQL, a search engine with vector support, or a managed vector database. What matters more is that it filters by access label and deletes by source ID.
Our knowledge base RAG work covers the whole assistant, including retrieval, prompts and testing after launch. This page is the data path underneath it: getting documents in correctly and keeping the index in step with its sources.
Yes. Pipelines can read from your warehouse, lake or change data capture streams and write to your vector store, scheduled by the orchestrator you already run.
Often, yes, when personal data goes to a third-party model or to labelers. HIPAA governs protected health information; for other data your privacy obligations and contracts decide, and the same removal and review steps apply.