Epic Clarity Data Integration

What is Epic Clarity data integration?

Epic Clarity data integration means taking data from Clarity, the relational reporting database that a health system loads from Epic's operational database on a schedule, and delivering it into another system for analytics. Because Clarity is refreshed by batch ETL, often nightly, the integration serves reporting, research and population work rather than real-time workflows.

The health system owns Clarity and decides who sees which data. An outside team works through the health system's reporting group, and the integration is as much an agreement about scope and delivery as it is a pipeline.

Clarity, Caboodle and Chronicles

Chronicles is Epic's operational database, Clarity is the relational reporting database loaded from it by ETL, and Caboodle is Epic's data warehouse. Each answers a different kind of question, and mixing them up is the root of most design errors.

• Chronicles: The live system clinicians and staff work in. Outside teams do not query it for reporting, and live data reaches outside systems through interfaces and APIs instead.
• Clarity: A relational copy of operational data, refreshed by batch loads, that reporting analysts query with SQL. It holds detailed history at the level of encounters, orders, results and billing transactions.
• Caboodle: Epic's data warehouse, organized for analytics. Where a health system already curates Caboodle, it can be the simpler source for measures and dashboards.

Clarity table and column structures are documented to Epic customers rather than publicly, so the health system's reporting analysts are the authority on which objects hold the data you need and how they behave at that site.

Plan around their knowledge. A short working session where the reporting team walks through the source objects for each data domain saves weeks of guessing from column names.

How does Clarity data reach an outside team?

Clarity data reaches an outside team in one of three ways, and the health system picks the one its policies allow. The choice decides who writes the queries, who maintains them and how failures are noticed.

• Scheduled file extract: The health system's reporting team runs agreed queries and delivers files to a secure location on a schedule. The vendor loads and validates them.
• Curated views: The health system exposes views over Clarity or its warehouse to a controlled account. The vendor queries within those views, and the health system controls what they include.
• Health system warehouse feed: The data comes from the health system's own enterprise warehouse, which may combine Epic data with other sources, rather than from Clarity directly.

Whichever route is used, ask for a written data specification: the data domains, the date range, the columns, the refresh schedule, and how corrections and deletions appear. That document becomes the contract both teams test against.

Agree the delivery time relative to the health system's own Clarity load. If the nightly load finishes late, an extract scheduled by the clock ships stale data, so key each run to a completion signal or a load date rather than a fixed time.

Legacy data often enters the same conversation. When a health system moved to Epic from an older EHR, history before the go-live date may sit in an archive rather than in Clarity, so ask where pre-Epic records live before promising a ten-year trend.

How do you build a Clarity extract step by step?

A Clarity extract is built in six steps, starting with scope and ending with proof that the warehouse matches the source. The health system owns access; the receiving team owns the model and the evidence.

• 1. Scope: Name the data domains, date range and population. Apply the minimum necessary standard: if a field does not serve the use case, leave it out.
• 2. Agreement: Put the business associate agreement or data use agreement in place, and confirm whether the data will be identified, a limited data set or de-identified.
• 3. Grain and keys: Fix the grain of each table (patient, encounter, order, result or daily snapshot) and the identifiers that join them.
• 4. Incremental loads: Decide how changed rows are detected, how deletes and corrections arrive, and how merged patients are handled after the first full load.
• 5. Late-arriving data: Allow for facts that land after the event, such as a diagnosis coded days after a visit, by reloading a rolling window rather than only the latest day.
• 6. Prove it: Reconcile row counts against a query the health system runs, check a set of held-out patients by hand against the chart, and alert on loads that fail or arrive empty.

Keep a load log with the date range, row counts and checksum of every delivery. When an analyst questions a number months later, the log shows exactly which load produced it.

Version the data specification alongside the pipeline code. When the health system adds a column or changes a code set, the change, its date and the first load it affected are recorded in one place.

Which HIPAA rules shape a reporting extract?

A patient-level extract from Clarity is ePHI in every copy, so the Privacy Rule decides what may be shared and the Security Rule decides how each copy is protected. These are the rules worth quoting in the data specification.

• The minimum necessary standard at 45 CFR 164.502(b) and 164.514(d) requires a covered entity to limit uses and disclosures of PHI to the minimum necessary for the purpose, which applies to the columns in an extract.
• 45 CFR 164.514(b) recognizes two de-identification methods: the safe harbor method, which removes listed identifiers, and expert determination.
• 45 CFR 164.514(e) allows a limited data set to be disclosed for research, public health or health care operations under a data use agreement.
• Audit controls are a required safeguard at 45 CFR 164.312(b), so every query run against an identified warehouse should leave a record someone can review.
• A vendor that receives identified extracts to perform a service for the health system is a business associate, and the contract must meet 45 CFR 164.504(e).

Decide the identification level per use case. A quality dashboard for the health system's own staff may need identified data, while a model-training set or a research cohort often works on a limited data set or de-identified data.

Count every copy: the landing folder, the staging schema, the warehouse, analyst exports and backups. Each one needs access control, a documented decision on encryption under the addressable specification, and a retention rule with an owner.

When is Bulk Data or Bridges the better source?

Use another Epic path when the job needs fresher data or a standard format. Clarity fits history and analytics; it is the wrong source for anything users expect to react within minutes.

• Events as they happen: Admissions, orders and results that must trigger work belong on an HL7 v2 interface built in Bridges.
• Standard-format population data: When the use case fits US Core resources, a FHIR Bulk Data export through a backend client gives NDJSON files in a standard shape, and certified EHRs support it under 45 CFR 170.315(g)(10).
• One patient on demand: A SMART on FHIR app reads the record of the patient a user has open.
• Deep operational history: Detailed billing, scheduling or departmental history that FHIR resources do not cover is where Clarity or Caboodle extracts earn their place.

Products often combine sources: a nightly Clarity extract for history and a live HL7 v2 feed for today. Keep one patient and encounter model so the two line up.

Clarity extract failure points

These are the failure points that make a warehouse look complete while being wrong. Each one can be checked before the first dashboard is published.

• Treating Clarity as live: An alert or worklist is built on data that is a day old.
• Copying everything: Whole tables are extracted just in case, creating a large identified copy with no use case.
• Loading only the latest day: Late-arriving diagnoses and corrections never reach the warehouse.
• Ignoring merges: Two patient keys for one person split history and double-count the population.
• No held-out check: A silent join error ships because nobody compared a sample of patients with the chart.
• Undocumented source objects: The health system changes a view and the extract breaks without anyone knowing why.
• Schedule drift: The extract runs before the nightly Clarity load completes and delivers the previous day twice.
• Shared PHI tables: Identified columns sit in tables that a wide analyst group can read.

How Agnotic can help

A Clarity integration is only as useful as the trust analysts place in its numbers. Agnotic Technologies builds the receiving side of Clarity and Caboodle extracts with that trust in mind: data specifications, secure delivery, incremental loads with rolling windows, merge handling, reconciliation against source queries and access controls on identified data.

Where a job needs fresher data, Agnotic adds the HL7 v2 or FHIR path alongside the extract, using the same patient and encounter model.

Sources

• 45 CFR 164.502: Uses and disclosures, including minimum necessary (b) and business associates (e)
• 45 CFR 164.514: De-identification, minimum necessary and limited data sets
• 45 CFR 164.312: Technical safeguards
• 45 CFR 164.504(e): Business associate contracts
• ASTP/ONC: Standardized API, 45 CFR 170.315(g)(10)
• HL7: FHIR Bulk Data Access implementation guide