Flint

Flint is a smart data catalog. It connects to your databases and SaaS tools, discovers their schemas, and uses LLMs to explain what that data means and what you could actually do with it. Traditional data catalogs tell you what data exists. Flint is built to answer the question that comes next: now that I have all of this data, what do I do with it?

This is the biggest thing I've built as a solo developer. I had two goals going in. The first was to work on something large and unfamiliar enough that I'd have to learn a lot of new things. The second was to build a product that solved a problem I'd actually watched play out with real customers.

Flint dashboard showing connected sources, catalog size, insight counts, and recent activity
The dashboard: connected sources, catalog size, and recent activity at a glance.

Why I Built This

I've worked at a few data platform companies. The great thing about a data platform is that it can bring in data from a lot of different sources, give people tools to transform it, help them drive insights, and send that data somewhere else.

But the most common question I heard from customers was always some version of: what do I do with my data? We gave them the tools to bring it in, work with it, and transform it. All of that assumes you already understand your data and know what's possible with it. That assumption turns out to be the hard part.

If you're a domain-specific platform, you can answer that question directly, because you know what the data is and what people typically do with it. If you're a general platform, you have two options. Either you go learn every source yourself, figuring out what's inside so you can help customers find insights, or you leave it to them to figure it out. The first doesn't scale past a handful of sources. The second is how customers end up with a warehouse full of data they never use.

I was curious whether LLMs could close that gap: surface the insights without anyone having to become an expert in every data source first. Flint is my take on using LLMs to bring this idea to life.

What Flint Does

You connect a source, Flint reads its structure, and then it starts explaining that structure back to you. The flow looks like this:

  1. Connect a source: PostgreSQL natively, SaaS tools like Stripe, HubSpot, and Salesforce through Airbyte.
  2. Sync the metadata: Flint discovers schemas, tables, and columns, plus estimated statistics like row counts, null rates, and common values.
  3. Generate descriptions: an LLM writes an overview of the source and a description of each table, in plain language.
  4. Suggest use cases: for each source, Flint proposes specific analyses you could run, with starter SQL.
  5. Discover across sources: pick two connected sources and Flint looks for ways to join them and reasons about what combining them would tell you.

Connecting Data

Flint supports native connectors with code in Flint to talk to the database directly. Everything else runs through PyAirbyte. Rather than building connectors one at a time, Flint borrows the 300+ sources Airbyte has already built. Stripe, HubSpot, and Salesforce ship today, and adding another is mostly a matter of writing a small migration.

There are three different shapes of Airbyte connector, and I ended up building an example of each: Stripe runs straight out of the Airbyte code, HubSpot uses a vendor-maintained package on PyPI that needs extra configuration, and Salesforce runs inside its own virtual environment. Flint handles all three, so adding a new source doesn't require knowing which kind you're dealing with. The README walks through how to probe a connector for its config shape and register it.

There's also a demo mode with pre-loaded HubSpot, Google Analytics, and Customer Database sources, so you can see what the catalog and the insights look like before wiring up real credentials.

Once a source is connected, you can test the connection, sync it on demand, or put it on a schedule.

Every sync and every LLM job runs asynchronously on Celery and Redis. You can kick off a sync, navigate away, and come back to it. Nothing makes you sit and watch a spinner.

From Metadata to Meaning

When a source syncs Flint generates an overview of the source: what's in it and, at a high level, what it's good for. Open an individual table and you get the same pattern one level down, with the columns, the statistics, and an LLM-written description of what that table holds and how you might use it.

At the source level, there's a generate use cases action that sends every table, every column, and the source overview to an LLM and asks it to propose concrete analyses. Each one comes back with a plain-language business description, the tables it depends on, and the starter SQL to run it. Every column the SQL references is validated against the schema Flint actually synced before the use case is shown to you, so you shouldn't get a query pointing at fields that don't exist.

Cross-Source Discovery

Use cases for a single source are useful, but the reason anyone consolidates data in the first place is to combine it. If I've connected my CRM and my online store, I don't just want to know what each one can tell me. I want to know what they can tell me together.

You pick two synced sources and run discovery through a LLM pipeline with a few distinct stages:

  1. Find the joins. The schemas of both sources go to an LLM that looks for plausible ways to connect them. A HubSpot contact ID and an online store user ID are never going to match, but both tables have an email address, and that's a real join.
  2. Ask if it's worth it. Being able to join two tables doesn't mean you should. A second pass generates hypotheses about what combining them would actually reveal.
  3. Score and rank. The hypotheses get scored, and only the most interesting handful move forward.
  4. Write the use cases. A higher-quality model turns the survivors into full use cases: what the analysis is, what the business value would be, which sources it spans, and starter SQL to get going.

Results land in a review queue where you accept or dismiss each one, so the catalog fills up with the ideas you actually thought were good.

Flint source detail page showing the LLM source overview, discovered schema, suggested use cases, and sync schedule and history
A source detail page: the LLM source overview, discovered schema, suggested use cases, and the sync schedule and history.
Flint cross-source discovery page proposing cross-source use cases with accept and dismiss actions
Cross-source discovery: pick two synced sources and review the AI-proposed use cases.

How It's Built

At a high level: Django with HTMX and Tailwind on the front end, PostgreSQL in production, and Celery with Redis handling every long-running job. Source credentials are encrypted at rest with Fernet, decrypted only at the moment a connector is instantiated, and never logged.

All LLM work sits behind a provider abstraction, with Anthropic as the runtime default and OpenAI available as an alternative. A fast model tier handles most of the generation; a higher-quality tier writes the cross-source use cases, where the reasoning matters more than the latency.

The README has the full architecture diagrams, the complete model reference, setup instructions, and a guide to adding new Airbyte-backed sources.

See It in Action

Here's a walkthrough of the full product: connecting and syncing a source, generating insights and use cases, and running cross-source discovery between two connected sources.