- AI for data engineering is sold as pipeline authorship, and that is the half it is worst at. A generated transform that is wrong is still valid SQL, so nothing fails and nobody finds out until a number moves.
- The half it can own is everything downstream of the pipeline: schema drift, identity resolution, a source that went quiet, a destination that started failing. Those checks are self-verifying, which is what makes them safe to hand over.
- The test is not how hard a task is. It is whether a wrong answer announces itself.
Most writing about AI for data engineering describes the same demo: a model writes the pipeline for you. It generates the dbt model, the transform, the connector config. It is the most impressive thing to show and the least useful thing to trust, and if you run the data layer behind a customer data platform it is also the half of the job you were least worried about.
The reason is not that the generated code is bad. It is usually fine. The reason is that when it is wrong, nothing tells you. A transform that quietly drops a join condition is still valid SQL. It runs. It returns rows. The pipeline is green, the dashboard renders, and the error surfaces six weeks later as a revenue number that moved for no reason anyone can name.
The other half of data engineering is different. Watching what actually arrives, whether the schema changed, whether identities are still merging, whether the destination is still accepting writes: every one of those answers can be checked against the data in about a minute. That is the half worth handing over first, and almost nobody frames it that way.
The two jobs sold as one
AI for data engineering splits into authorship and operation. Authorship is writing the pipeline: transforms, SQL, connector code, schema definitions. Operation is running it: verifying sources fired, catching a property that stopped arriving, checking identity resolution, watching destinations for failures. Vendors sell both under one label, and the two have opposite risk profiles.
Authorship failures are silent and they compound. Operation failures are loud and immediate. That single difference decides which one you can automate this quarter.
It also explains a pattern most teams recognise once it is named. Adopting a coding assistant for the data layer produces more output without producing more trust, because the constraint was never how fast anyone typed a transform. The constraint is the distance between a break and the moment someone notices it. Generating code faster does not touch that distance.
The verifiability test
Here is the rule that decides what to hand over: an AI-generated output is safe to automate when a wrong answer announces itself. Not when the task is easy, and not when the model scores well on it. When being wrong is loud.
Apply it task by task and the split stops being a matter of taste.
| Data engineering task | How a wrong answer shows up | Safe to automate now |
|---|---|---|
| Write a transform or dbt model | Valid SQL, plausible rows, no error. Found weeks later. | No |
| Design an event schema | Nothing breaks. The data stops answering the question. | No |
| Define identity resolution rules | Two people merge into one. Reads as a churn dip. | No |
| Check every source is still firing | Compare against the stream. Wrong within the hour. | Yes |
| Detect a property that stopped arriving | Present or absent in the last 24 hours. Binary. | Yes |
| Verify identity merge rate | Count orphaned anonymous profiles. One query. | Yes |
| Watch destination write failures | The destination returns an error or it does not. | Yes |
| Flag segments depending on a changed property | Enumerable from the segment definitions. | Yes |
Every row in the bottom half has the same shape. The output is a claim about a fact that already exists somewhere you can look. Every row in the top half produces something new, and new things have no reference to check against except a human reading them.
This is also why the second half went last. It is on-call work. It does not demo well. It is most of the week.
What the operation half actually contains
The operational failures that hurt a GTM data layer are not warehouse failures. They sit one layer out, and they break the tools that act on the data before anything in the warehouse looks unusual.
- A source went quiet. The web SDK stopped firing after a deploy and nobody noticed for four days.
- A property disappeared. Someone renamed a field, and every segment built on it stopped matching.
- Identity stopped resolving. A signup form changed, anonymous sessions no longer stitch to known profiles, and the funnel reads as a conversion drop.
- A destination started failing. The topic rejects writes on a schema mismatch, and the consumer downstream reads stale records.
- Consent state drifted. A record is being published that should have been suppressed.
None of these produce an error anyone sees. They produce a number that moved. The gap between the break and the discovery is where the cost sits, and closing that gap is a monitoring problem, not an intelligence problem.
What it looks like when an agent runs the watching half
This is the scope we gave the Data Engineer agent inside Intempt, and the boundary is deliberate. It does not invent a destination. It publishes only to topics and buckets you have already declared, fails loudly rather than creating infrastructure on your account, and tells you what a schema change would break before it breaks it.
In practice that is three jobs. Event sync to infrastructure you own, routing enriched events to your own Kafka topics and S3 buckets in the format your consumers already read, with consent checked before the send. Schema and identity upkeep, watching what is arriving, flagging properties that appeared or stopped, and showing which segments depend on them. Source and destination health, so a source that went quiet or a destination that started failing surfaces as it happens rather than the week someone notices the numbers moved.
The data quality routine underneath it runs six checks: every source connection verified, event volumes watched for drops and spikes, schema drift flagged on changed or missing properties, identity resolution checked by merge rate and orphaned anonymous profiles, Slack alerts configured on breaks, and one health dashboard composed from all of it. Every one of those is a bottom-half task by the verifiability test. That is the selection rule, not a coincidence.
Activation stays under your control on the same principle. Publishing to Kafka and other destinations uses ordered topic routing, your own partition keys, and JSON or Avro output, because the format contract belongs to the consumer team. And because profiles and events share one schema, with anonymous sessions stitching to known profiles the moment someone identifies, there is no warehouse round-trip to go stale between the event and the action.
Three checks worth automating first
If you are choosing where to start, pick the checks that are cheap to verify and expensive to miss. All three below fail loudly, which means an automation error costs you a day rather than a quarter.
- Source liveness. For every connected integration, has it sent an event in the last 24 hours? A source going quiet is the most common break and the slowest to notice, because absence never throws.
- Schema drift on your top events. Take the 10 or 15 events your reports and segments depend on, and alert on any property that appeared, disappeared, or changed type. Then list the segments that reference it, because that is the blast radius.
- Identity merge rate. Track the share of anonymous sessions that stitch to a known profile, and the count of orphaned anonymous profiles. A form change or a script-order change moves this number before it moves any funnel, which makes it the earliest warning you have.
Get those three running before automating anything in the authorship half. If a team cannot yet tell you within a day that a source stopped firing, generated pipelines add surface area to a system nobody is watching.
What this does to the job
The honest read is that a real chunk of data engineering goes away, and it is the chunk nobody defends. Nobody chose this work because they enjoyed finding out on Thursday that Tuesday's deploy killed an event.
What stays is the modelling. Deciding what an event means and what it must never mean. Deciding what counts as one customer across three anonymous devices and two email addresses. Deciding which downstream contract cannot break, and what the fallback is when a source changes shape. Those are the calls that make the top half of the table unsafe to automate, and they are the same calls that make the role worth having.
The shift matches what already happened to the analyst role, where the mechanical shaping layer moved to agents and the judgment layer did not. We wrote that up in the data analyst is the GTM job AI actually changes, and the pattern here is the same one arriving a layer down. For the tooling picture around it, generative AI for data analytics covers where the query and reporting layer landed.
This is how we build the agentic GTM software: agents own the parts of a job where being wrong is immediately visible, and hand back the parts where it is not. Applied to the data layer, that is the whole argument for AI for data engineering worth making today.
Frequently asked questions.Answered.
- AI for data engineering covers two different jobs that get sold as one. The first is authorship: a model writes transforms, SQL, and connector code. The second is operation: a model watches what is arriving, flags schema changes, checks identity resolution, and reports which sources and destinations broke. The second job is the one where AI is dependable today, because a monitoring result can be checked against the data itself, and a generated transform cannot be checked without a human reading it.
