Recognising Compound Problems - Complexity and Complicated

If you manage complex things as if they are merely complicated, you’re likely to be setting your company up for failure.
Rick Nason

A few weeks ago I stumbled into a lively LinkedIn thread about the Modern Data Stack (MDS). While his piece was solid, it was the comments that stuck with me. Dozens of smart people were saying, in one way or another, “I can’t wrap my head around this MDS thing — it’s so complex.” I had a sip of tea, scrolled some more, and thought: is it actually complex? Or just complicated?

This distinction isn’t academic nitpicking. It changes how you plan, how you estimate, and ultimately whether your project works.

Complicated versus complex (in plain English)

Both complicated and complex systems involve lots of moving parts. That’s where the confusion starts. But they aren’t the same beast.

• Complicated problems are knotty but knowable. They have clear cause and effect. If you bring the right expertise, break the work down, and follow the steps, you can expect a repeatable outcome. Think: building a jet engine, migrating a data warehouse, performing heart surgery. Hard? Absolutely. Predictable with the right plan and skills? Largely, yes.
• Complex problems have shifting patterns and feedback loops. Cause and effect only make sense in hindsight. The system adapts to your interventions. You can’t “solve” it once and for all; you manage it over time. Think: an economy, a large organisation’s culture, public health responses, or how teams adopt (or quietly resist) a new data platform.

To borrow a simple test: if you can write a runbook and expect it to work everywhere, every time, you’re probably in complicated territory. If the system fights back, morphs, or produces side effects you didn’t predict, you’re in complex land.

Where MDS actually sits

The technology side of MDS — warehouses like Snowflake or BigQuery, orchestration with Airflow, transformations with dbt, streaming with Kafka, observability with tools like Monte Carlo — is mostly complicated. There’s a lot to learn, and there are plenty of ways to do it badly, but you can draw a map. With enough diagrams, linting rules, and a good CI pipeline, you can get repeatable builds.

Where it appears to tip into complexity is everything around the tech:

• Data producers are changing upstream schemas without telling anyone, because their priorities shifted last week.
• Compliance stepping in mid-project with a new interpretation of retention rules.
• A leadership shuffle that reorders incentives, suddenly making “self-serve analytics” less appealing to team leads guarding their backlogs.
• A finance team asking for a single definition of “active customer” when three different product lines all have defensible versions.

At that point, the MDS project isn’t just pipelines and tables. It’s people, politics, language, power, and time. Those behaviours are emergent, not fully predictable. That’s what makes parts of MDS complex in practice, even if the tools themselves aren’t.

Why the distinction matters for architects and engineers

If you treat a complex problem as if it were merely complicated, you overcommit to big designs, long roadmaps, and fantasy deadlines. You end up confused when the “obvious” solution stops working after go-live. On the flip side, if you treat a complicated problem as complex, you can drift: endless workshops, vague principles, nothing shippable.

A few signs you’re facing each type:

Complicated

• You can point to a reference architecture that is known to work.
• Specialists can predict effort with tolerable error bars.
• Checklists, runbooks and playbooks actually reduce incidents.

Complex

• Stakeholders “agree” in the meeting, then behave differently a week later.
• Small changes have outsized, surprising effects (usage spikes, shadow processes).
• Metrics improve while outcomes don’t, or vice versa.

What to do differently

For complicated work, go crisp:

• Specify the outcome. Migrate this workload. Reduce refresh time from 4 hours to 30 minutes. Backfill 24 months without exceeding budgeted slot time.
• Decompose the work. Dependency diagrams, DAGs, interface contracts, proper estimates.
• Standardise. Templates for dbt models, naming conventions, test thresholds, clear ownership in the CODEOWNERS file.
• Use experts. Bring in the person who’s actually scaled Kafka connectors at 10k messages/sec, not just someone who watched a two-hour tutorial.

For complex work, change the playbook:

• Start with probes, not pronouncements. Run a safe-to-fail experiment: pilot a semantic layer with a single business unit; watch adoption for six weeks.
• Optimise for feedback. Shorten the loop between data producers and consumers. Lightweight data contracts, real humans in the Slack channel, office hours that people actually attend.
• Look for narrative and numbers. Dashboards won’t tell you why teams aren’t using the new warehouse. Short interviews might.
• Stabilise on values, not just tools. “We don’t break downstream teams without notice” is a value. The mechanism (contracts, CI checks, canary releases) can evolve.
• Expect adaptation. The first policy you write will be too strict or too vague. Adjust it. Write it down. Move on.

How to win people over when they want tidy plans

If you’re asking your organisation to handle complex work differently, you’re not just pitching a new method. You’re poking at comfort. Big plans, fixed dates and green RAG statuses feel safe. Experiments, short feedback loops and “we don’t know yet” do not. Expect raised eyebrows. Plan for them.

A simple reframing to start

• Skip the theory. Instead of “this is complex so we need a new framework,” try “we’ve got moving targets and partial information; here’s how we ship without betting the farm.”
• Talk risk, not rituals. Smaller blasts, tighter guardrails, quicker feedback. Most leaders will back anything that reduces ugly surprises.

Prove it in a small corner first

Run a tiny pilot. One team, one outcome, four to six weeks. Write a single page before you start: what you’ll try, what could go wrong, what signals count as progress, and the exact conditions for stopping. When you show the results, include what surprised you. People smell honesty, and it travels.

Translate your approach for each audience

• Executives: “Two lanes.” The knowables (e.g. migrations, integrations) get dates. The unknowables (e.g. adoption, definitions, policy) get decision checkpoints and review cadences.
• Finance: “Smaller bets with faster stop/scale calls.” Cap the spend per experiment. Put a Go/No-Go date on the calendar. Share the cost of carrying a wrong assumption.
• Risk/Compliance: “Controls-first experiments.” Use synthetic data or shadow mode, canary cohorts, approvals in the pipeline, and a rollback you can execute in minutes.
• Delivery teams: “Less whiplash.” Shorter commitments, fewer big-bang reveals, and a genuine say in what sticks.

For complicated work, go crisp:

• Specify the outcome. Migrate this workload. Reduce refresh time from 4 hours to 30 minutes. Backfill 24 months without exceeding budgeted slot time.
• Decompose the work. Dependency diagrams, DAGs, interface contracts, proper estimates.
• Standardise. Templates for dbt models, naming conventions, test thresholds, clear ownership in the CODEOWNERS file.
• Use experts. Bring in the person who’s actually scaled Kafka connectors at 10k messages/sec, not just someone who watched a two-hour tutorial.

For complex work, change the playbook:

• Start with probes, not pronouncements. Run a safe-to-fail experiment: pilot a semantic layer with a single business unit; watch adoption for six weeks.
• Optimise for feedback. Shorten the loop between data producers and consumers. Lightweight data contracts, real humans in the Slack channel, office hours that people actually attend.
• Look for narrative and numbers. Dashboards won’t tell you why teams aren’t using the new warehouse. Short interviews might.
• Stabilise on values, not just tools. “We don’t break downstream teams without notice” is a value. The mechanism (contracts, CI checks, canary releases) can evolve.
• Expect adaptation. The first policy you write will be too strict or too vague. Adjust it. Write it down. Move on.

Language that lowers resistance

• “You’ll get a firm date for the next decision, not a fantasy date for the final answer.”
• “We’ll plan the bits we understand; we’ll learn the bits we don’t—cheaply and in public.”
• “Tighter controls where the blast radius is big; looser where it isn’t.”

Common objections you’ll hear (and calm replies)

• “We need firm dates.” Split the work. Dates for the knowable stream. Timeboxed probes with checkpoints for the learnable stream. Show both lanes on one page.
• “This will slow us down.” Put two options side by side: a nine‑month big bang with a history of 60% rework versus three six‑week probes you can kill or scale. If you’ve got a past rework bill, quote it.
• “Compliance won’t allow it.” Invite them in early. Co‑design guardrails. Offer synthetic data, shadow traffic or a canary cohort. Show the rollback plan.
• “We tried agile; it didn’t work.” Acknowledge the fatigue. “This isn’t ceremonies. It’s about cutting the cost of being wrong.” Then change one concrete practice, e.g. decisions recorded with exit criteria.
• - “Stakeholders will get confused.” Send a weekly narrative update: what we tried, what happened, what we’ll do next, and what we’re watching. One page, no jargon.

Put a small, undeniable win on the board

Pick a people-heavy knot. For example: three teams arguing about “active customer.” Run three definitions in parallel for one quarter with clear, public rules for each. Let real usage and decision speed decide the winner. When it settles, document the outcome and who owns it. You’ll have a story, not a slide, and stories tend to win arguments.

A dual structure that actually works

• Keep your normal governance for the complicated stream (budgets, milestones, owners).
• Stand up a tiny “learning board” for the complex stream (entry/exit criteria for experiments, a small pot of money you can redeploy quickly).
• Report both streams together so leaders see predictability where it exists and uncertainty where it doesn’t.

Make the cost of business‑as‑usual visible

Plot the last big initiative: scope churn, rework, surprise dependencies. No finger‑pointing—just dates and facts. Then translate the delays into money (lost revenue, extra headcount burn, penalties). It’s blunt, but it tends to land.

A short script for your sponsor

“We’ve got two kinds of work here. The ‘build to spec’ bits we can schedule and lock. The ‘humans will react’ bits we can’t. If we treat them the same, we’ll sign up to dates we can’t meet and bury a pile of rework. Let’s timebox three small probes—with clear stop rules—alongside the planned work. You’ll get fewer nasty surprises and faster proof of what actually moves the metric. If they flop, we stop. We’ve lost weeks, not months.”

A concrete example to make it less airy

Context: self‑serve analytics kept stalling. Everyone blamed “the complexity of the tooling.” The tools were fine. The snag was meaning.

Probe: one squad, one domain (Customer Success), one metric (“churn risk”), one semantic layer. Success signals: fewer ad‑hoc requests, same‑day answers, no new spreadsheet forks. Guardrails: synthetic data in staging, canary access for 20 users, 30‑day window, pre‑approved rollback.

Outcome: adoption jumped in that corner; elsewhere stayed flat. We learned the friction lived in definitions, not dashboards. The wider roll‑out changed accordingly.

If you try just one thing

Write a one‑page pilot charter that names the uncertainty and the signals you’ll watch. Get it signed. That tiny document shifts the conversation from “promise me it will work” to “let’s find out quickly and safely.” Once people feel the reduced risk in practice, the rest of the change is, surprisingly often, straightforward.

Common traps worth avoiding

• Calling everything complex. It can become a polite way to dodge accountability. If your ingestion jobs fail nightly because of a missing PRIMARY KEY, that’s not complexity; that’s a bug.
• Thinking complexity equals sophistication. Adding eight microservices to a workflow that fits in one process doesn’t make you clever; it makes on-call harder.
• Mistaking scale for complexity. Ten times the data doesn’t automatically change the problem class. Sometimes you just need better partitioning and a bigger warehouse budget.
• Over-indexing on frameworks. Cynefin is useful language (complicated vs complex), but it won’t choose your next sprint goal.

So, is MDS complex or complicated?

The fairest answer may be: the stack is complicated; the programme is complex once people and incentives are included. Treat the pipelines, schemas and infrastructure as complicated work: plan it, automate it, test it. Treat governance, adoption and shared meaning as complex work: probe, sense, respond.

If you do only one thing before your next architecture review, ask out loud: “Are we treating a complex problem as if it’s only complicated? Or the other way around?” That tiny pause often changes the plan in useful ways.

And if you’re still unsure, try this small, unfancy exercise. Write a one-page “how it works” for your current challenge. If the page reads like a checklist with clear steps, you’re probably in complicated territory. If it reads like a story with characters, tensions and turning points, you’re dealing with complexity. Plan accordingly.