Pipeline stages and caching

A CI pipeline is a series of checks, and each one costs a different amount of time and money. The order matters. Put cheap checks first, so a bad commit fails early. Run independent jobs at the same time. Cache what has not changed since the last run. Those three levers decide whether the pipeline takes five minutes or forty.

Walking through the stages

A practical default path looks like this.

Lint runs first because it costs seconds. A commit with a type error never spends minutes on Docker builds.

Caching a real pipeline

A Node service takes 14 minutes on a cold run. npm ci takes 8 minutes, the tests take 3 and the Docker build takes 3.

Key the npm cache on the package-lock.json hash, and key the Docker build on layer digests. After caching, npm ci drops to about 40 seconds, and the image rebuild reuses its base layers. The wall-clock time falls to roughly 5 to 6 minutes. That gap is the difference between waiting for CI and merging before standup.

Lint, unit tests and a dependency audit usually do not depend on each other. Run them in parallel after one shared checkout step. Parallelism is free latency reduction.

Integration tests need honest dependencies

Use a real, short-lived Postgres or Redis for each job, through Testcontainers or Compose services. Avoid a shared "dev" database, because other pull requests will change it under you. Contract tests, such as Pact or an OpenAPI diff, catch a consumer and provider drifting apart before deploy.

Common mistakes

• Caching with no key. A cache that leftover files from another job fill is unreliable.
• Running the slow stages first. A commit with a lint error still spends minutes on integration tests.
• Sharing one database across pipelines. Tests become flaky, because the data depends on who ran last.

Check yourself

1. Why run lint before integration tests? Answer: Cheap checks fail fast, so expensive stages do not run on a commit that would have failed anyway.
2. What should a dependency cache be keyed on? Answer: The lockfile hash, so the cache changes exactly when the dependencies do.

Key takeaways

• Order stages from cheap to expensive, so bad commits fail early.
• Run independent jobs in parallel after a shared checkout.
• Cache by lockfile hash and by layer digest, and invalidate on change.
• Use ephemeral dependencies for integration tests.

Further learning

Next, see how large repositories avoid running every job on every commit: monorepos and affected builds.