What Is Kubernetes? – Definition and How It Works

Learn what Kubernetes is, which objects matter, and how to run scraping and browser workloads on it sensibly.

Kubernetes is a container orchestrator. You declare the desired state of a workload and the cluster continuously reconciles reality toward it — scheduling containers, restarting failures, and scaling replicas.

1. The Declarative Model

You do not tell Kubernetes to start a container. You declare that three replicas should exist, and a controller makes it so — including after a node dies.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crawler
spec:
  replicas: 3
  selector:
    matchLabels: { app: crawler }
  template:
    metadata:
      labels: { app: crawler }
    spec:
      containers:
        - name: crawler
          image: registry.example.com/crawler:1.4.2
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }

2. The Objects That Matter

For collection work, Job and CronJob matter more than Deployment. A scheduled collection run is a Job that finishes, not a service that stays up.

3. Scheduled Collection as a CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-collect
spec:
  schedule: "17 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 7200
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: collect
              image: registry.example.com/collector:1.4.2

Three settings do the real work. concurrencyPolicy: Forbid stops a slow run overlapping the next — which is what causes duplicate collection. activeDeadlineSeconds kills a hung run instead of letting it occupy a slot forever. backoffLimit bounds retries so a broken job does not retry all night.

4. Browser Workloads Need Specific Care

Headless browsers on Kubernetes fail in predictable ways:

Shared memory. The default /dev/shm is small, and Chrome crashes when it runs out. Mount a larger one:

volumes:
  - name: dshm
    emptyDir: { medium: Memory, sizeLimit: 1Gi }

Memory limits. A browser that exceeds its limit is OOM-killed abruptly, which looks like a random crash. Set limits from measurement, not guesswork.

Zombie processes. Browsers spawn children; without an init process they accumulate. Set shareProcessNamespace or use an init such as tini as PID 1.

Liveness probes. A probe that only checks the process misses a wedged browser. Probe something that requires the browser to actually work.

5. Secrets

Credentials belong in Secret objects, referenced rather than baked into an image:

env:
  - name: API_TOKEN
    valueFrom:
      secretKeyRef: { name: collector-creds, key: api-token }

Note that a Kubernetes Secret is base64-encoded, not encrypted, by default. Treat it as an injection mechanism rather than a secure store, and keep the canonical value in a real secret manager that syncs into the cluster.

6. Where Kubernetes Helps and Where It Does Not

Helps: many concurrent workers, geographic distribution across clusters, resource isolation between jobs, automatic restart, and a uniform deployment path.

Does not help: a single scheduled script, which a plain cron entry runs with none of the operational surface. Kubernetes adds real complexity — networking, RBAC, storage classes, upgrades — and that cost is only worth paying when the scale or the isolation justifies it.

The honest test: if one machine running cron would do, use it.

7. Practical Notes

• Set both requests and limits. Without requests the scheduler cannot place pods sensibly; without limits one pod starves its neighbours.
• Use Jobs for finite work, not Deployments with a sleep loop.
• Log to stdout, so cluster log collection works.
• Make the workload idempotent, because a pod can be killed and restarted at any moment.
• Expect eviction. Nodes are drained for upgrades; a job that cannot survive a restart will not survive production.

8. Summary

Kubernetes reconciles declared desired state for containerised workloads, with Jobs and CronJobs the relevant objects for scheduled collection rather than long-running Deployments.

Use concurrencyPolicy: Forbid, an active deadline, and a bounded backoff for scheduled runs; give browser pods a larger /dev/shm, measured memory limits, and a real init process — and skip Kubernetes entirely when one machine and cron would do the job.

Potential Related Glossary

• Headless Browser
• Concurrency vs Parallelism
• Web Crawling
• IP Rotation