GitOps with ArgoCD: Things I Wish I Knew Before Starting

GitOps in theory: Git is your source of truth, push a change, things update automatically.

In practice: there are about twelve gotchas that will bite you in production. I learned most of them the hard way setting up ArgoCD at work.

#GitOps vs traditional deployment

The traditional approach: your CI pipeline builds an image and pushes it directly to the cluster. The pipeline needs cluster credentials and runs kubectl apply.

GitOps approach: CI builds the image, updates a Git repo with the new tag. ArgoCD watches that repo and applies changes to the cluster.

Traditional:  Code Repo → CI Build → CI Deploys → Cluster
GitOps:       Code Repo → CI Build → Updates Config Repo → ArgoCD → Cluster

The big difference: nothing outside the cluster pushes changes to it. ArgoCD pulls from Git. Your CI pipeline doesn't need cluster credentials.

#Repository structure

I tried a few different layouts. This is what stuck:

infrastructure/
├── apps/
│   ├── api-service/
│   │   ├── base/
│   │   │   ├── deployment.yml
│   │   │   ├── service.yml
│   │   │   └── kustomization.yml
│   │   └── overlays/
│   │       ├── staging/
│   │       │   ├── kustomization.yml
│   │       │   └── replicas-patch.yml
│   │       └── production/
│   │           ├── kustomization.yml
│   │           └── replicas-patch.yml
│   └── worker-service/
│       └── ...
├── argocd/
│   ├── api-service.yml
│   └── worker-service.yml
└── README.md

#ArgoCD application definition

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/infrastructure.git
    targetRevision: main
    path: apps/api-service/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m

Important settings:

• selfHeal: true: If someone manually edits a resource with kubectl edit, ArgoCD reverts it back to match Git. This is the whole point of GitOps—Git is the source of truth.
• prune: true: If you delete a manifest from Git, ArgoCD deletes the resource from the cluster. Without this, you accumulate orphaned resources forever.
• Retry with backoff: Transient failures (API server hiccups, webhook timeouts) don't permanently fail the sync.

#The image update problem

Biggest gotcha in GitOps: how do you update the image tag after CI builds it?

#Option 1: CI updates the config repo

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Build and push image
        run: |
          docker build -t registry.example.com/api:${{ github.sha }} .
          docker push registry.example.com/api:${{ github.sha }}
 
      - name: Update config repo
        run: |
          git clone https://github.com/myorg/infrastructure.git
          cd infrastructure
          kustomize edit set image \
            api=registry.example.com/api:${{ github.sha }} \
            -C apps/api-service/overlays/production
          git add .
          git commit -m "deploy: api-service ${{ github.sha }}"
          git push

This works but creates tight coupling between CI and your config repo.

#Option 2: ArgoCD Image Updater (what I use now)

ArgoCD Image Updater watches your container registry and automatically updates image tags in Git:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=registry.example.com/api
    argocd-image-updater.argoproj.io/api.update-strategy: semver
    argocd-image-updater.argoproj.io/write-back-method: git

This decouples everything. CI pushes the image, Image Updater detects it, updates Git, ArgoCD syncs. No tight coupling.

#Handling secrets

Secrets are the hardest part of GitOps. You can't commit plaintext secrets to Git, but Git is supposed to be your source of truth.

I use Sealed Secrets. You encrypt secrets locally, commit the encrypted version to Git, and the Sealed Secrets controller in the cluster decrypts them.

# Create a regular secret
kubectl create secret generic db-credentials \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml > secret.yml
 
# Seal it (encrypt for the cluster's public key)
kubeseal --format=yaml < secret.yml > sealed-secret.yml
 
# Commit the sealed version — safe to store in Git
rm secret.yml  # Never commit the plaintext version
git add sealed-secret.yml
git commit -m "chore: update db credentials"

#What can go wrong

#Sync waves and dependencies

If Service B depends on Service A, you need sync waves to make sure A deploys first:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"

Without sync waves, ArgoCD applies everything in parallel. Service B might crash because Service A isn't ready yet.

#Drift detection false positives

Some Kubernetes controllers modify resources after creation—adding default fields, mutating webhooks, etc. ArgoCD sees these as "drift" and tries to revert them. You end up in an endless sync loop.

Fix with ignoreDifferences:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # If using HPA, ignore replica count
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP  # K8s assigns this, changes on recreation

Took me a while to figure this out. Watch ArgoCD's sync status—if something keeps syncing forever, you probably need to ignore some fields.

#Things I've learned

Keep config repos separate from code repos. Makes ArgoCD's job way easier.

Enable selfHeal and prune. Without these, you're just doing Git-triggered deploys. Not really GitOps.

Use ArgoCD Image Updater instead of coupling CI to your config repo.

Figure out secrets before you go all-in on GitOps. Sealed Secrets or External Secrets Operator both work.

You will need ignoreDifferences in production. Plan for it.