Skip to content

Update and Rollback

This document describes the procedures for updating IQV Integration API to a new version and rolling back to a previous version if needed.

flowchart TD
  pre[Preflight: identity, SHA, ports, actor format] --> snap[Snapshot / backup]
  snap --> chk[check]
  chk --> mig[migrate forward only]
  mig --> dep[Deploy binary or compose]
  dep --> hl[health live + ready]
  hl -->|pass| ok[Write success metadata]
  hl -->|fail| rb[Restore snapshot]
  rb --> hl2[health again]
  hl2 --> fail[Non-zero exit — DB not rolled back]

Production Deployment Recommendations

Platform Recommended Method
Windows Native Windows Service
Linux systemd or Docker

Docker is a separate deployment type from native OS services. Update procedures differ by deployment method.

Update Decision Model

Updates do not decide solely from local HEAD. Three commit concepts are separated:

Concept Meaning
CurrentRepositoryCommit Local repository HEAD
TargetCommit Normal mode: origin/<Branch> after git fetch. With -SkipGitPull / --skip-git-pull: local HEAD
DeployedCommit Commit recorded in runtime deployment metadata (commit / git_commit / image tag). If metadata is missing: unknown

Decision matrix

Deployed Target Flags Action
abc abc (none) NO-OP (exit 0)
abc def (none) UPDATE
unknown def (none) UPDATE (warning: metadata missing)
def def -ForceRedeploy / --force-redeploy REDEPLOY (full pipeline)

NO-OP behavior

When DeployedCommit == TargetCommit and force-redeploy is not set, the updater exits successfully and performs none of: go mod / vet / test / build, application check, backup, migration, service stop/swap/restart, firewall changes, config copy, metadata rewrite, Docker rebuild / compose recreate / container restart.

NO-OP is a successful update result (exit code 0).

Runtime config preserve (update)

Operation Config behavior
Install Source .env → runtime config\.env (secure copy)
Update (default) Preserve existing runtime C:\IQVizyon\IQVIntegrationAPI\config\.env
Update self-copy If source path equals runtime path, Copy-Item is skipped
ForceRedeploy Forces pipeline only; does not overwrite runtime config
ReplaceRuntimeConfig Optional explicit overwrite (Windows); default is preserve

Transactional rollback

Pre-deployment failures (git/mod/vet/test/build/check): runtime untouched; no rollback.

Transaction boundary: after successful pre-update backup (immediately before service stop).

Failures after that boundary (stop, binary swap, migration, start, live, ready, metadata) trigger automatic rollback unless -NoRollback / --no-rollback:

  1. Stop service if needed
  2. Restore binary + config + deployment metadata from the specific backup path
  3. Start service
  4. Verify /health/live and /health/ready

Successful rollback prints [ROLLBACK] Previous deployment restored successfully. Update still exits with a failure code (not 0). Database migration is not rolled back (Database rollback: NOT PERFORMED).

-NoRollback prints [WARN] Automatic rollback disabled by -NoRollback. (production default: rollback enabled).

Lab-only (Windows native): -TestFailAt AfterBinarySwap|AfterServiceStop|... injects a deterministic failure to exercise rollback. Must be set explicitly; never enabled via environment variables.

Lab-only (Windows/Linux Docker): -TestFailAt AfterComposeUp / --test-fail-at AfterComposeUp. Before mutating the runtime image, update snapshots the running API image ID to iqv-integration-api:rollback-<short-id>-<timestamp> so same-tag ForceRedeploy rollback does not depend on an overwritten target tag. MongoDB is not rolled back. Successful rollback still returns the original non-zero update exit code. Windows Docker controlled rollback live status: RETEST REQUIRED.

Backup rule

A runtime backup is created only when a real deployment will proceed, and only after target differs (or force-redeploy), git sync (when needed), mod/vet/test, successful build, and application check. Backup is taken immediately before service stop so failed tests/builds never create unnecessary runtime backups.

Force redeploy

Windows: -ForceRedeploy. Linux: --force-redeploy. Prints [WARN] Force redeploy requested for already deployed commit. Not a production default.

Skip git pull and dirty tree

-SkipGitPull / --skip-git-pull sets TargetCommit = local HEAD, then still compares to DeployedCommit. Lab same-commit rebuild: combine with force-redeploy.

-SkipGitPull alone does not disable dirty-tree protection. Use -AllowDirty / --allow-dirty (legacy -Force / --force alias). When allowed: [WARN] Dirty working tree deployment explicitly allowed. and metadata may set source_dirty: true. Production normal updates use source_dirty: false.

Git fail-safe scenarios

Scenario Result
Deployed == Target NO-OP
Deployed != Target, local behind, fast-forward possible git pull --ff-only then deploy
Local HEAD == Target, Deployed older Deploy pipeline without requiring pull
Local ahead of remote Fail (no reset)
Local and remote diverged Fail (no reset)
Metadata missing Warning + treat as update required

Scripts never run git reset --hard or force checkout.

Runtime deployment metadata

Windows native: C:\IQVizyon\IQVIntegrationAPI\deployment\deployment.json

Minimum fields (no secrets): deployment_type, commit, version, build_date_utc, deployed_at_utc, source_dirty.

Windows config path standard

Canonical runtime config: C:\IQVizyon\IQVIntegrationAPI\config\.env

Incorrect flat path C:\IQVizyon\IQVIntegrationAPI\config.env must not be used or displayed.

Pre-Update Checklist

Before any update:

  • [ ] Verify the current service is running and healthy (/health/live and /health/ready return HTTP 200).
  • [ ] Review the release notes and CHANGELOG for the target version.
  • [ ] Confirm MongoDB is accessible and healthy.
  • [ ] Plan a maintenance window if the update includes database migrations.
  • [ ] Ensure you have access to the previous version archive for rollback.

MongoDB Backup

MongoDB backup is not part of the application installer's backup process. Backing up MongoDB data is a separate operational concern. Coordinate with your database operations team to perform a MongoDB backup before major updates, especially when database migrations are involved.

Update Procedures

Windows Service Update

powershell powershell.exe -ExecutionPolicy Bypass -File .\scripts\windows\service\update.ps1

The update script resolves Deployed vs Target (NO-OP when current), syncs git when needed, builds/tests, creates a backup only after successful build/check, then stops/swaps/migrates/starts/health-checks and writes deployment.json.

Linux systemd Update (production: prebuilt binary)

bash sudo ./scripts/update.sh --binary-path ./bin/iqv-integration-api

Production Linux native update does not run git pull, go build, or Go auto-install. Decision is SHA256-based: same checksum → NO-OP; --force-redeploy redeploys. The canonical runtime env (including INTEGRATION_ACTOR_USER_ID) is preserved and never randomized. After swap, read-only check verifies actor exists/active; failure enters transactional application rollback. Database rollback: NOT PERFORMED. Backups: /var/lib/iqvizyon/iqv-integration-api/backups/.

Docker Update

Windows

powershell powershell.exe -ExecutionPolicy Bypass -File .\scripts\windows\docker\update.ps1

Linux

bash sudo bash ./scripts/linux/docker/update.sh

Docker target image: iqv-integration-api:<target-short-sha>. Matching deployed commit ⇒ NO-OP unless force-redeploy.

Source-Based Updates

text git fetch origin git pull --ff-only # only when local is behind target

If local already equals target, pull may be skipped while deployment still runs when Deployed is older.

No Force Reset

The update scripts never use git reset --hard. If fast-forward is impossible (ahead/diverged/dirty without -AllowDirty), the update is aborted. Resolve the conflict manually before retrying. Never force-reset a production deployment.

Backup Structure

Windows

text C:\IQVizyon\IQVIntegrationAPI\backups\ YYYY-MM-DD_HH-MM-SS\ bin\ # Previous binary config\ # Previous configuration deployment\ # Previous deployment metadata

Linux

text /var/lib/iqvizyon/iqv-integration-api/backups/ YYYY-MM-DD_HH-MM-SS/ bin/ # Previous binary deployment/ # Previous deployment metadata

What Backups Include

Application backups include the binary, configuration metadata, and deployment state. They do not include MongoDB data. MongoDB backup is a separate operational responsibility. NO-OP updates create no backup.

Rollback Procedures

Windows Service Rollback

  1. Stop the current service: powershell Stop-Service IQVIntegrationAPI

  2. Identify the backup to restore: powershell Get-ChildItem C:\IQVizyon\IQVIntegrationAPI\backups\

  3. Replace the current binary with the backed-up version: powershell Copy-Item C:\IQVizyon\IQVIntegrationAPI\backups\<timestamp>\bin\* C:\IQVizyon\IQVIntegrationAPI\bin\ -Force

  4. Start the service: powershell Start-Service IQVIntegrationAPI

  5. Verify health: powershell Invoke-RestMethod http://localhost:8080/health/live Invoke-RestMethod http://localhost:8080/health/ready

Linux systemd Rollback

  1. Stop the current service: bash sudo systemctl stop iqv-integration-api.service

  2. Identify the backup to restore: bash ls /var/lib/iqvizyon/iqv-integration-api/backups/

  3. Replace the current binary with the backed-up version: bash sudo cp /var/lib/iqvizyon/iqv-integration-api/backups/<timestamp>/bin/* /opt/iqvizyon/iqv-integration-api/bin/

  4. Start the service: bash sudo systemctl start iqv-integration-api.service

  5. Verify health: bash curl -s http://localhost:8080/health/live curl -s http://localhost:8080/health/ready

Linux Docker Rollback (offline production)

Do not git checkout or docker compose up --build on the customer host.

update.sh snapshots the running API image (iqv-integration-api:rollback-<id>-<timestamp>) before mutation. If live/ready fails:

  1. Previous immutable image is restored.
  2. Compose is brought up with --no-build.
  3. Failed-version metadata is not written.
  4. The original non-zero update exit code is returned.
  5. Database rollback: NOT PERFORMED. External MongoDB is not modified.

Same loaded image digest/ID without --force-redeploy is a NO-OP (no container restart).

Optional auto-update reuses this same update.sh path after a verified artifact download. Auto-update is not git pull.

Windows / lab Docker Rollback

Windows Docker continues to use its existing snapshot + -TestFailAt AfterComposeUp path (source-tree / build allowed on the lab machine). Live status: RETEST REQUIRED.

Migration Failure Recovery

If a database migration fails (exit code 14):

  1. Do not retry the update automatically.
  2. Inspect the migration logs for the exact failure.
  3. Determine if the migration was partially applied.
  4. If partially applied, consult the release notes for manual remediation steps.
  5. Roll back the binary to the previous version.
  6. Contact the development team if the migration cannot be safely reversed.

Data Integrity

Never truncate, drop, or force-modify collections to work around a migration failure. Failed migrations may leave data in an intermediate state that requires careful manual resolution.

MongoDB Backup (Separate Operational Concern)

MongoDB backup is not managed by the application installer or update scripts. It is the responsibility of your database operations team.

Recommended practices:

  • Perform a MongoDB backup before any update that includes database migrations.
  • Use mongodump or your organization's standard backup tool.
  • Store backups in a secure location separate from the application server.
  • Test backup restoration periodically.
  • Document your MongoDB backup schedule and retention policy.

bash mongodump --uri="mongodb://localhost:27017/iqv_integration" --out=/backup/path/

Warning

Never delete MongoDB data during an application uninstall. The uninstaller does not touch MongoDB — keep it that way.

Health Check Requirements

Both liveness and readiness probes must pass before an update is considered successful:

Endpoint Purpose
GET /health/live Process is alive
GET /health/ready Process is alive and MongoDB is reachable

If either probe fails after an update, the update has not succeeded. Investigate the failure or roll back.

Windows E2E Acceptance Matrix (lab)

Evidence: Windows lab, August 2026. Full narrative: Windows Service E2E.

Scenario Status
Clean Install PASS
Reboot Auto Start PASS
NO-OP Update PASS
ForceRedeploy PASS
Config Preserve Update PASS
Automatic Update Rollback PASS
Rollback Exit Code PASS (failed update + successful rollback → exit 11, not 0)
Status PASS
Uninstall Preserve Runtime PASS
Preserved Runtime Reinstall PASS
ServiceRepair / Existing Listener Ownership PASS
Firewall idempotency PASS
Step counter [1/29]…[29/29] PASS

Linux systemd: offline deployment / reboot / NO-OP LIVE PASS; inbound POST BLOCKED / RETEST REQUIRED. Docker: IMPLEMENTED / NOT E2E VALIDATED.

Exit Codes Reference

Code Meaning
0 Success
1 General error
2 Configuration error
3 MongoDB connection failure
4 MongoDB authentication failure
5 MongoDB timeout
6 Port already in use
7 TLS certificate error
8 Permission denied
9 Binary not found
10 Service registration failure
11 Health check failure
12 Backup failure
13 Restore failure
14 Migration failure
15 Dirty repository
16 Network error
17 Unknown error

Uninstall Reminder

When uninstalling:

  • The uninstaller removes the application binary, service registration, and application directories.
  • The uninstaller never deletes MongoDB data.
  • The .env file may or may not be removed depending on the uninstall script — always keep a secure copy of your configuration.
  • After uninstall, MongoDB continues to run independently.