Automated regression testing for Acumatica configuration-as-code (GitOps)

TLDR

Acumatica ERP configuration usually ships after manual click-through with no automated gate that rebuilds the tenant, exercises a business path, and fails on drift or a wrong outcome. With configuration-as-code for Acumatica in git, the GitOps gate is three commands: acu apply, acu run scenario/…, acu diff — each with an exit code CI can enforce. On a tenant that already has the seed applied, the path completes on the order of one minute; from an empty tenant it is a few minutes.

Manual click-through leaves indirect flows untested

Acumatica configuration ships after click-through on familiar screens, so indirect flows fail later — and Acumatica ERP still has no CI-style gate like software teams expect.

An Acumatica configuration change usually ships after click-through testing on the screens that come to mind. The flows the change touches only indirectly are rarely re-run, so the miss shows up later: a posting class that books the wrong account, an order type that stops assigning document numbers, or inventory that does not move the way the kit specification implies.

Software teams gate every code change with automated CI. Acumatica ERP configuration usually has no equivalent gate. There is no pipeline that rebuilds the tenant, exercises a business path end to end, and fails when either the configuration or the business outcome drifts.

The configuration-as-code post established desired settings as YAML in git, acu apply to write them, and acu diff to prove the live tenant matches. This post adds the missing exercise step in that GitOps loop: a transaction scenario that must pass after the seed is applied.

Build, exercise, compare as three exit codes

With Acumatica configuration-as-code in git, the GitOps regression gate is acu apply, acu run, and acu diff — three steps, three exit codes a pipeline can enforce.

The regression gate has three parts. Each maps to one command with a meaningful exit code that a pipeline can enforce.

Step	Command	Meaning
Build	`acu apply`	Seed the Acumatica tenant from YAML (idempotent upserts).
Exercise	`acu run scenario/buy-build-sell.yaml`	Run a transaction scenario (exit 1 on any miss).
Compare	`acu diff`	Prove live configuration still matches the repo (exit 2 on drift).

From the Acumatica GitOps data repo (kborovik/acumatica-gitops), with acu already pointed at a target tenant, bare acu apply and acu diff walk the seed tree (bootstrap/, baseline/, setup/) in that order:

acu apply
acu run scenario/buy-build-sell.yaml
acu diff

Any CI runner can gate on those exit codes.

Order matters for the proof. acu diff runs after acu run so a green gate means the scenario left Acumatica configuration unchanged. A scenario that dirties the config diff is a bug by definition. (You can also run acu diff right after acu apply to prove the seed alone is clean before exercising transactions.)

The buy-build-sell scenario

scenario/buy-build-sell.yaml is one end-to-end Acumatica path on the demo company: buy, assemble kits, sell, invoice, collect — then assert inventory and GL deltas.

The seed from acu apply creates a small manufacturing demo company in Acumatica: vendors, customers, stock items, and kit bills of materials. scenario/buy-build-sell.yaml is the happy path on that company. It buys parts from four vendors, receives them, assembles three kit types, sells to three customers, invoices, and collects one payment. It then asserts inventory and GL outcomes against the deltas those steps should produce.

Steps are ordered operations against Acumatica ERP. The important pattern is capture: Acumatica assigns document numbers, and capture stores them in ${var} references for later steps. A receipt can then reference the purchase order Acumatica just numbered:

- id: po-shenzhen
  put: PurchaseOrder
  record:
    VendorID: SHENZHEN
    Hold: false
    Details:
    - {InventoryID: MB-CM4, OrderQty: 25, UnitCost: 85.00}
    - {InventoryID: MOD-LTE, OrderQty: 6, UnitCost: 45.00}
  capture: {OrderNbr: po_shenzhen}

- id: rcpt-shenzhen
  put: PurchaseReceipt
  record:
    Type: Receipt
    VendorID: SHENZHEN
    Hold: false
    Details:
    - {POOrderType: Normal, POOrderNbr: "${po_shenzhen}", POLineNbr: 1, ReceiptQty: 25}
    - {POOrderType: Normal, POOrderNbr: "${po_shenzhen}", POLineNbr: 2, ReceiptQty: 6}
  capture: {ReceiptNbr: rcpt_shenzhen}

The format contract is scenario/README.md. A valid scenario parses under acu run --dry-run, and every ${var} resolves from an earlier capture.

Delta expectations, not absolute balances

Business-outcome assertions are deltas from pre/post inquiry snapshots so the scenario re-runs clean on an Acumatica tenant that already has data, without a reset.

The scenario asserts two kinds of outcome.

Document status is absolute: a sales invoice and its payment must both end Closed. Customer ACMEMFG is one of the demo customers the seed creates; the scenario sells to it, invoices, and collects.

Business balances are deltas, not absolute totals. The runner snapshots selected inquiries before step 1, re-probes after the last step, and asserts the difference. From the real file:

expect:
- get:
    entity: SalesInvoice
    keys: [Invoice, "${inv_acme}"]
  fields:
    Status: Closed
- get:
    entity: Payment
    keys: [Payment, "${pmt_acme}"]
  fields:
    Status: Closed
- inquire: InventorySummaryInquiry
  parameters: {InventoryID: MB-CM4, WarehouseID: WH01}
  delta: {QtyOnHand: 5.0}
- inquire: InventorySummaryInquiry
  parameters: {InventoryID: GW-EDGE, WarehouseID: WH01}
  delta: {QtyOnHand: 3.0}
- inquire: AccountSummaryInquiry
  parameters: {Ledger: ACTUAL, Period: "072026"}
  match: {Account: '20200'}
  delta: {EndingBalance: 3859.00}

One inventory line is easy to follow by hand: part MB-CM4 is bought at 25 units and consumed at 20 in kit assemblies, so QtyOnHand must rise by 5. Account 20200 is PO accrual; the four vendor receipts together credit it by 3859 before bills are entered. Other inventory and GL lines use the same pattern (built minus sold for kits; revenue, COGS, AR, and cash on the sales leg).

Deltas are why the scenario re-runs clean on a tenant that already holds prior runs. There is no tenant reset between runs; absolute balances would fail on the second pass.

Exit 0 means every step completed and every expectation held. Exit 1 means a step error or a missed assertion.

What the GitOps gate catches

Two failure modes, two exit codes: diff exit 2 for Acumatica configuration drift, run exit 1 for a business-outcome miss.

Two failure modes show up as different exit codes.

Configuration drift. Someone changes a field on a live Acumatica screen and never writes that value back into the configuration-as-code YAML. acu diff names the entity and field, shows source versus live, and exits 2. The pipeline fails before anyone can assume the tenant still matches the repo.

Example shape (credit terms edited in the UI, repo still net-30):

CreditTerms [NET30].DayDue00: source=30 live=45

If the live edit was accidental, acu apply writes the repo state back onto the tenant. If it was intentional, the new value is codified into YAML in a pull request (configuration-as-code post), then apply and the gate both go green.

Business-outcome miss. The configuration still matches the files, but a change breaks the path the scenario exercises. acu run fails an expect line: wrong inventory delta, wrong GL account movement, or a document stuck open. Exit 1. Close that loop by fixing the Acumatica configuration that broke the path, or by correcting an expectation that no longer matches the intended design — then re-run the gate.

What the gate costs

On an Acumatica tenant that already has the seed, apply, run, and diff complete on the order of one minute; from an empty tenant the path is a few minutes — cheap enough for every configuration-as-code change.

A warm tenant already has the seed from a prior acu apply. On that tenant, apply, run, and diff complete on the order of one minute (the buy-build-sell scenario alone has been measured around 50 seconds on the lab instance).

A virgin path starts from an empty Acumatica tenant: create it, apply the whole seed tree, run the scenario, then diff. That path is a few minutes:

acu tenant create
acu apply
acu run scenario/buy-build-sell.yaml
acu diff

Either path is cheap enough to run on every configuration-as-code change, not only before go-live.

Transactions never write configuration

Scenarios create documents only; after run, diff must still be clean, or the Acumatica config change belongs in the seed tree, not the scenario.

Scenarios create documents; they do not upsert posting classes, order types, or preferences. The GitOps gate asserts that separation: after run, diff must still be clean. If a scenario step had to change configuration to pass, that change belongs in the configuration-as-code seed directories bootstrap/ or baseline/ (see the configuration-as-code post), not in the scenario file.

Where AI enters later

This post uses one hand-written Acumatica scenario as the format contract; generating consistent linked transaction sets from plain language is a later post.

The sample scenario was written by hand once, as the format contract and the first green path. A later direction is to generate scenarios from a plain-language request: a manufacturing client, dozens of customers, months of orders, some overdue AR. The hard part is consistency (orders link to customers, invoices to orders, payments to invoices), not inventing row counts. That generation path — consistent AR/AP cycles from plain language, not just fake rows — is a later post. This post only needs one hand-written scenario that already fails the gate when the configuration or the deltas are wrong.

Acumatica GitOps as three commands

An Acumatica configuration-as-code change gets build, exercise, and compare steps with pipeline-enforceable exit codes.

That is the GitOps service claim in this series: not a slide, but files and commands a pipeline can run against Acumatica ERP. The configuration-as-code substrate is the configuration-as-code post. The instance the tenant runs on is built by kborovik/acumatica-devops (DEV/TEST environments post). The CLI surface is documented in kborovik/acumatica-cli; a dedicated CLI post is 0005.