Auto-add missing entity.json keys as null; null now means ignore, not error

On each run the engine compares entity.json against the full set of known
keys, adds any that are absent with a null value, and writes the file back -
so nobody has to hand-type key names (the typo class of config error goes
away). A null/blank value now means "this entity doesn't use this": files
that would need it are ignored with a yellow notice and left in place,
instead of counting as failures. Insurance-unconfigured no longer skips the
patient step for that practice (was continue, now else-branch).

README.md

@@ -129,11 +129,13 @@ leave its login and every other path alone.
 folder name — so one `entity.json` can serve every practice as long as they
 all follow the same folder-naming pattern on the server.
 
-**Every "Yes" field must be present in `entity.json`** unless *every single
+**Missing keys fix themselves:** on every run, the tool checks `entity.json`
-practice* under the entity supplies it in its own `practice.json`. If a
+for any keys it knows about that aren't in the file, adds them with a value
-required field is missing or blank, the run stops that practice with a
+of `null`, and saves the file — so you never have to type key names by hand.
-`CONFIG ERROR` message naming exactly which fields it couldn't find — nothing
+A `null` value means **"we don't use this — ignore it"**: any files that
-uploads and nothing archives for that practice until it's fixed.
+would need that setting are left in place untouched (with a yellow notice in
+the output), not treated as failures. To turn a feature on, just open
+`entity.json` and replace the `null` with a real value in quotes.
 
 **Editing JSON — the rules that bite:**
 - Key names must match **exactly** (all lowercase, underscores): `host`, not
@@ -207,16 +209,17 @@ those are entity-wide only.
 - **Files failed to upload** — they stay in the date folder. Fix the issue and run again.
 - **A date folder is still there after running** — one or more files failed. Check the output window.
 - **WinSCP error on startup** — run `setup.bat` again to re-download WinSCP.
-- **`CONFIG ERROR: entity.json is missing or has blank: ...`** — the fields it
+- **`... upload not configured (ftps.xyz is null/blank) - N file(s) ignored`** —
-  lists couldn't be found in that entity's `entity.json` (or the practice's
+  there are files to send, but the listed settings are still `null` (or blank)
-  `practice.json`). Open the file and check those exact key names against the
+  in that entity's `entity.json`. If those files *should* upload, open
-  example above — a typo in the key name counts as missing.
+  `entity.json` and fill in real values for the listed keys. If that upload
+  type genuinely doesn't apply to this entity, the message is normal and the
+  files just stay where they are.
+- **`Added missing key(s) to entity.json ...`** — informational, not an error.
+  The tool added keys that were absent (as `null`) so they're ready to fill in.
 - **`No .pdf files in <date>\` but the folder isn't empty** — the files in the
-  date folder are probably TIFs. Add `"insurance_file_type": "tif",` at the top
+  date folder are probably TIFs. Set `"insurance_file_type": "tif",` at the top
   level of that entity's `entity.json` (right under `workflow`).
-- **Every field shows missing at once** — the section name itself is probably
-  wrong (must be exactly `ftps` / `sftp`), or the file's structure got damaged
-  while editing. Compare the overall shape against `Entities\EXAMPLE\entity.json`.
 - **The startup banner shows the wrong file type** — the banner line
   (`Entity: ... | File type: .pdf | ...`) shows what was actually read from
   `entity.json`, so it's the quickest way to confirm your edit took effect.

engine.ps1

@@ -76,6 +76,39 @@ if (-not (Test-Path $entityConfigFile)) {
 }
 
 $entityConfig = Get-Content $entityConfigFile -Raw | ConvertFrom-Json
+
+# Add any missing keys to entity.json as null so they never have to be typed
+# by hand (a null value means "not used - ignore"). Writes the file back only
+# when something was actually added.
+$configTemplate = [ordered]@{
+    workflow            = $null
+    insurance_file_type = $null
+    ftps = [ordered]@{ host = $null; port = $null; tls = $null; username = $null; password = $null; tiff_path = $null; pdf_path = $null }
+    sftp = [ordered]@{ host = $null; port = $null; username = $null; password = $null; patient_path = $null }
+}
+$addedKeys = @()
+foreach ($topKey in $configTemplate.Keys) {
+    if ($null -eq $entityConfig.PSObject.Properties[$topKey]) {
+        $value = if ($configTemplate[$topKey] -is [System.Collections.Specialized.OrderedDictionary]) {
+            [PSCustomObject]$configTemplate[$topKey]
+        } else { $configTemplate[$topKey] }
+        $entityConfig | Add-Member -NotePropertyName $topKey -NotePropertyValue $value
+        $addedKeys += $topKey
+    } elseif ($configTemplate[$topKey] -is [System.Collections.Specialized.OrderedDictionary]) {
+        foreach ($subKey in $configTemplate[$topKey].Keys) {
+            if ($null -eq $entityConfig.$topKey.PSObject.Properties[$subKey]) {
+                $entityConfig.$topKey | Add-Member -NotePropertyName $subKey -NotePropertyValue $null
+                $addedKeys += "$topKey.$subKey"
+            }
+        }
+    }
+}
+if ($addedKeys.Count -gt 0) {
+    $entityConfig | ConvertTo-Json -Depth 10 | Set-Content -Path $entityConfigFile -Encoding UTF8
+    Write-Host "Added missing key(s) to entity.json (as null = ignored): $($addedKeys -join ', ')" -ForegroundColor Yellow
+    Write-Info "CONFIG added missing keys to entity.json: $($addedKeys -join ',')"
+}
+
 $workflow     = $entityConfig.workflow
 $insuranceExt = if ($entityConfig.insurance_file_type) { $entityConfig.insurance_file_type.TrimStart(".") } else { "pdf" }
 $export1      = Join-Path $EntityDir "Export1"
@@ -269,14 +302,9 @@ foreach ($practice in $practices) {
 
         $missing = Get-MissingFields $config.ftps "ftps" @("host", "username", "password", "tiff_path", "pdf_path")
         if (($insFiles.Count -gt 0 -or $ptStmts.Count -gt 0) -and $missing.Count -gt 0) {
-            Write-Host "    CONFIG ERROR: entity.json is missing or has blank: $($missing -join ', ')" -ForegroundColor Red
+            Write-Host "    Insurance upload not configured ($($missing -join ', ') is null/blank) - $($insFiles.Count + $ptStmts.Count) file(s) ignored, left in place." -ForegroundColor Yellow
-            Write-Host "    Check spelling of those keys in entity.json (and practice.json if present)." -ForegroundColor Yellow
+            Write-Info "IGNORE entity=$entityName practice=$pracName reason=unconfigured fields=$($missing -join ',')"
-            Write-Log "error" "CONFIG entity=$entityName practice=$pracName missing=$($missing -join ',')"
+        } else {
-            $totalFail += $insFiles.Count + $ptStmts.Count
-            Write-Host "    $today\ left for retry." -ForegroundColor Yellow
-            continue
-        }
-
             if ($insFiles.Count -gt 0) {
                 Write-Host "    Insurance ($($insFiles.Count)) -> $insPath" -ForegroundColor White
                 $r = Invoke-FTPSUpload $config.ftps $insFiles $insPath $entityName $pracName
@@ -303,6 +331,7 @@ foreach ($practice in $practices) {
 
             $totalOk += $insResult.Ok; $totalFail += $insResult.Fail
         }
+    }
 
     # -- Patient files -> SFTP -------------------------------------------------
     if ($workflow -eq "insurance+patient") {
@@ -326,11 +355,8 @@ foreach ($practice in $practices) {
                     Write-Host "    No txt/tif files in $($ptFolder.Name)\ - skipping." -ForegroundColor Gray
                     Write-Info "SKIP entity=$entityName practice=$pracName reason=no txt/tif files"
                 } elseif (($sftpMissing = Get-MissingFields $config.sftp "sftp" @("host", "username", "password", "patient_path")).Count -gt 0) {
-                    Write-Host "    CONFIG ERROR: entity.json is missing or has blank: $($sftpMissing -join ', ')" -ForegroundColor Red
+                    Write-Host "    Patient upload not configured ($($sftpMissing -join ', ') is null/blank) - $($patFiles.Count) file(s) ignored, left in place." -ForegroundColor Yellow
-                    Write-Host "    Check spelling of those keys in entity.json (and practice.json if present)." -ForegroundColor Yellow
+                    Write-Info "IGNORE entity=$entityName practice=$pracName reason=unconfigured fields=$($sftpMissing -join ',')"
-                    Write-Log "error" "CONFIG entity=$entityName practice=$pracName missing=$($sftpMissing -join ',')"
-                    $totalFail += $patFiles.Count
-                    Write-Host "    $($ptFolder.Name)\ left for retry." -ForegroundColor Yellow
                 } else {
                     $patPath = Expand-Path $config.sftp.patient_path $pracName
                     Write-Host "    Patient ($($patFiles.Count)) -> $patPath" -ForegroundColor White