[Feature] Emergency Ingestion Module (EIM) for Volatile Volume Evacuation

[aj1126]

Here is a fully mapped out GitHub Issue Template for the Emergency Ingestion Module (EIM). It uses Document Driven Design (DDD), meaning we are defining the exact interface, technical specifications, and behavioral expectations before writing the implementation.

This issue is explicitly tailored to your existing MyDriveTools architecture—specifically referencing your high-performance concurrency experiments and Rclone handoff pipeline.

---

[Feature] Emergency Ingestion Module (EIM) for Volatile Volume Evacuation #42

📋 Overview & User Story

As a developer managing aging hardware assets, I need a lightweight, low-overhead, fault-tolerant ingestion pipeline within MyDriveTools to evacuate critical top-level data directories from a degrading local volume (D:\) to an external target volume (M:\).

Standard tree-walking and recursive file-copy methods (like Explorer or unoptimized scripts) place unsustainable read strain on failing storage media. The Emergency Ingestion Module (EIM) will prioritize minimized disk-head thrashing, non-destructive additive syncing, and highly isolated thread operations to secure assets before hardware failure occurs.

---

🛠️ Technical Specification (Document Driven Design)

1. The Failing Drive Dilemma: Concurrency vs. Thrashing

⚠️ Critical Architectural Guardrail: While MyDriveTools utilizes highly concurrent Runspace/Thread processing for cloud ingestion, high concurrency on a failing physical HDD causes head thrashing, accelerating drive failure.
Design Choice: The EIM will implement a Single-Reader, Multi-Writer architecture or a low-throttle parallel processing stream (-ThrottleLimit 2 or 3 max for reads) while allowing writing and logging threads to run concurrently at full speed.

2. Core Functional Requirements

• Zero-Purge Policy: Operations must be strictly additive. Missing files on the source drive must never trigger deletions on the target M:\ drive.
• Delta Skipping: Fast metadata lookup (Size + Modified Timestamp) to skip identical files instantly, minimizing unnecessary disk reads.
• Transient Cache Exclusions: Hardcoded bypass arrays for build assets (/obj, /Library, /.vs) to focus purely on core source code and project data assets.

---

💻 Proposed Interface Blueprint

The module will expose a clean, public cmdlet named Start-MdtEmergencyIngest following our existing naming conventions.

<#
.SYNOPSIS
    Performs a high-efficiency emergency evacuation of specific source directories to a safe destination.
.EXAMPLE
    Start-MdtEmergencyIngest -SourceDrive "D:\" -DestinationRoot "M:\" -TargetFolders @("Data", "Projects") -DryRun
#>
function Start-MdtEmergencyIngest {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory = $true)]
        [string]$SourceDrive,

        [Parameter(Mandatory = $true)]
        [string]$DestinationRoot,

        [Parameter(Mandatory = $true)]
        [string[]]$TargetFolders,

        [Parameter(Mandatory = $false)]
        [switch]$DryRun,

        [Parameter(Mandatory = $false)]
        [int]$MaxReadThreads = 2
    )
    # Architecture Implementation Goes Here
}

---

🧱 Component Breakdown & Data Flow

[ Volatile Source D:\ ] 
         │
         ▼ (Low Concurrency Read / Throttle Limit: 2)
┌────────────────────────────────────────────────────────┐
│      MyDriveTools Emergency Ingestion Engine          │
├────────────────────────────────────────────────────────┤
│  1. Pre-Flight Mount Checks (Verify D:\ and M:\)        │
│  2. Metadata Delta Evaluation (Skip matching files)   │
│  3. Exclusions Filter (Drop .vs, obj, Library)        │
└────────────────────────────────────────────────────────┘
         │
         ▼ (High Concurrency Write / Multi-Threaded 16+)
[ Secure Target M:\ ] ────► [ Rclone Cloud Sync Pipeline Pipeline Trigger ]

1. Pre-Flight Phase

• Verify both volumes are mounted. If a hardware drop occurs mid-flight, immediately break execution loops and log the failure offset to prevent writing corrupted streams.

2. Delta & Filter Phase

• Extract paths using a clean string pattern to insulate folder spaces or nested commas.
• Apply block lists to separate program configurations from actual data payloads.

3. Execution Phase

• Bind directly to the native robocopy.exe execution matrix via structured array splatting (@Args).
• Inject the /L parameter cleanly if the user passes the -DryRun switch parameter boundary.

---

🎯 Acceptance Criteria

• Data Integrity: Running a live ingest on an existing folder structure on M:\ merges data seamlessly without modifying or dropping pre-existing file configurations.
• Path Safety: Paths containing special string literals, quotes, or trailing commas do not break the argument parser boundary.
• Fault Tolerance: Unplugging the target drive M:\ mid-process causes the ingestion engine to exit cleanly within 5 seconds without locking up the host PowerShell instance.
• Framework Alignment: The cmdlet successfully runs within the MyDriveTools module structure and compiles with PSScriptAnalyzer without returning null-pointer regressions.

---

What do you think?

Does this design capture the balance between your framework's performance targets and the real-world safety constraints of that failing D:\ drive? If this blueprint looks solid, we can use it to build out the precise script module components to check into your local dev branch.