Purview Data Discovery Methods Lab Navigator — MarcusJ Lab

Validate M365 E5 licensing, PowerShell module versions, and service connectivity. Initialize directory structure and logging infrastructure. Update global-config.json before any other lab.

• 📜
  Test-Prerequisites.ps1

  Validates M365 licensing, module versions, and service connectivity

  00-Prerequisites-Setup/scripts/Test-Prerequisites.ps1
• 📜
  Initialize-SimulationEnvironment.ps1

  Creates directory structure (logs, output, generated-documents, reports, temp) and initializes logging

  00-Prerequisites-Setup/scripts/Initialize-SimulationEnvironment.ps1

Provision SharePoint department sites (HR, Finance, Legal, Marketing, IT). Configure site templates (Communication vs Team), set permissions and ownership. Supports -SkipExisting for idempotent re-runs.

• 📜
  New-SimulatedSharePointSites.ps1

  Creates SharePoint sites from global-config.json specifications

  01-SharePoint-Site-Creation/scripts/New-SimulatedSharePointSites.ps1
• 📜
  Set-SitePermissions.ps1

  Configures site permissions and ownership

  01-SharePoint-Site-Creation/scripts/Set-SitePermissions.ps1
• 📜
  Verify-SiteCreation.ps1

  Validates that all sites were created successfully

  01-SharePoint-Site-Creation/scripts/Verify-SiteCreation.ps1
• 📜
  Show-SiteCreationReport.ps1

  Generates a report of created sites and their status

  01-SharePoint-Site-Creation/scripts/Show-SiteCreationReport.ps1
• 📜
  Show-ValidationReport.ps1

  Displays site validation results

  01-SharePoint-Site-Creation/scripts/Show-ValidationReport.ps1

Generate realistic documents with built-in Sensitive Information Types (SSN, Credit Cards, Passports, Driver's Licenses, ITIN, Bank Accounts). Mixed formats: 45% docx · 30% xlsx · 15% pdf · 10% txt. Scale: 500–1,000 (Small) · 5,000 (Medium) · 20,000 (Large).

• 📜
  Invoke-BulkDocumentGeneration.ps1

  Orchestrates bulk document generation at configured scale level

  02-Test-Data-Generation/scripts/Invoke-BulkDocumentGeneration.ps1
• 📜
  New-SimulatedHRDocuments.ps1

  Generates HR documents containing SSNs and employee PII

  02-Test-Data-Generation/scripts/New-SimulatedHRDocuments.ps1
• 📜
  New-SimulatedFinancialRecords.ps1

  Creates financial records with credit card, bank account, and ABA routing numbers

  02-Test-Data-Generation/scripts/New-SimulatedFinancialRecords.ps1
• 📜
  New-SimulatedPIIContent.ps1

  Generates identity documents (passports, driver's licenses, ITIN)

  02-Test-Data-Generation/scripts/New-SimulatedPIIContent.ps1
• 📜
  New-MixedContentDocuments.ps1

  Creates documents with multiple PII types across formats

  02-Test-Data-Generation/scripts/New-MixedContentDocuments.ps1
• 📜
  Remove-GeneratedDocuments.ps1

  Removes locally generated documents for cleanup

  02-Test-Data-Generation/scripts/Remove-GeneratedDocuments.ps1

Bulk upload documents to SharePoint with intelligent distribution (HR docs → HR site, Finance → Finance site). Applies SharePoint metadata: Department, ContentType, PIIDensity, GeneratedDate. Throttling-aware with automatic retry logic (600 req/min, 2,500 req/hr).

• 📜
  Upload-AllDocuments.ps1

  Main entry point — orchestrates bulk upload across all sites

  03-Document-Upload-Distribution/scripts/Upload-AllDocuments.ps1
• 📜
  Upload-ToSingleSite.ps1

  Uploads documents to a specific SharePoint site

  03-Document-Upload-Distribution/scripts/Upload-ToSingleSite.ps1
• 📜
  Test-UploadValidation.ps1

  Validates upload completion and document count accuracy

  03-Document-Upload-Distribution/scripts/Test-UploadValidation.ps1
• 📜
  Grant-AppSitePermissions.ps1

  Grants required permissions for upload operations

  03-Document-Upload-Distribution/scripts/Grant-AppSitePermissions.ps1

Execute On-Demand Classification via purview.microsoft.com. Review estimation (300–500 items) and cost ($5–$15 typical). Classification runs asynchronously for up to 7 days. Validates 100% Purview SIT detection accuracy in Content Explorer.

1. Navigate to purview.microsoft.com → Information Protection → Classification
2. Create On-Demand Classification scan for simulation sites
3. Review estimation results and cost analysis
4. Start classification → monitor via portal status
5. After 7 days: validate results in Content Explorer

Nine distinct discovery methods with different timelines, accuracy, and use cases. Choose based on your timing requirements. See the Discovery Methods page for the full comparison matrix and decision guide.

Consolidate discovery results from Labs 05a/05b/05c. Classify files into High/Medium/Low severity tiers. Preview deletions with dry-run (-WhatIf), then apply with explicit -Apply gate. SharePoint Recycle Bin used for reversible cleanup.

• 📜
  Aggregate-RemediationReport.ps1

  Consolidates CSV outputs from multiple discovery methods into unified remediation plan

  06-Reporting-and-Remediation/scripts/Aggregate-RemediationReport.ps1
• 📜
  Analyze-SeverityBasedRemediation.ps1

  Classifies detected files into High/Medium/Low severity tiers

  06-Reporting-and-Remediation/scripts/Analyze-SeverityBasedRemediation.ps1
• 📜
  Remove-SharePointDuplicates.ps1

  Safely removes low-risk duplicate files via SharePoint Recycle Bin

  06-Reporting-and-Remediation/scripts/Remove-SharePointDuplicates.ps1

Full environment reset: removes simulation sites, deletes local generated documents, restores global-config.json to template defaults, and exports cleanup documentation. Supports -WhatIf preview mode and confirmation prompts for safety.

• 📜
  Remove-SimulationResources.ps1

  Removes SharePoint sites and all uploaded simulation content

  07-Cleanup-Reset/scripts/Remove-SimulationResources.ps1
• 📜
  Reset-Environment.ps1

  Full environment reset including local files and config restoration

  07-Cleanup-Reset/scripts/Reset-Environment.ps1
• 📜
  Test-CleanupCompletion.ps1

  Validates that all simulation resources have been removed

  07-Cleanup-Reset/scripts/Test-CleanupCompletion.ps1
• 📜
  Export-FinalDocumentation.ps1

  Exports cleanup documentation and audit trail

  07-Cleanup-Reset/scripts/Export-FinalDocumentation.ps1

4-interval temporal tracking (24hr, 7d, 14d, 21d) to observe classification drift and evolution. Research-focused longitudinal study. Scripts: Invoke-TemporalScan.ps1, Compare-TemporalScans.ps1, Invoke-TemporalAnalysis.ps1

Compare results across all discovery methods with side-by-side accuracy, coverage, and confidence analysis.

• 📜
  Invoke-CrossLabAnalysis.ps1

  Produces comparative report across all discovery method outputs

  05-Discovery-Methods/scripts/Invoke-CrossLabAnalysis.ps1
• 📜
  Resolve-UnmappedSITGuids.ps1

  Maps unmapped SIT GUIDs to human-readable names in discovery reports

  05-Discovery-Methods/scripts/Resolve-UnmappedSITGuids.ps1
• 📜
  Expand-eDiscoveryExportPackages.ps1

  Expands and processes eDiscovery export packages for analysis

  05-Discovery-Methods/scripts/Expand-eDiscoveryExportPackages.ps1

• 🔑
  Connect-PurviewServices.ps1

  Browser-based interactive authentication to SharePoint, Exchange, and Graph

  Shared-Utilities/Connect-PurviewServices.ps1
• ⚙️
  Import-GlobalConfig.ps1

  Reads and validates global-config.json — single source of truth for all env values

  Shared-Utilities/Import-GlobalConfig.ps1
• 📦
  Import-PurviewModules.ps1

  Imports PnP.PowerShell, ExchangeOnlineManagement, Microsoft.Graph with minimum version checks

  Shared-Utilities/Import-PurviewModules.ps1
• 📝
  Write-SimulationLog.ps1

  Canonical logger — DO NOT use bare Write-Host for state logging

  Shared-Utilities/Write-SimulationLog.ps1
• 🩺
  Test-ServiceConnection.ps1

  Validates active connections to SharePoint, Exchange, and Graph services

  Shared-Utilities/Test-ServiceConnection.ps1
• 📊
  Get-SimulationProgress.ps1

  Reports current simulation state and progress across all labs

  Shared-Utilities/Get-SimulationProgress.ps1
• 🔗
  Merge-LabConfig.ps1

  Merges lab-specific config with global config for per-lab overrides

  Shared-Utilities/Merge-LabConfig.ps1
• 🗺️
  Resolve-GlobalReference.ps1

  Resolves dynamic references within configuration values

  Shared-Utilities/Resolve-GlobalReference.ps1
• 🔌
  Functions/Connection-Management.ps1

  Reusable connection management functions (retry, session validation, throttle handling)

  Shared-Utilities/Functions/Connection-Management.ps1

• 📜
  Invoke-PnPDirectFileDiscovery.ps1

  Main discovery script — PnP file enumeration + regex SIT matching across all sites

  05-Discovery-Methods/05a-PnP-Direct-File-Access/scripts/Invoke-PnPDirectFileDiscovery.ps1
• 📜
  Open-LatestReport.ps1

  Opens the most recent discovery CSV report for review

  05-Discovery-Methods/05a-PnP-Direct-File-Access/scripts/Open-LatestReport.ps1

• 📜
  Test-ContentIndexingStatus.ps1

  Checks SharePoint Search indexing readiness before eDiscovery search

  05-Discovery-Methods/05b-eDiscovery-Compliance-Search/scripts/Test-ContentIndexingStatus.ps1
• 📜
  Invoke-eDiscoveryResultsAnalysis.ps1

  Analyzes eDiscovery search results and generates summary report

  05-Discovery-Methods/05b-eDiscovery-Compliance-Search/scripts/Invoke-eDiscoveryResultsAnalysis.ps1
• 📜
  Expand-eDiscoveryExportPackages.ps1

  Expands downloaded eDiscovery export packages for analysis

  05-Discovery-Methods/05b-eDiscovery-Compliance-Search/scripts/Expand-eDiscoveryExportPackages.ps1

• 📜
  Invoke-GraphSitDiscovery.ps1

  Main Graph API discovery — automated eDiscovery via Microsoft Graph

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Invoke-GraphSitDiscovery.ps1
• 📜
  Test-GraphConnectivity.ps1

  Validates Graph API connection and required delegated permissions

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Test-GraphConnectivity.ps1
• 📜
  Grant-GraphPermissions.ps1

  Grants required delegated Graph API permissions for discovery

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Grant-GraphPermissions.ps1
• 📜
  Export-SearchResults.ps1

  Exports Graph eDiscovery search results to CSV

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Export-SearchResults.ps1
• 📜
  Get-ExistingCases.ps1

  Lists existing eDiscovery cases in the tenant

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Get-ExistingCases.ps1
• 📜
  Invoke-GraphDiscoveryAnalysis.ps1

  Analyzes Graph discovery results and generates comparison reports

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Invoke-GraphDiscoveryAnalysis.ps1
• 📜
  Compare-Discovery-Results.ps1

  Side-by-side comparison of 05a (regex) vs 05c (Graph API) results

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Compare-Discovery-Results.ps1
• 📜
  Test-SiteAvailability.ps1

  Verifies SharePoint sites are available and indexed for Graph search

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Test-SiteAvailability.ps1
• 📜
  Expand-eDiscoveryExportPackages.ps1

  Expands and processes eDiscovery export packages from Graph API

  05-Discovery-Methods/05c-Graph-API-Discovery/scripts/Expand-eDiscoveryExportPackages.ps1

• 📜
  Invoke-ActivityExplorerAnalysis.ps1

  Exports and analyzes Activity Explorer audit events for classified content

  05-Discovery-Methods/05d-Activity-Explorer-Monitoring/scripts/Invoke-ActivityExplorerAnalysis.ps1
• 📜
  Invoke-PolicySimulationAnalysis.ps1

  Analyzes auto-apply policy simulation results and enforcement statistics

  05-Discovery-Methods/05e-Auto-Apply-Policies-Discovery/scripts/Invoke-PolicySimulationAnalysis.ps1
• 📜
  Analyze-ScannerDetailedReport.ps1

  Processes and analyzes On-Premises Scanner detailed report output

  05-Discovery-Methods/05h-OnPrem-Scanner-Discovery/scripts/Analyze-ScannerDetailedReport.ps1

PnP PowerShell: 600 requests/min · 2,500 requests/hour
All upload and discovery scripts include automatic retry logic with exponential back-off.
Monitor throttling in script output — scripts will pause automatically when limits are hit.

Tier-1 orchestrator that activates exactly one persona per request. Routes work to the appropriate Tier-2 agent based on the change type. All requests should start here.

• No work without an issue from @idea-intake
• No PR opened by any path other than @artifact-resolver
• No direct push to main
• No merge without the owner-approved label and marcusjacobson's go-ahead
• No persona reviews its own PR
• No writes to .squad/memory/** outside the Scribe handoff workflow (append-only)
• All agent outputs are proposals until the owner approves