ExpertsQuest

Cyber Data Lake Onboarding Document

Cyber Data Lake Onboarding Document

Platform: Cloud-native data lake (Snowflake / Databricks-style)
Audience: Engineers onboarding sources, SOC analysts, leadership/governance review
Owner: [fill in]
Last updated: [fill in]

Section A: Data Lake Onboarding Guide

A1. Architecture Overview

  • Ingestion layer: [collectors, forwarders, cloud-native connectors, streaming vs. batch]
  • Storage layer: [raw/bronze, curated/silver, analytics/gold zones]
  • Compute layer: [warehouses/clusters used for query and detection workloads]
  • Consumption layer: [SIEM correlation, dashboards, detection engineering, ad hoc hunting]

A2. Access Model (RBAC)

Role Access Level Notes
SOC Analyst Read, curated zone No raw PHI/PII access by default
Detection Engineer Read/write, curated + detection schemas
Data Engineer Full pipeline access Owns ingestion jobs
Leadership/Governance Read, aggregated/reporting views only

Row-level security applied to any table tagged Moderate sensitivity or above (see A4).
Access requests go through: [process/ticket system]

A3. Naming Conventions

  • Database: [env]_[domain] e.g. prod_security
  • Schema: [source_category] e.g. edr, firewall, identity
  • Table: [vendor]_[log_type] e.g. crowdstrike_detections

A4. Sensitivity Classification (NIST FIPS 199-aligned, applies lake-wide)

Level FIPS 199 Basis Definition Handling
Public N/A No confidentiality, integrity, or availability impact Standard access
Low FIPS 199 Low Limited adverse impact if disclosed (indirect identifiers only) Standard access, logged
Moderate FIPS 199 Moderate Serious adverse impact if disclosed (PII: usernames, IPs tied to individuals) RBAC-restricted, masked in non-prod
High FIPS 199 High Severe or catastrophic impact if disclosed (PHI, PCI cardholder data, financial data) RBAC-restricted, encrypted at rest, audit-logged access, retention/legal hold rules apply

Overall level is set by the highest-impact security objective (confidentiality, integrity, or availability), per FIPS 199’s high-water-mark rule. Any data that also meets NARA’s CUI Registry criteria (e.g. government-origin data, export-controlled technical data) carries CUI marking and handling requirements in addition to its FIPS 199 level, per CISA/NARA CUI guidance.

A5. Retention & Lifecycle

  • Hot tier (queryable, fast): [duration]
  • Warm tier (queryable, slower/cheaper): [duration]
  • Cold/archive: [duration, storage target]
  • Regulated-Critical data retention overrides: [legal/compliance requirement]

A6. Cost & Compute Governance

  • Warehouse/cluster sizing guidelines: [fill in]
  • Query cost review cadence: [fill in]

A7. Change Process

  • New source onboarding requires: completed Section B template + Compliance sign-off on sensitivity level + Architecture review for schema fit.
  • Approval chain: [fill in]

Section B: Log Source Onboarding Template

Duplicate this block per log source. Required fields must be complete before a source goes live; field-level inventory and validation may be completed incrementally.

Source: [vendor/product name]

Required at onboarding:

  • Source category: [EDR / Firewall / Identity / Cloud / Network / Application / etc.]
  • Destination schema.table: [e.g., prod_security.edr.crowdstrike_detections]
  • Owner (team/individual): [fill in]
  • Ingestion method: [batch / streaming / API pull / syslog]
  • MITRE ATT&CK technique coverage: [list applicable techniques/tactics this source provides visibility into]
  • Sensitivity level (per A4 taxonomy): [None / Low / Moderate / High / Regulated-Critical]
  • Sensitivity justification: [which fields drive the classification, e.g. contains usernames + source IPs]

Filled in incrementally:

Field name Data type Description Sensitive?
  • Known threats / use cases this source supports: [e.g. lateral movement detection, credential abuse]
  • Log validation status: Not started / In progress / Validated
    • Validation notes: [confirm logs are generating, fields populated as expected]
  • Known gaps or limitations: [fill in]

Manager’s Post-Mortem

Residual risks:

  • Log-generation validation is deferred by design; sources can go “live” with the required fields complete but validation still Not started. Track this so it doesn’t silently become permanent.
  • Sensitivity classification quality depends on whoever fills out the template being rigorous about A4, not defaulting everything to “Low.” Consider a Compliance sign-off gate for High/Regulated-Critical claims.
  • As source count grows, Section B entries should live in a structured store (ties back to your logsource-directory tool) rather than as flat blocks on a page, or this becomes unmaintainable past ~20-30 sources.

Recommended next steps:

  • Fill in the bracketed fields in Section A with your actual environment specifics.
  • Pilot Section B with 2-3 existing log sources to stress-test whether the required fields are realistic to complete at onboarding time.
  • Decide whether Section B templates eventually migrate into the logsource-directory webapp as structured records instead of static page content.