Privacy ingestion methods

Open Privacy›Data map, click Sources, then Connect, and choose Connect GitHub. On an empty map the button is Connect data source.

Click Connect a GitHub account, then install or authorize the privacy GitHub App in GitHub and pick the repositories it may read. Your GitHub organization may require an owner to approve the installation.

Back in Noru, select the repositories and click Connect selected.

Expand each repository to check its branch. Scans use the saved branch; change it and click Save branch before scanning.

Click Scan now. When the status reads Scanned, the systems and datasets appear in the data map. Scan again whenever you want to refresh them.

Generate and review the map

Run the plugin locally, ideally with an agent to explain how the code uses each field. Someone on your team reviews every classification and purpose and records the decision. The plugin writes three files you commit next to the code:

Fields you confirm as non-personal stay in the reviewed manifest and the baseline but are left out of the export. The local walkthrough shows the commands.

Store the API key as a CI secret

In the Noru organization that should receive the map, an admin creates an API key with the write:datamaps scope. The key decides the organization; the slug decides which source inside it.

The publisher reads these variables:

The publisher never reads CI-provider variables on its own, so map your provider's branch and commit to the last two, as the examples below do.

Require the privacy check before merge

The check runs offline and needs no credentials. It fails when the code and the reviewed map disagree: unknown taxonomy values, unresolved or expired reviews, schema drift, changed processing evidence, or an export that no longer matches the accepted map.

python3 "$TOOLKIT/plugins/privacy-datamap/scripts/check_privacy.py" \
  --repo="$REPOSITORY"

The check only watches processing evidence that is registered in the baseline; anything outside that is not monitored.

Publish after merge

On the default branch, run the check again and publish only if it passes. Keep the API key in this job alone, run one publication per source at a time, and stop older runs from publishing after newer ones.

.privacy-toolkit:
  # Use the Python minor version the baseline was sealed with.
  image: python:3.12
  variables:
    TOOLKIT: /tmp/noru-grc-engineering
  before_script:
    - apt-get update -qq && apt-get install -y -qq nodejs > /dev/null
    - >-
      git clone --depth 1 --branch v0.11.0
      https://github.com/noru-tech/noru-grc-engineering.git "$TOOLKIT"

privacy-check:
  extends: .privacy-toolkit
  stage: test
  script:
    - >-
      python3 "$TOOLKIT/plugins/privacy-datamap/scripts/check_privacy.py"
      --repo="$CI_PROJECT_DIR"

privacy-publish:
  extends: .privacy-toolkit
  stage: deploy
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  resource_group: privacy-datamap # one publication at a time
  variables:
    NORU_API_BASE_URL: https://api.noru.tech
    NORU_SOURCE_SLUG: $CI_PROJECT_PATH
    NORU_SOURCE_BRANCH: $CI_COMMIT_BRANCH
    NORU_SOURCE_COMMIT_SHA: $CI_COMMIT_SHA
  script:
    # NORU_API_KEY comes from the masked, protected CI/CD variable.
    - >-
      python3 "$TOOLKIT/plugins/privacy-datamap/scripts/check_privacy.py"
      --repo="$CI_PROJECT_DIR"
    - >-
      python3 "$TOOLKIT/plugins/privacy-datamap/scripts/publish_datamap.py"
      "$CI_PROJECT_DIR/.fides/datamap.yml" --publish

Collect and reconcile

node "$scripts/collect.mjs" --repo="$REPOSITORY" --output=json
python3 "$scripts/reconcile.py" --repo="$REPOSITORY" --output=json

On a first run this creates .noru/privacy-datamap.yml with every field flagged for review. On later runs, proposed changes land in .noru/.cache/ and accepted decisions are never overwritten.

Review the proposed map

Follow the privacy-datamap review workflow: look at how the code uses each field, settle its classification and purpose, and record a named, dated review with an expiry. Resolve any datastore-boundary, discovery, or evidence questions before you accept.

Validate, seal, and export

python3 "$scripts/validate_manifest.py" \
  "$REPOSITORY/.noru/privacy-datamap.yml" \
  --emit-parsed="$REPOSITORY/.noru/.cache/privacy-datamap.parsed.json" && \
python3 "$scripts/reconcile.py" --repo="$REPOSITORY" --seal --output=json && \
node "$scripts/collect.mjs" --repo="$REPOSITORY" --output=json

This seals the accepted baseline and writes .fides/datamap.yml. Commit the three files with the code changes they describe, then run the privacy check once more:

python3 "$scripts/check_privacy.py" --repo="$REPOSITORY"

Build the request body

The API takes JSON, so the YAML export is parsed with the toolkit's own reader and wrapped with the slug and provenance. The result goes to the ignored cache folder and never contains the API key.

export NORU_SOURCE_SLUG=customer-service   # stable, unique per source
python3 - <<'PY'
import json, os, subprocess, sys
from pathlib import Path

repo = Path(os.environ["REPOSITORY"]).expanduser().resolve()
sys.path.insert(0, str(Path(os.environ["TOOLKIT"]).expanduser() / "plugins/privacy-datamap/scripts"))
from publish_datamap import read_manifest

def git(*args):
    return subprocess.check_output(["git", "-C", str(repo), *args], text=True).strip()

payload = {
    "slug": os.environ["NORU_SOURCE_SLUG"],
    "manifest": read_manifest(repo / ".fides/datamap.yml"),
    "commitSha": git("rev-parse", "HEAD"),
}
if branch := git("branch", "--show-current"):
    payload["branch"] = branch

out = repo / ".noru/.cache/datamap-request.json"
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(payload, allow_nan=False), encoding="utf-8")
print(out)
PY

Keep .noru/.cache/ out of version control.

Send it

Load the key from your password manager or a local secret store rather than typing it into a script you might commit.

curl --fail-with-body --silent --show-error --max-time 60 \
  --request POST "https://api.noru.tech/v1/privacy/datamaps" \
  --header "Authorization: Bearer $NORU_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Accept: application/json" \
  --data-binary "@$REPOSITORY/.noru/.cache/datamap-request.json"

A successful response looks like this:

{
  "data": {
    "dataSourceId": "NORU-PSRC-1",
    "unchanged": false,
    "counts": { "systems": 2, "datasets": 3, "activities": 4 },
    "warnings": []
  }
}

The response also carries the new version's id and a change summary. curl fails on HTTP errors but not on warnings, so read them yourself. If the request times out, check the source in Noru before retrying: the write may already have completed.