Technical note verified Sep 22, 2026

Private Encrypted Artifact Storage and Scoped Retrieval Foundation

KESPA implemented a private upload and artifact foundation with encrypted originals outside the public web root, chat/project/account scoping, owner-scoped metadata and indexing structures, and runtime-storage exclusion from Git. A live artifact was successfully stored as an encrypted .kespa blob with 600 permissions after two production upload failures were diagnosed and corrected; semantic cross-chat retrieval and enrichment remain outside the verified boundary of this checkpoint.

private-artifacts encrypted-storage scope-aware-retrieval privacy uploads provenance

MARKDOWN

README.md

6,978 bytes SHA-256 85722dbfc7de443f…

KESPA-NOTE-025 — Private Encrypted Artifact Storage and Scoped Retrieval Foundation

Type: Technical Note Date: 2026-09-22 Status: verified

Summary

KESPA implemented a private artifact foundation for user-uploaded files and images. The architecture validates uploads, encrypts original artifacts at rest, stores them outside the public web root, records owner-scoped metadata/index data, and associates artifacts with chat, project, or account scope so later retrieval can respect where the user intended the artifact to be available.

A live production upload was successfully stored as an encrypted .kespa blob after two independent failures were diagnosed and corrected. The stored artifact was recorded as owned by the PHP service account and permissioned 600. Runtime storage was also removed from Git tracking so uploaded artifacts, logs, caches, backups, and other private state remain outside source control.

This note verifies the private storage and scoping foundation. It does not claim that semantic cross-chat artifact recall, multimodal enrichment, or every supported file extractor has been fully validated end to end.

Technical Goal

The artifact system is intended to preserve private user material as evidence that can be retrieved later without collapsing files into ordinary global memory.

The recorded workflow is:

``text Upload → validate → encrypt → store outside public web root → extract / OCR where available → index → apply scope → retrieve later under owner/scope controls ``

This creates a distinct boundary between:

conversational context,
durable user memory,
public Trusted Knowledge,
private user artifacts.

Scope Model

KESPA defines three artifact scopes:

| Scope | Intended availability | |---|---| | Chat | Available to the conversation that owns the artifact. | | Project | Available within the associated project workspace. | | Account | Available to later conversations for the same user when retrieval policy selects it. |

The scope model is designed to prevent a chat-only file from becoming globally available simply because it was uploaded once.

Private Storage Boundary

The configured private artifact root is outside the public web root:

``text /var/www/kespa/storage/private_artifacts ``

Artifact binaries are stored as encrypted .kespa blobs using per-user UUID-based paths. The public artifact intentionally omits the specific test user's identifier and stored artifact UUID.

The recorded live storage checkpoint established that the successfully stored artifact was:

encrypted at rest,
outside the public web root,
owned by the PHP service account,
permissioned 600.

Metadata and Retrieval Index Boundary

The supplied implementation records separate responsibilities for artifact metadata, chunk/index information, search terms, and message associations. Explicitly named structures include:

user_artifacts
user_artifact_chunks
user_artifact_terms
message_artifacts

Additional multimodal/enrichment structures exist in the later integration work, but they are outside the verified scope of this note.

The design allows extracted/indexed representations to support later retrieval while the original binary remains encrypted.

Supported Input Coverage

The current upload path is intended to handle common text, data, code/configuration, document, Office, and image formats, including:

TXT / Markdown
CSV / JSON
code/configuration files
PDF
DOCX
XLSX
PPTX
JPG / PNG / WebP / GIF

Extraction quality depends on the host tools and libraries available for a particular format. The source documentation specifically identifies pdftotext, tesseract, and PHP ZIP support as useful host capabilities.

Production Failure and Correction Record

Two independent upload failures were recorded during live bring-up.

1. Private storage was not writable

PHP-FPM could not initially create the private artifact storage path.

Correction:

create the required storage directory,
grant the PHP service account the required write access,
verify the service account could write to the private path.

2. Numeric lexical terms triggered a strict type error

Numeric-string array keys were converted by PHP into integers before a term-hashing function expecting a string was called.

Correction:

explicitly cast search terms to string before hashing.

After these corrections, the final test artifact was successfully stored.

Git / Runtime Separation

Runtime storage is excluded from Git using the repository storage ignore boundary:

``text /storage/* ``

Two historical backup JSON files that had been tracked before the ignore rule existed were removed from the Git index. The recorded follow-up git ls-files storage returned no tracked storage content.

This keeps private/runtime material such as uploads, logs, backups, caches, synchronized research runtime data, and enrichment logs out of source control.

Security Boundary

The current design records the following non-negotiable artifact rules:

originals remain encrypted at rest,
binaries stay outside the public web root,
every read/search/download/delete path must enforce ownership and scope,
MIME/type/size validation is server-side,
upload/plan limits are enforced from server-side database policy rather than browser state,
private runtime storage is excluded from Git.

No keys, encryption material, private file contents, user identifiers, or private artifact UUIDs are published in this package.

Validation Boundary

Directly demonstrated or explicitly recorded as complete at this checkpoint:

private artifact architecture implemented,
chat/project/account scoping defined,
encrypted artifact storage outside the public web root,
a live artifact physically stored after production fixes,
stored artifact permission mode recorded as 600,
four core metadata/index tables explicitly identified,
two upload failures diagnosed and corrected,
runtime storage/ content removed from Git tracking.

Not yet claimed verified end to end:

semantic cross-chat account-scoped retrieval quality,
embedding-based retrieval,
multimodal/vision enrichment,
background enrichment worker under production cron,
every Office/PDF/image extractor path,
artifact deduplication/versioning,
cold/object-storage migration,
backup/restore and retention lifecycle.

Next Technical Direction

The recorded roadmap moves from secure storage toward durable artifact intelligence:

``text encrypted private artifact → extraction / OCR → structured metadata → chunks + embeddings → lexical + semantic + metadata retrieval → reranking → provenance back to the original artifact ``

The target user experience is to locate a previously uploaded artifact from the correct private scope and answer from that evidence rather than relying on vague conversational memory.

TEXT

SHA256SUMS.txt

393 bytes SHA-256 05a8d98347b441f3…

85722dbfc7de443fdbe1a7075927108ca16681edb53e010e51439ae70c906387  README.md
78e395e8184914fdce6b97ccb6be80497f63c6c866a79472ef854bac6b8bb8de  note.json
dab767861a1c9b4823b0c0cded21f56c26ddafdcf1010144d13f3a4abe0fe525  metrics.csv
1428be8fa5ab09f7bb193c103c0cf3269e83bde58c8769ee3d01b4f9390ab269  methodology.md
60cb450c56c7838fd8e2ee17ebb1d3fd37edb21175c379acd482a8b039e3a16b  provenance.json

MARKDOWN

methodology.md

2,983 bytes SHA-256 1428be8fa5ab09f7…

Methodology

Objective

Document the first production-capable private artifact-storage boundary in KESPA without overstating later semantic or multimodal capabilities.

Source review

This note was built from three user-supplied September 22, 2026 project documents:

1. the detailed KESPA Jira/project update,

2. the current-state README addendum,

3. the future-plan document.

The existing public research manifest was also used to allocate the next technical-note ID and to check for overlap with prior archive entries.

Deduplication decision

This artifact was kept separate because the existing archive documents public Trusted Knowledge indexing, user/community telemetry, research automation, and freshness/reverification, but does not document the private user-artifact storage boundary.

Routine chat UI work, account-menu cleanup, and other ordinary product changes from the same development batch were not promoted into separate public artifacts.

Evidence classification

Recorded completed state

The source directly records that:

the private artifact architecture is implemented,
artifacts support chat/project/account scoping,
original artifacts are encrypted and stored outside the public web root,
a live artifact was successfully stored after two production failures were fixed,
the successful artifact had permission mode 600,
core metadata/index structures are present,
runtime storage was removed from Git tracking.

Derived values

Only simple counts were derived:

3 scope classes from chat/project/account,
4 explicitly named core artifact tables,
2 enumerated upload failures.

No performance, capacity, encryption-strength, retrieval-quality, OCR-quality, or latency values were inferred.

Design / future work

The supplied sources describe OCR, embeddings, semantic retrieval, multimodal enrichment, deduplication, versioning, and storage abstraction as implemented direction or future work. Where production end-to-end validation was explicitly still open, this note keeps those capabilities outside the verified boundary.

Privacy and publication method

The public package intentionally excludes:

the test user's identifier,
the artifact UUID and exact user-specific artifact path,
uploaded file contents,
encryption keys or secret material,
private application source code,
credentials and provider secrets.

Only architecture-level storage paths, table names, high-level failure causes, and validation outcomes needed to describe the technical boundary are retained.

Status rationale

Status is verified because the primary claim of this note is the private storage/scoping foundation, and the source records a successful live encrypted artifact store plus the corresponding production corrections.

The status does not extend verification to semantic retrieval, background enrichment, or future storage features; those limitations are called out explicitly in the package.

CSV

metrics.csv

1,196 bytes SHA-256 dab767861a1c9b48…

metric,value,unit,status,notes
artifact_scope_classes,3,classes,derived,"Counted from the explicitly defined chat, project, and account artifact scopes."
explicit_core_artifact_tables,4,tables,derived,"Counted from user_artifacts, user_artifact_chunks, user_artifact_terms, and message_artifacts explicitly named in the source."
live_encrypted_artifact_store_verified,1,boolean,recorded,"The project update records a final successful live artifact store after the upload hotfixes."
upload_failures_resolved,2,failures,derived,"Two independent production upload failures are explicitly enumerated in the source."
stored_blob_permission_mode,600,unix_mode,recorded,"The successfully stored artifact was recorded with permission mode 600."
tracked_storage_content_after_cleanup,0,files,recorded,"The recorded git ls-files storage check returned no tracked storage content."
semantic_cross_chat_retrieval_verified,0,boolean,boundary,"The supplied update lists semantic cross-chat artifact recall as still requiring deliberate production QA."
artifact_enrichment_cron_verified,0,boolean,boundary,"Implementation exists, but the supplied update states the production cron still requires verification."

JSON

note.json

1,718 bytes SHA-256 78e395e8184914fd…

{
    "id": "KESPA-NOTE-025",
    "type": "technical_note",
    "title": "Private Encrypted Artifact Storage and Scoped Retrieval Foundation",
    "date": "2026-09-22",
    "status": "verified",
    "summary": "KESPA implemented a private upload and artifact foundation with encrypted originals outside the public web root, chat/project/account scoping, owner-scoped metadata and indexing structures, and runtime-storage exclusion from Git. A live artifact was successfully stored as an encrypted .kespa blob with 600 permissions after two production upload failures were diagnosed and corrected; semantic cross-chat retrieval and enrichment remain outside the verified boundary of this checkpoint.",
    "scope_classes": [
        "chat",
        "project",
        "account"
    ],
    "explicit_core_tables": [
        "user_artifacts",
        "user_artifact_chunks",
        "user_artifact_terms",
        "message_artifacts"
    ],
    "verified_at_checkpoint": {
        "private_artifact_architecture_implemented": true,
        "encrypted_storage_outside_public_web_root": true,
        "live_artifact_physically_stored": true,
        "stored_blob_permission_mode": "600",
        "runtime_storage_removed_from_git_tracking": true,
        "upload_failures_resolved": 2
    },
    "not_yet_claimed_verified": [
        "semantic_cross_chat_artifact_retrieval",
        "embedding_based_artifact_retrieval",
        "multimodal_vision_enrichment",
        "artifact_enrichment_cron_end_to_end",
        "all_supported_format_extractors",
        "artifact_deduplication_and_versioning",
        "cold_or_object_storage",
        "backup_restore_and_retention_lifecycle"
    ],
    "tags": [
        "private-artifacts",
        "encrypted-storage",
        "scope-aware-retrieval",
        "privacy",
        "uploads",
        "provenance"
    ]
}

JSON

provenance.json

4,101 bytes SHA-256 60cb450c56c7838f…

{
    "artifact_id": "KESPA-NOTE-025",
    "source_material": [
        {
            "type": "user_provided_project_update",
            "description": "KESPA Jira/project update dated 2026-09-22 covering private artifact architecture, live upload debugging, storage hygiene, plan guardrails, security boundaries, and open QA items."
        },
        {
            "type": "user_provided_readme_addendum",
            "description": "KESPA current-state README addendum dated 2026-09-22 covering artifact scope, storage model, retrieval design, processing options, and operational status."
        },
        {
            "type": "user_provided_future_plan",
            "description": "KESPA future-plan document dated 2026-09-22 used only to identify explicitly future artifact-intelligence work and avoid presenting planned features as completed results."
        },
        {
            "type": "existing_public_manifest",
            "description": "Current KESPA public research manifest with schema kespa.public_research_index.v1, 37 existing entries, and KESPA-NOTE-024 as the latest technical-note ID before this package."
        }
    ],
    "directly_recorded_facts": [
        "KESPA has a private artifact architecture for uploaded files and images.",
        "Artifacts can be scoped to chat, project, or account.",
        "The private artifact storage root is outside the public web root.",
        "Original artifact blobs are encrypted at rest.",
        "A live artifact was successfully stored after the recorded upload fixes.",
        "The successfully stored artifact was recorded as owned by the PHP service account and permissioned 600.",
        "The implementation explicitly names user_artifacts, user_artifact_chunks, user_artifact_terms, and message_artifacts.",
        "PHP-FPM private-storage write access was one recorded upload failure and was corrected.",
        "Numeric search-term type conversion was a second recorded upload failure and was corrected by casting terms to string before hashing.",
        "Runtime storage is ignored by Git and the recorded git ls-files storage check returned no tracked storage content.",
        "Semantic cross-chat artifact recall and the artifact-enrichment cron remain items requiring deliberate production verification."
    ],
    "derived_values": [
        {
            "name": "artifact_scope_classes",
            "value": 3,
            "formula": "count(chat, project, account)"
        },
        {
            "name": "explicit_core_artifact_tables",
            "value": 4,
            "formula": "count(user_artifacts, user_artifact_chunks, user_artifact_terms, message_artifacts)"
        },
        {
            "name": "upload_failures_resolved",
            "value": 2,
            "formula": "count(the two independently enumerated live upload failures)"
        }
    ],
    "interpretations": [
        "The artifact system establishes a distinct private-evidence boundary rather than treating uploaded files as ordinary global memory.",
        "The current evidence supports verification of encrypted storage and scope foundations but not a verified claim of semantic cross-chat retrieval quality or multimodal enrichment."
    ],
    "validation_boundary": {
        "recorded_complete_or_live_verified": [
            "private artifact architecture",
            "chat/project/account scope model",
            "encrypted non-public artifact storage",
            "successful live artifact persistence after fixes",
            "permission mode 600 on the recorded stored artifact",
            "core metadata/index tables",
            "runtime storage exclusion from Git"
        ],
        "not_yet_verified_end_to_end": [
            "semantic cross-chat artifact recall",
            "embedding-based retrieval quality",
            "vision/multimodal enrichment",
            "artifact enrichment worker under production cron",
            "all Office/PDF/image extraction paths",
            "artifact deduplication/versioning",
            "object/cold-storage lifecycle",
            "backup/restore/retention lifecycle"
        ]
    },
    "publication_exclusions": [
        "test user identifier",
        "artifact UUID and user-specific stored path",
        "uploaded artifact contents",
        "encryption keys and secrets",
        "provider credentials",
        "private application source code",
        "private user information"
    ]
}