Artificial Intelligence

Artificial Intelligence in RawCull

This section explains how RawCull uses reusable AI components without allowing model-runtime details to spread through the application. It is written as a learning path: first understand the boundary between RawCull and PhotoAIKit, then study the package itself, and finally follow CLIP from application startup to a persisted similarity artifact.

The source for this section comes from the pinned PhotoAIKit dependency and the RawCull repository. Paths beginning with Sources/ refer to PhotoAIKit at the revision in Package.resolved. RawCull intelligence code lives under RawCull/Intelligence; app composition and presentation remain under RawCull/Main, RawCull/Model, and RawCull/Views.

What This Section Documents

PhotoAIKit and RawCull use these AI backends:

• CLIP image embeddings for visual similarity and semantic search.
• SAM 3 and EfficientSAM subject segmentation for subject masks.
• Apple Vision feature prints for always-available image similarity.
• Qwen3-VL for local photo analysis and for concept discovery and assessment of numbered SAM 3 objects.

The 3.2.6 production download catalog exposes DataComp CLIP, Meta SAM 3, and Qwen3-VL-2B-Instruct. OpenAI CLIP remains excluded and EfficientSAM is not a production download. SAM 3 requires acceptance of its verified bundled licence before download. The App Store build uses Apple-hosted Managed Background Assets; the Direct/Developer ID build retains a self-hosted v3 manifest. See AI Model Downloads for the exact distinction.

Vision is the startup and service-selection fallback: RawCull uses it when CLIP is disabled or the selected CLIP bundle cannot produce a validated provider. A selected CLIP indexing pass keeps its valid per-file artifacts and records the files that fail; it does not mix Vision artifacts into that pass or automatically rerun the whole batch.

The AI Models in RawCull guide follows code connected to similarity, semantic search, burst analysis, Deep Review, Qwen, and Objects. The PhotoAIKit architecture guide covers SAM 3 contracts, workflows, and storage so that the package design is understandable. Not every reusable package capability is necessarily exposed as a finished RawCull user workflow.

Objects test-release status

The maintainer is preparing a user test of AI Analysis → Objects. The feature combines Qwen concept discovery, separate SAM 3 masks, and Qwen assessment of numbered crops. The structural decoder checks board IDs and response fields; it cannot verify that generated text matches the photograph. A recent two-puffin result had two retained objects but an invented third bird in its summary, and a separate two-puffin result has an unresolved crop/description mismatch. Treat Objects output as advisory and verify it against the source. See Publishing and Testing RawCull AI Models for the known issues, tester checks, and release evidence still needed.

The Central Design Idea

RawCull and PhotoAIKit answer different kinds of questions.

PhotoAIKit asks:

• What does an image source look like at a package boundary?
• How is a model bundle validated and identified?
• How does a backend produce and compare a similarity artifact?
• How is bounded indexing or segmentation orchestrated?
• How can reusable artifacts and masks be encoded or cached?

RawCull asks:

• Where are models installed for this application?
• How is a Sony, Nikon, or DNG RAW file decoded for AI input?
• Which backend did the user request?
• When should a catalog be indexed or reindexed?
• How do similarity distances affect burst grouping and culling?
• What state and wording should SwiftUI present?

This is dependency inversion in practical form. PhotoAIKit defines small protocols such as ImageDecoding, ImageSimilarityArtifactProviding, and ImageSimilarityArtifactComparing. RawCull injects app-specific implementations and keeps its file model, UI, sandbox policy, and culling decisions outside the package.

flowchart LR
    UI["RawCull SwiftUI and settings"] --> Integration["RawCullApplicationState composition root"]
    Integration --> AppAdapter["RawCull adapters: paths, RAW decoding, policy"]
    AppAdapter --> Contracts["PhotoAIContracts"]
    Integration --> CLIP["CoreAICLIPBackend"]
    Integration --> SAM3["CoreAISAM3Backend"]
    Integration --> Vision["VisionFeaturePrintBackend"]
    AppAdapter --> Workflows["PhotoAIWorkflows"]
    Workflows --> Contracts
    Storage["PhotoAIStorage"] --> Contracts
    CLIP --> Contracts
    SAM3 --> Contracts
    Vision --> Contracts

The arrow direction matters: PhotoAIKit does not import RawCull. A reusable package should not need to know what FileItem, RawCullViewModel, an app-specific model folder, or a burst winner means.

Responsibility Boundary

Current Runtime Shape

RawCullApplicationState is the object-graph assembly boundary. It creates one RawCullAIModelRuntime, one shared SimilarityScoringModel, the focused similarity and semantic-search features, Deep Review, Qwen and Objects features, the main view model, and one RawCullIntelligenceRuntime. Identity assertions protect against accidentally constructing parallel observable state.

RawCullAIModelRuntime owns concrete providers, resource managers, Qwen inference, and separate subject/object mask stores. RawCullIntelligenceRuntime owns stable feature lifetimes and applies complete revisioned configurations from settings. See The RawCull AI Runtime for the full construction and refresh sequence. Views receive focused feature surfaces instead of the composition root or low-level scoring model:

The safe startup and refresh path is:

1. RawCullApp calls RawCullApplicationState.live() and retains its view model and intelligence runtime as stable @State roots.
2. Assembly creates the shared scoring model and focused features from the initial Vision-backed configuration.
3. RawCullAISettingsModel.refresh() asks the model runtime to validate both CLIP and both segmentation-model candidates.
4. PhotoAIKit validates model bundles and derives model-asset fingerprints.
5. Settings publishes a monotonically revisioned configuration. The runtime replaces similarity or semantic-search services only when their identities changed and applies segmentation selection independently.
6. Missing, invalid, or disabled CLIP leaves burst similarity on Vision and semantic search unavailable.
7. CLIP indexing retains valid files and logs per-file failures; Vision is not inserted into that CLIP result set.
8. Changing the segmentation selection immediately updates the active provider. A full refresh rechecks every candidate and saved-evidence state.

stateDiagram-v2
    [*] --> VisionStartup
    VisionStartup --> CheckingModels: settings refresh
    CheckingModels --> Configuration: publish newer configuration revision
    Configuration --> CLIPSelected: preference enabled and provider ready
    Configuration --> VisionSelected: selected CLIP unavailable
    CLIPSelected --> CLIPArtifacts: keep valid per-file artifacts
    CLIPSelected --> PartialCLIP: record and exclude failed files
    VisionSelected --> VisionArtifacts: index catalog
    CheckingModels --> SegmentationSelected: activate SAM 3 or EfficientSAM

Burst similarity, semantic search, and Deep Review are separate features even when they share package code. Burst similarity may use Vision or CLIP. Semantic search requires CLIP image artifacts whose descriptor exactly matches the text provider. Deep Review uses segmentation masks and has its own availability, selection, and storage lifecycle.

Vocabulary