commit f6da346e1810bea5eeef05544724151b44e4da0a Author: Bastian Wagner Date: Sat Aug 15 09:00:41 2026 +0200 plans + specs diff --git a/docs/superpowers/plans/2026-08-15-fit-rewriter.md b/docs/superpowers/plans/2026-08-15-fit-rewriter.md new file mode 100644 index 0000000..573e73b --- /dev/null +++ b/docs/superpowers/plans/2026-08-15-fit-rewriter.md @@ -0,0 +1,577 @@ +# FIT Edge 1030 Plus Rewriter Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Implement a binary-preserving FIT metadata rewriter that changes only Garmin creator-device metadata plus required CRC bytes, targeting Garmin Edge 1030 Plus product ID `3570`. + +**Architecture:** Parse FIT definition/data records only far enough to locate `file_id` and creator `device_info` fields, patch bytes in place, recalculate header/file CRC, and validate the output. Do not fully decode/re-encode the activity, so unknown/developer fields and all non-target bytes remain untouched. + +**Tech Stack:** Python 3.12 standard library (`struct`, `dataclasses`, `pathlib`), pytest. + +## Global Constraints + +- Garmin manufacturer ID is `1`. +- Garmin Edge 1030 Plus product ID is `3570`. +- Product name is `Edge 1030 Plus`. +- Support FIT header sizes 12 and 14. +- Validate declared data size and `.FIT` signature. +- Validate header CRC when a 14-byte header is present. +- Validate file CRC before and after modification. +- Support little- and big-endian definition architectures, compressed timestamp records, developer fields, and changing local message definitions. +- Patch `file_id` device fields when present. +- Patch `device_info` only when it is safely identified as the creator (`device_index == 0`). +- If creator-specific `device_info` cannot be identified safely, leave it unchanged rather than rewriting all device records. +- Preserve every non-target byte except CRC fields. + +--- + +## File Structure + +```text +app/fit/ + __init__.py + crc.py + models.py + rewriter.py +tests/fit/ + builders.py + test_crc.py + test_rewriter_validation.py + test_rewriter_patching.py + test_rewriter_preservation.py +``` + +## Task 1: Implement Garmin FIT CRC + +**Files:** +- Create: `app/fit/crc.py` +- Create: `tests/fit/test_crc.py` + +**Interfaces:** +- Produces: `fit_crc(data: bytes | bytearray | memoryview) -> int`. + +- [ ] **Step 1: Write failing CRC vector tests** + +```python +# tests/fit/test_crc.py +from app.fit.crc import fit_crc + + +def test_empty_crc_is_zero() -> None: + assert fit_crc(b"") == 0 + + +def test_crc_is_incremental_equivalent() -> None: + payload = b".FIT-device-metadata" + assert fit_crc(payload) == fit_crc(memoryview(payload)) + assert 0 <= fit_crc(payload) <= 0xFFFF +``` + +- [ ] **Step 2: Run and verify failure** + +Run: `pytest tests/fit/test_crc.py -v` + +Expected: import failure. + +- [ ] **Step 3: Implement the FIT nibble-table CRC algorithm** + +```python +# app/fit/crc.py +CRC_TABLE = ( + 0x0000, 0xCC01, 0xD801, 0x1400, + 0xF001, 0x3C00, 0x2800, 0xE401, + 0xA001, 0x6C00, 0x7800, 0xB401, + 0x5000, 0x9C01, 0x8801, 0x4400, +) + + +def fit_crc(data: bytes | bytearray | memoryview) -> int: + crc = 0 + for byte in data: + tmp = CRC_TABLE[crc & 0xF] + crc = (crc >> 4) & 0x0FFF + crc ^= tmp ^ CRC_TABLE[byte & 0xF] + tmp = CRC_TABLE[crc & 0xF] + crc = (crc >> 4) & 0x0FFF + crc ^= tmp ^ CRC_TABLE[(byte >> 4) & 0xF] + return crc & 0xFFFF +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest tests/fit/test_crc.py -v` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add app/fit/crc.py tests/fit/test_crc.py +git commit -m "feat: add FIT CRC calculation" +``` + +## Task 2: Parse and validate the FIT container safely + +**Files:** +- Create: `app/fit/models.py` +- Create: `app/fit/rewriter.py` +- Create: `tests/fit/builders.py` +- Create: `tests/fit/test_rewriter_validation.py` + +**Interfaces:** +- Produces: `FitFormatError`, `FieldDefinition`, `LocalDefinition`, `_validate_fit_container()`, `_iter_data_fields()`. +- `_iter_data_fields(data)` returns data records with exact field offsets without decoding unrelated values. + +- [ ] **Step 1: Create a deterministic minimal FIT fixture builder** + +```python +# tests/fit/builders.py +import struct + +from app.fit.crc import fit_crc + + +def make_fit(data_records: bytes, *, header_size: int = 14) -> bytes: + if header_size not in {12, 14}: + raise ValueError(header_size) + header = bytearray(header_size) + header[0] = header_size + header[1] = 0x20 + struct.pack_into(" bytes: + architecture = 1 if endian == ">" else 0 + payload = bytearray([0, architecture]) + payload.extend(struct.pack(f"{endian}H", global_num)) + payload.append(len(fields)) + for num, size, base_type in fields: + payload.extend(bytes([num, size, base_type])) + return bytes([0x40 | local]) + bytes(payload) + + +def data(local: int, payload: bytes) -> bytes: + return bytes([local]) + payload +``` + +- [ ] **Step 2: Write validation tests** + +```python +# tests/fit/test_rewriter_validation.py +from pathlib import Path + +import pytest + +from app.fit.rewriter import FitFormatError, is_fit_file +from tests.fit.builders import make_fit + + +def test_valid_12_and_14_byte_headers(tmp_path: Path) -> None: + for size in (12, 14): + path = tmp_path / f"valid-{size}.fit" + path.write_bytes(make_fit(b"", header_size=size)) + assert is_fit_file(path) is True + + +def test_bad_file_crc_is_rejected(tmp_path: Path) -> None: + payload = bytearray(make_fit(b"")) + payload[-1] ^= 0xFF + path = tmp_path / "bad.fit" + path.write_bytes(payload) + assert is_fit_file(path) is False +``` + +- [ ] **Step 3: Run and verify failure** + +Run: `pytest tests/fit/test_rewriter_validation.py -v` + +Expected: import failure. + +- [ ] **Step 4: Implement the parser structures and validation** + +```python +# app/fit/models.py +from dataclasses import dataclass + + +@dataclass(frozen=True) +class FieldDefinition: + num: int + size: int + base_type: int + + +@dataclass(frozen=True) +class LocalDefinition: + global_message_num: int + endian: str + fields: tuple[FieldDefinition, ...] + developer_field_size: int +``` + +Implement in `app/fit/rewriter.py` the validated logic from the known working implementation: + +```python +class FitFormatError(ValueError): + pass + + +def _validate_fit_container(data: bytearray) -> None: + if len(data) < 14: + raise FitFormatError("FIT file is too small") + header_size = data[0] + if header_size not in {12, 14}: + raise FitFormatError(f"Unsupported FIT header size: {header_size}") + if len(data) < header_size + 2: + raise FitFormatError("FIT file is shorter than its header") + if bytes(data[8:12]) != b".FIT": + raise FitFormatError("Missing .FIT signature") + data_size = struct.unpack_from(" tuple[LocalDefinition, int]: + if offset + 5 > end_offset: + raise FitFormatError("Truncated FIT definition message") + offset += 1 # reserved byte + architecture = data[offset] + offset += 1 + if architecture not in {0, 1}: + raise FitFormatError(f"Unsupported FIT architecture: {architecture}") + endian = ">" if architecture == 1 else "<" + global_message_num = struct.unpack_from(f"{endian}H", data, offset)[0] + offset += 2 + field_count = data[offset] + offset += 1 + + fields: list[FieldDefinition] = [] + for _ in range(field_count): + if offset + 3 > end_offset: + raise FitFormatError("Truncated FIT field definition") + fields.append(FieldDefinition(data[offset], data[offset + 1], data[offset + 2])) + offset += 3 + + developer_field_size = 0 + if has_developer_fields: + if offset >= end_offset: + raise FitFormatError("Truncated FIT developer field count") + developer_count = data[offset] + offset += 1 + for _ in range(developer_count): + if offset + 3 > end_offset: + raise FitFormatError("Truncated FIT developer field definition") + developer_field_size += data[offset + 1] + offset += 3 + + return LocalDefinition(global_message_num, endian, tuple(fields), developer_field_size), offset + + +def _collect_field_offsets( + definition: LocalDefinition, + offset: int, + end_offset: int, +) -> tuple[list[tuple[FieldDefinition, int]], int]: + result: list[tuple[FieldDefinition, int]] = [] + current = offset + for field in definition.fields: + if current + field.size > end_offset: + raise FitFormatError("Truncated FIT data record") + result.append((field, current)) + current += field.size + if current + definition.developer_field_size > end_offset: + raise FitFormatError("Truncated FIT developer field payload") + current += definition.developer_field_size + return result, current + + +def _iter_data_fields( + data: bytearray, +) -> list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]]: + header_size = data[0] + data_size = struct.unpack_from("> 5) & 0x03 + definition = definitions.get(local) + if definition is None: + raise FitFormatError(f"Compressed timestamp record used unknown local definition {local}") + field_offsets, offset = _collect_field_offsets(definition, offset, end_offset) + records.append((definition, field_offsets)) + continue + + local = record_header & 0x0F + is_definition = bool(record_header & 0x40) + has_developer_fields = bool(record_header & 0x20) + if is_definition: + definition, offset = _read_definition(data, offset, has_developer_fields, end_offset) + definitions[local] = definition + continue + + definition = definitions.get(local) + if definition is None: + raise FitFormatError(f"Data record used unknown local definition {local}") + field_offsets, offset = _collect_field_offsets(definition, offset, end_offset) + records.append((definition, field_offsets)) + + if offset != end_offset: + raise FitFormatError("FIT parser did not end on data boundary") + return records +``` + +- [ ] **Step 5: Expose `is_fit_file` and verify parser boundary failures** + +```python +def is_fit_file(path: Path) -> bool: + try: + data = bytearray(path.read_bytes()) + _validate_fit_container(data) + _iter_data_fields(data) + except (OSError, FitFormatError): + return False + return True +``` + +- [ ] **Step 6: Run validation tests** + +Run: `pytest tests/fit/test_rewriter_validation.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add app/fit/models.py app/fit/rewriter.py tests/fit/builders.py tests/fit/test_rewriter_validation.py +git commit -m "feat: parse and validate FIT containers" +``` + +## Task 3: Patch `file_id` and only creator `device_info` + +**Files:** +- Modify: `app/fit/models.py` +- Modify: `app/fit/rewriter.py` +- Create: `tests/fit/test_rewriter_patching.py` + +**Interfaces:** +- Produces: `GarminDevice`, `FitConversionResult`, `DeviceFieldValue`, `convert_fit_device()`, `read_device_field_values()`. +- Default device is Garmin manufacturer `1`, Edge 1030 Plus product `3570`, name `Edge 1030 Plus`. + +- [ ] **Step 1: Add model types** + +```python +# app/fit/models.py additions +from pathlib import Path + +GARMIN_MANUFACTURER_ID = 1 + + +@dataclass(frozen=True) +class GarminDevice: + manufacturer_id: int = GARMIN_MANUFACTURER_ID + product_id: int = 3570 + product_name: str = "Edge 1030 Plus" + serial_number: int | None = None + + +@dataclass(frozen=True) +class FitConversionResult: + source_path: Path + output_path: Path + patched_field_count: int + header_crc: int | None + file_crc: int + + +@dataclass(frozen=True) +class DeviceFieldValue: + global_message_num: int + field_num: int + value: int | str +``` + +- [ ] **Step 2: Write a fixture containing one creator and one sensor `device_info`** + +Use these FIT field numbers in `tests/fit/test_rewriter_patching.py`: + +```python +FILE_ID_MESG_NUM = 0 +DEVICE_INFO_MESG_NUM = 23 + +# file_id: manufacturer(1/u16), product(2/u16) +file_def = definition(0, FILE_ID_MESG_NUM, [(1, 2, 0x84), (2, 2, 0x84)]) +file_data = data(0, struct.pack(" FitConversionResult: + resolved = device or GarminDevice() + data = bytearray(source_path.read_bytes()) + _validate_fit_container(data) + patched_count = _patch_device_metadata(data, resolved) + header_crc = _rewrite_header_crc(data) + file_crc = _rewrite_file_crc(data) + _validate_fit_container(data) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_bytes(data) + return FitConversionResult(source_path, output_path, patched_count, header_crc, file_crc) +``` + +- [ ] **Step 6: Run patching tests** + +Run: `pytest tests/fit/test_rewriter_patching.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add app/fit tests/fit/test_rewriter_patching.py +git commit -m "feat: patch FIT creator as Edge 1030 Plus" +``` + +## Task 4: Prove binary preservation and advanced record support + +**Files:** +- Create: `tests/fit/test_rewriter_preservation.py` +- Modify: `tests/fit/builders.py` +- Modify: `app/fit/rewriter.py` only if a test exposes an actual parser defect + +**Interfaces:** +- No new public interface; this task hardens `convert_fit_device()`. + +- [ ] **Step 1: Write a preservation test that records changed byte positions** + +```python +# tests/fit/test_rewriter_preservation.py +from pathlib import Path + +from app.fit.rewriter import convert_fit_device + + +def test_only_target_fields_and_crcs_change(tmp_path: Path, complex_fit_bytes: bytes) -> None: + source = tmp_path / "source.fit" + output = tmp_path / "output.fit" + source.write_bytes(complex_fit_bytes) + + convert_fit_device(source, output) + + before = source.read_bytes() + after = output.read_bytes() + assert len(before) == len(after) + + changed = {index for index, (a, b) in enumerate(zip(before, after)) if a != b} + expected_metadata_offsets = set(find_expected_device_metadata_offsets(before)) + crc_offsets = {12, 13, len(before) - 2, len(before) - 1} + assert changed <= expected_metadata_offsets | crc_offsets +``` + +The fixture helper `find_expected_device_metadata_offsets()` must use the test fixture's known construction offsets, not production parser code, so the test is independent. + +- [ ] **Step 2: Add fixtures for compressed timestamps, developer fields, and local-definition replacement** + +Construct one synthetic FIT data section containing: + +1. a normal `file_id` definition/data pair; +2. a `device_info` definition with creator record; +3. a record definition with one developer field and one data record; +4. a compressed-timestamp data header referring to a known local definition; +5. a later replacement definition for the same local message number. + +The test passes if conversion completes, output CRC validates, and non-target payload bytes remain identical. + +- [ ] **Step 3: Run the focused advanced tests** + +Run: `pytest tests/fit/test_rewriter_preservation.py -v` + +Expected: PASS. If a parser guard fails, fix only the smallest parser defect required by the test. + +- [ ] **Step 4: Add rejection tests for malformed/truncated definitions** + +```python +def test_truncated_definition_is_non_recoverable(tmp_path: Path) -> None: + path = tmp_path / "truncated.fit" + path.write_bytes(make_fit(bytes([0x40, 0x00, 0x00]))) + assert is_fit_file(path) is False +``` + +Also assert that `convert_fit_device()` raises `FitFormatError` for the same input. + +- [ ] **Step 5: Run the entire FIT suite** + +Run: `pytest tests/fit -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add tests/fit app/fit/rewriter.py +git commit -m "test: verify FIT binary preservation" +``` diff --git a/docs/superpowers/plans/2026-08-15-foundation-admin-data.md b/docs/superpowers/plans/2026-08-15-foundation-admin-data.md new file mode 100644 index 0000000..026dd2a --- /dev/null +++ b/docs/superpowers/plans/2026-08-15-foundation-admin-data.md @@ -0,0 +1,1087 @@ +# Foundation, Admin UI, and Data Layer Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the runnable FastAPI application shell with SQLite persistence, encrypted per-user credentials, single-password admin authentication, CSRF protection, local user CRUD, and Docker packaging. + +**Architecture:** Use one FastAPI process with synchronous SQLAlchemy sessions for low-volume SQLite access, server-rendered Jinja2 templates, signed cookie sessions, and Fernet encryption for stored service credentials. Deployment secrets and configuration come only from environment variables; persistent state lives under `/data`. + +**Tech Stack:** Python 3.12, FastAPI, Starlette sessions, SQLAlchemy 2.x, Pydantic Settings, cryptography/Fernet, Jinja2, python-multipart, pytest, Docker. + +## Global Constraints + +- The admin UI is intended for local-network use only. +- Use one admin password from `ADMIN_PASSWORD`; no username and no per-user web logins. +- Use SQLite for persistent application state. +- Store deployment secrets/configuration in environment variables, not in SQLite. +- Encrypt MyWhoosh and Garmin credentials before writing them to SQLite. +- Never return stored passwords to templates or API responses. +- Use `HttpOnly` and `SameSite=Lax` session cookies. +- Protect every state-changing web request with CSRF validation. +- Keep all persistent application data under `DATA_DIR`, defaulting to `/data`. +- Do not introduce Angular, React, Tailwind, Bootstrap, OAuth/OIDC, or an external queue in v1. + +--- + +## File Structure + +```text +pyproject.toml +Dockerfile +docker-compose.example.yml +.env.example +app/ + __init__.py + main.py + config.py + auth/ + __init__.py + admin.py + csrf.py + db/ + __init__.py + models.py + session.py + repositories.py + security/ + __init__.py + credentials.py + web/ + __init__.py + routes.py + forms.py + templates/ + base.html + login.html + dashboard.html + users/form.html + users/detail.html + static/ + app.css +tests/ + conftest.py + test_config.py + db/test_repositories.py + security/test_credentials.py + web/test_auth.py + web/test_users.py +``` + +## Task 1: Bootstrap configuration and application factory + +**Files:** +- Create: `pyproject.toml` +- Create: `app/config.py` +- Create: `app/main.py` +- Create: `tests/test_config.py` +- Create: `tests/conftest.py` + +**Interfaces:** +- Produces: `Settings`, `get_settings()`, `create_app(settings: Settings | None = None) -> FastAPI`. +- Later tasks consume `Settings.data_dir`, `Settings.database_url`, `Settings.admin_password`, `Settings.secret_key`, and `Settings.credential_encryption_key`. + +- [ ] **Step 1: Write failing configuration tests** + +```python +# tests/test_config.py +from pathlib import Path + +from app.config import Settings + + +def test_settings_build_default_data_paths(tmp_path: Path) -> None: + settings = Settings( + ADMIN_PASSWORD="admin-secret", + SECRET_KEY="session-secret", + CREDENTIAL_ENCRYPTION_KEY="ZmFrZS1rZXktZm9yLXRlc3RzLW11c3QtYmUtNDQtY2hhcnM=", + DATA_DIR=str(tmp_path), + SYNC_INTERVAL_MINUTES=5, + ) + + assert settings.data_dir == tmp_path + assert settings.database_url == f"sqlite:///{tmp_path / 'app.db'}" + assert settings.tokens_dir == tmp_path / "tokens" + assert settings.activities_dir == tmp_path / "activities" + + +def test_sync_interval_must_be_positive(tmp_path: Path) -> None: + try: + Settings( + ADMIN_PASSWORD="admin-secret", + SECRET_KEY="session-secret", + CREDENTIAL_ENCRYPTION_KEY="ZmFrZS1rZXktZm9yLXRlc3RzLW11c3QtYmUtNDQtY2hhcnM=", + DATA_DIR=str(tmp_path), + SYNC_INTERVAL_MINUTES=0, + ) + except ValueError: + return + raise AssertionError("Expected validation failure for non-positive interval") +``` + +- [ ] **Step 2: Run the tests and verify they fail** + +Run: `pytest tests/test_config.py -v` + +Expected: import/definition failure because `app.config.Settings` does not exist yet. + +- [ ] **Step 3: Add project dependencies and implement `Settings`** + +```toml +# pyproject.toml +[project] +name = "mywhoosh-garmin-sync" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = [ + "fastapi>=0.115,<1", + "uvicorn[standard]>=0.30,<1", + "sqlalchemy>=2.0,<3", + "pydantic-settings>=2.0,<3", + "cryptography>=43,<50", + "jinja2>=3.1,<4", + "python-multipart>=0.0.9,<1", +] + +[project.optional-dependencies] +test = [ + "pytest>=8,<9", + "httpx>=0.27,<1", +] + +[tool.pytest.ini_options] +testpaths = ["tests"] +``` + +```python +# app/config.py +from functools import lru_cache +from pathlib import Path + +from pydantic import Field, PositiveInt, model_validator +from pydantic_settings import BaseSettings, SettingsConfigDict + + +class Settings(BaseSettings): + model_config = SettingsConfigDict(env_file=".env", extra="ignore", case_sensitive=False) + + admin_password: str = Field(alias="ADMIN_PASSWORD", min_length=1) + secret_key: str = Field(alias="SECRET_KEY", min_length=16) + credential_encryption_key: str = Field(alias="CREDENTIAL_ENCRYPTION_KEY", min_length=1) + data_dir: Path = Field(default=Path("/data"), alias="DATA_DIR") + database_url: str | None = Field(default=None, alias="DATABASE_URL") + sync_interval_minutes: PositiveInt = Field(default=5, alias="SYNC_INTERVAL_MINUTES") + + @model_validator(mode="after") + def derive_paths(self) -> "Settings": + self.data_dir = self.data_dir.expanduser().resolve() + if self.database_url is None: + self.database_url = f"sqlite:///{self.data_dir / 'app.db'}" + return self + + @property + def tokens_dir(self) -> Path: + return self.data_dir / "tokens" + + @property + def activities_dir(self) -> Path: + return self.data_dir / "activities" + + +@lru_cache +def get_settings() -> Settings: + return Settings() +``` + +- [ ] **Step 4: Implement a minimal application factory** + +```python +# app/main.py +from fastapi import FastAPI + +from app.config import Settings, get_settings + + +def create_app(settings: Settings | None = None) -> FastAPI: + resolved = settings or get_settings() + resolved.data_dir.mkdir(parents=True, exist_ok=True) + resolved.tokens_dir.mkdir(parents=True, exist_ok=True) + resolved.activities_dir.mkdir(parents=True, exist_ok=True) + + app = FastAPI(title="MyWhoosh Garmin Sync") + app.state.settings = resolved + + @app.get("/healthz") + def healthz() -> dict[str, str]: + return {"status": "ok"} + + return app + + +app = create_app() +``` + +- [ ] **Step 5: Run tests and smoke-test the app factory** + +Run: `pytest tests/test_config.py -v` + +Expected: PASS. + +Run: `python -c "from app.main import create_app; print(create_app)"` + +Expected: prints the function object without configuration-time crashes. + +- [ ] **Step 6: Commit** + +```bash +git add pyproject.toml app/config.py app/main.py tests/test_config.py tests/conftest.py +git commit -m "feat: bootstrap FastAPI configuration" +``` + +## Task 2: Add SQLite models and repositories + +**Files:** +- Create: `app/db/models.py` +- Create: `app/db/session.py` +- Create: `app/db/repositories.py` +- Create: `tests/db/test_repositories.py` +- Modify: `app/main.py` + +**Interfaces:** +- Produces: `SyncUser`, `Activity`, `SyncRun`, `UserRepository`, `ActivityRepository`, `SyncRunRepository`, `create_db_engine()`, `create_session_factory()`. +- `Activity` must enforce unique `(user_id, mywhoosh_activity_id)`. +- To make resumability explicit, store both `status` and `last_completed_stage`; `status="failed"` does not erase the last durable stage. + +- [ ] **Step 1: Write repository tests for isolated users and idempotent activities** + +```python +# tests/db/test_repositories.py +from app.db.models import ActivityStatus, HealthState + + +def test_create_two_independent_users(db_session, user_repository) -> None: + first = user_repository.create( + name="Max", + enabled=True, + health_state=HealthState.HEALTHY, + mywhoosh_email_enc="mw-1", + mywhoosh_password_enc="mw-pw-1", + garmin_email_enc="g-1", + garmin_password_enc="g-pw-1", + ) + second = user_repository.create( + name="Anna", + enabled=True, + health_state=HealthState.HEALTHY, + mywhoosh_email_enc="mw-2", + mywhoosh_password_enc="mw-pw-2", + garmin_email_enc="g-2", + garmin_password_enc="g-pw-2", + ) + + assert first.id != second.id + assert {u.name for u in user_repository.list_enabled()} == {"Max", "Anna"} + + +def test_activity_external_id_is_unique_per_user(user_repository, activity_repository) -> None: + user = user_repository.create( + name="Max", + enabled=True, + health_state=HealthState.HEALTHY, + mywhoosh_email_enc="a", + mywhoosh_password_enc="b", + garmin_email_enc="c", + garmin_password_enc="d", + ) + created, inserted = activity_repository.get_or_create_discovered( + user_id=user.id, + mywhoosh_activity_id="mw-123", + activity_name="Morning Ride", + activity_timestamp=None, + ) + same, inserted_again = activity_repository.get_or_create_discovered( + user_id=user.id, + mywhoosh_activity_id="mw-123", + activity_name="Morning Ride", + activity_timestamp=None, + ) + + assert inserted is True + assert inserted_again is False + assert created.id == same.id + assert same.status == ActivityStatus.DISCOVERED +``` + +- [ ] **Step 2: Run the repository tests and verify failure** + +Run: `pytest tests/db/test_repositories.py -v` + +Expected: imports fail because database modules are not implemented. + +- [ ] **Step 3: Define enums and models** + +```python +# app/db/models.py +from __future__ import annotations + +import enum +from datetime import datetime, timezone + +from sqlalchemy import Boolean, DateTime, Enum, ForeignKey, Integer, String, Text, UniqueConstraint +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship + + +def utcnow() -> datetime: + return datetime.now(timezone.utc) + + +class Base(DeclarativeBase): + pass + + +class HealthState(str, enum.Enum): + HEALTHY = "healthy" + SYNCING = "syncing" + DEGRADED = "degraded" + ACTION_REQUIRED = "action_required" + DISABLED = "disabled" + + +class ActivityStatus(str, enum.Enum): + DISCOVERED = "discovered" + DOWNLOADED = "downloaded" + CONVERTED = "converted" + IMPORTED = "imported" + DUPLICATE = "duplicate" + FAILED = "failed" + + +class SyncRunStatus(str, enum.Enum): + RUNNING = "running" + SUCCESS = "success" + PARTIAL = "partial" + FAILED = "failed" + + +class SyncUser(Base): + __tablename__ = "sync_users" + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + name: Mapped[str] = mapped_column(String(120), nullable=False) + enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) + health_state: Mapped[HealthState] = mapped_column(Enum(HealthState), nullable=False, default=HealthState.HEALTHY) + mywhoosh_state: Mapped[str] = mapped_column(String(32), nullable=False, default="unknown") + garmin_state: Mapped[str] = mapped_column(String(32), nullable=False, default="unknown") + action_reason: Mapped[str | None] = mapped_column(Text) + mywhoosh_email_enc: Mapped[str] = mapped_column(Text, nullable=False) + mywhoosh_password_enc: Mapped[str] = mapped_column(Text, nullable=False) + garmin_email_enc: Mapped[str] = mapped_column(Text, nullable=False) + garmin_password_enc: Mapped[str] = mapped_column(Text, nullable=False) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, onupdate=utcnow) + + +class Activity(Base): + __tablename__ = "activities" + __table_args__ = (UniqueConstraint("user_id", "mywhoosh_activity_id", name="uq_activity_user_mywhoosh"),) + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + user_id: Mapped[int] = mapped_column(ForeignKey("sync_users.id", ondelete="CASCADE"), nullable=False, index=True) + mywhoosh_activity_id: Mapped[str] = mapped_column(String(255), nullable=False) + activity_name: Mapped[str] = mapped_column(String(255), nullable=False) + activity_timestamp: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + source_fit_path: Mapped[str | None] = mapped_column(Text) + converted_fit_path: Mapped[str | None] = mapped_column(Text) + status: Mapped[ActivityStatus] = mapped_column(Enum(ActivityStatus), nullable=False, default=ActivityStatus.DISCOVERED) + last_completed_stage: Mapped[ActivityStatus] = mapped_column(Enum(ActivityStatus), nullable=False, default=ActivityStatus.DISCOVERED) + retryable: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) + garmin_activity_id: Mapped[str | None] = mapped_column(String(255)) + last_error: Mapped[str | None] = mapped_column(Text) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, onupdate=utcnow) + + +class SyncRun(Base): + __tablename__ = "sync_runs" + + id: Mapped[int] = mapped_column(Integer, primary_key=True) + user_id: Mapped[int] = mapped_column(ForeignKey("sync_users.id", ondelete="CASCADE"), nullable=False, index=True) + started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) + finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + status: Mapped[SyncRunStatus] = mapped_column(Enum(SyncRunStatus), nullable=False, default=SyncRunStatus.RUNNING) + discovered_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + imported_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + skipped_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + failed_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) + summary_error: Mapped[str | None] = mapped_column(Text) +``` + +- [ ] **Step 4: Implement session factory and focused repositories** + +```python +# app/db/session.py +from sqlalchemy import create_engine +from sqlalchemy.engine import Engine +from sqlalchemy.orm import Session, sessionmaker + +from app.db.models import Base + + +def create_db_engine(database_url: str) -> Engine: + connect_args = {"check_same_thread": False} if database_url.startswith("sqlite") else {} + return create_engine(database_url, connect_args=connect_args, future=True) + + +def create_session_factory(engine: Engine) -> sessionmaker[Session]: + return sessionmaker(bind=engine, autoflush=False, expire_on_commit=False) + + +def initialize_schema(engine: Engine) -> None: + Base.metadata.create_all(engine) +``` + +```python +# app/db/repositories.py +from datetime import datetime + +from sqlalchemy import select +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.db.models import Activity, ActivityStatus, HealthState, SyncUser + + +class UserRepository: + def __init__(self, session: Session) -> None: + self.session = session + + def create(self, **values) -> SyncUser: + user = SyncUser(**values) + self.session.add(user) + self.session.commit() + return user + + def get(self, user_id: int) -> SyncUser | None: + return self.session.get(SyncUser, user_id) + + def list_enabled(self) -> list[SyncUser]: + return list(self.session.scalars(select(SyncUser).where(SyncUser.enabled.is_(True)).order_by(SyncUser.id))) + + +class ActivityRepository: + def __init__(self, session: Session) -> None: + self.session = session + + def get_or_create_discovered( + self, + *, + user_id: int, + mywhoosh_activity_id: str, + activity_name: str, + activity_timestamp: datetime | None, + ) -> tuple[Activity, bool]: + existing = self.session.scalar( + select(Activity).where( + Activity.user_id == user_id, + Activity.mywhoosh_activity_id == mywhoosh_activity_id, + ) + ) + if existing is not None: + return existing, False + activity = Activity( + user_id=user_id, + mywhoosh_activity_id=mywhoosh_activity_id, + activity_name=activity_name, + activity_timestamp=activity_timestamp, + status=ActivityStatus.DISCOVERED, + last_completed_stage=ActivityStatus.DISCOVERED, + ) + self.session.add(activity) + try: + self.session.commit() + except IntegrityError: + self.session.rollback() + existing = self.session.scalar( + select(Activity).where( + Activity.user_id == user_id, + Activity.mywhoosh_activity_id == mywhoosh_activity_id, + ) + ) + if existing is None: + raise + return existing, False + return activity, True +``` + +- [ ] **Step 5: Wire database initialization into `create_app` and add DB test fixtures** + +```python +# add inside create_app in app/main.py +from app.db.session import create_db_engine, create_session_factory, initialize_schema + +engine = create_db_engine(resolved.database_url) +initialize_schema(engine) +app.state.db_engine = engine +app.state.session_factory = create_session_factory(engine) +``` + +```python +# tests/conftest.py +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import Session, sessionmaker +from sqlalchemy.pool import StaticPool + +from app.db.models import Base +from app.db.repositories import ActivityRepository, UserRepository + + +@pytest.fixture +def db_session() -> Session: + engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + factory = sessionmaker(bind=engine, expire_on_commit=False) + with factory() as session: + yield session + + +@pytest.fixture +def user_repository(db_session: Session) -> UserRepository: + return UserRepository(db_session) + + +@pytest.fixture +def activity_repository(db_session: Session) -> ActivityRepository: + return ActivityRepository(db_session) +``` + +- [ ] **Step 6: Run tests** + +Run: `pytest tests/db/test_repositories.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add app/db app/main.py tests/conftest.py tests/db/test_repositories.py +git commit -m "feat: add SQLite persistence models" +``` + +## Task 3: Encrypt credentials at rest + +**Files:** +- Create: `app/security/credentials.py` +- Create: `tests/security/test_credentials.py` + +**Interfaces:** +- Produces: `CredentialCipher.encrypt(value: str) -> str`, `CredentialCipher.decrypt(token: str) -> str`. +- The constructor accepts exactly the value of `CREDENTIAL_ENCRYPTION_KEY`. + +- [ ] **Step 1: Write failing encryption tests** + +```python +# tests/security/test_credentials.py +from cryptography.fernet import Fernet + +from app.security.credentials import CredentialCipher + + +def test_round_trip_and_ciphertext_does_not_contain_plaintext() -> None: + cipher = CredentialCipher(Fernet.generate_key().decode("ascii")) + encrypted = cipher.encrypt("secret-password") + + assert "secret-password" not in encrypted + assert cipher.decrypt(encrypted) == "secret-password" + + +def test_empty_credentials_are_rejected() -> None: + cipher = CredentialCipher(Fernet.generate_key().decode("ascii")) + try: + cipher.encrypt("") + except ValueError: + return + raise AssertionError("empty secrets must be rejected") +``` + +- [ ] **Step 2: Run tests and verify failure** + +Run: `pytest tests/security/test_credentials.py -v` + +Expected: import failure. + +- [ ] **Step 3: Implement the cipher** + +```python +# app/security/credentials.py +from cryptography.fernet import Fernet, InvalidToken + + +class CredentialCipher: + def __init__(self, key: str) -> None: + try: + self._fernet = Fernet(key.encode("ascii")) + except Exception as exc: + raise ValueError("CREDENTIAL_ENCRYPTION_KEY must be a valid Fernet key") from exc + + def encrypt(self, value: str) -> str: + if not value: + raise ValueError("credential value must not be empty") + return self._fernet.encrypt(value.encode("utf-8")).decode("ascii") + + def decrypt(self, token: str) -> str: + try: + return self._fernet.decrypt(token.encode("ascii")).decode("utf-8") + except InvalidToken as exc: + raise ValueError("stored credential cannot be decrypted") from exc +``` + +- [ ] **Step 4: Run tests** + +Run: `pytest tests/security/test_credentials.py -v` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add app/security/credentials.py tests/security/test_credentials.py +git commit -m "feat: encrypt stored service credentials" +``` + +## Task 4: Add admin login, signed session, and CSRF protection + +**Files:** +- Create: `app/auth/admin.py` +- Create: `app/auth/csrf.py` +- Create: `app/web/routes.py` +- Create: `app/web/templates/base.html` +- Create: `app/web/templates/login.html` +- Create: `tests/web/test_auth.py` +- Modify: `app/main.py` + +**Interfaces:** +- Produces: `require_admin(request)`, `ensure_csrf_token(request)`, `validate_csrf(request, submitted_token)`. +- Session key for authentication is `request.session["admin_authenticated"] is True`. +- Session key for CSRF is `request.session["csrf_token"]`. + +- [ ] **Step 1: Write failing auth and CSRF tests** + +```python +# tests/web/test_auth.py +from fastapi.testclient import TestClient + + +def test_dashboard_redirects_when_not_logged_in(client: TestClient) -> None: + response = client.get("/", follow_redirects=False) + assert response.status_code == 303 + assert response.headers["location"] == "/login" + + +def extract_csrf(html: str) -> str: + marker = 'name="csrf_token" value="' + start = html.index(marker) + len(marker) + return html[start:html.index('"', start)] + + +def test_login_rejects_wrong_password(client: TestClient) -> None: + login_page = client.get("/login") + csrf = extract_csrf(login_page.text) + response = client.post( + "/login", + data={"password": "wrong", "csrf_token": csrf}, + follow_redirects=False, + ) + assert response.status_code == 401 + + +def test_login_accepts_configured_password(client: TestClient) -> None: + login_page = client.get("/login") + csrf = extract_csrf(login_page.text) + response = client.post( + "/login", + data={"password": "admin-secret", "csrf_token": csrf}, + follow_redirects=False, + ) + assert response.status_code == 303 + assert response.headers["location"] == "/" +``` + +- [ ] **Step 2: Run tests and verify failure** + +Run: `pytest tests/web/test_auth.py -v` + +Expected: route/import failures. + +- [ ] **Step 3: Implement admin comparison and CSRF helpers** + +```python +# app/auth/admin.py +import hmac + +from fastapi import HTTPException, Request, status + + +def password_matches(submitted: str, configured: str) -> bool: + return hmac.compare_digest(submitted.encode("utf-8"), configured.encode("utf-8")) + + +def require_admin(request: Request) -> None: + if request.session.get("admin_authenticated") is not True: + raise HTTPException(status_code=status.HTTP_303_SEE_OTHER, headers={"Location": "/login"}) +``` + +```python +# app/auth/csrf.py +import hmac +import secrets + +from fastapi import HTTPException, Request, status + + +def ensure_csrf_token(request: Request) -> str: + token = request.session.get("csrf_token") + if not isinstance(token, str): + token = secrets.token_urlsafe(32) + request.session["csrf_token"] = token + return token + + +def validate_csrf(request: Request, submitted_token: str) -> None: + expected = request.session.get("csrf_token") + if not isinstance(expected, str) or not hmac.compare_digest(expected, submitted_token): + raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid CSRF token") +``` + +- [ ] **Step 4: Add login/dashboard routes and templates** + +```python +# app/web/routes.py +from fastapi import APIRouter, Form, Request +from fastapi.responses import HTMLResponse, RedirectResponse +from fastapi.templating import Jinja2Templates + +from app.auth.admin import password_matches, require_admin +from app.auth.csrf import ensure_csrf_token, validate_csrf + +router = APIRouter() +templates = Jinja2Templates(directory="app/web/templates") + + +@router.get("/login", response_class=HTMLResponse) +def login_page(request: Request): + return templates.TemplateResponse(request, "login.html", {"csrf_token": ensure_csrf_token(request)}) + + +@router.post("/login") +def login( + request: Request, + password: str = Form(...), + csrf_token: str = Form(...), +): + validate_csrf(request, csrf_token) + settings = request.app.state.settings + if not password_matches(password, settings.admin_password): + return templates.TemplateResponse( + request, + "login.html", + {"csrf_token": ensure_csrf_token(request), "error": "Invalid password"}, + status_code=401, + ) + request.session["admin_authenticated"] = True + return RedirectResponse("/", status_code=303) + + +@router.get("/", response_class=HTMLResponse) +def dashboard(request: Request): + require_admin(request) + return templates.TemplateResponse(request, "dashboard.html", {"csrf_token": ensure_csrf_token(request), "users": []}) +``` + +- [ ] **Step 5: Install `SessionMiddleware` and include routes** + +```python +# app/main.py additions +from starlette.middleware.sessions import SessionMiddleware +from app.web.routes import router as web_router + +app.add_middleware( + SessionMiddleware, + secret_key=resolved.secret_key, + same_site="lax", + https_only=False, +) +app.include_router(web_router) +``` + +- [ ] **Step 6: Add the concrete FastAPI test client fixture and run auth tests** + +```python +# tests/conftest.py additions +from cryptography.fernet import Fernet +from fastapi.testclient import TestClient + +from app.config import Settings +from app.main import create_app + + +@pytest.fixture +def client(tmp_path: Path) -> TestClient: + settings = Settings( + ADMIN_PASSWORD="admin-secret", + SECRET_KEY="0123456789abcdef0123456789abcdef", + CREDENTIAL_ENCRYPTION_KEY=Fernet.generate_key().decode("ascii"), + DATA_DIR=str(tmp_path), + DATABASE_URL=f"sqlite:///{tmp_path / 'app.db'}", + SYNC_INTERVAL_MINUTES=5, + ) + return TestClient(create_app(settings)) +``` + +Run: `pytest tests/web/test_auth.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add app/auth app/web app/main.py tests/web/test_auth.py tests/conftest.py +git commit -m "feat: add local admin authentication" +``` + +## Task 5: Add user CRUD without exposing stored secrets + +**Files:** +- Create: `app/web/forms.py` +- Create: `app/web/templates/users/form.html` +- Create: `app/web/templates/users/detail.html` +- Modify: `app/web/routes.py` +- Modify: `app/db/repositories.py` +- Create: `tests/web/test_users.py` + +**Interfaces:** +- Produces routes: `GET /users/new`, `POST /users`, `GET /users/{id}`, `GET /users/{id}/edit`, `POST /users/{id}`. +- Empty password fields on edit preserve existing encrypted passwords. +- Templates receive only non-secret fields. + +- [ ] **Step 1: Write failing user CRUD tests** + +```python +# tests/web/test_users.py +from fastapi.testclient import TestClient + + +def login(client: TestClient) -> None: + page = client.get("/login") + csrf = extract_csrf(page.text) + response = client.post( + "/login", + data={"password": "admin-secret", "csrf_token": csrf}, + follow_redirects=False, + ) + assert response.status_code == 303 + + +def extract_csrf(html: str) -> str: + marker = 'name="csrf_token" value="' + start = html.index(marker) + len(marker) + end = html.index('"', start) + return html[start:end] + + +def test_create_user_encrypts_credentials_and_never_renders_them(client: TestClient) -> None: + login(client) + page = client.get("/users/new") + assert page.status_code == 200 + csrf = extract_csrf(page.text) + response = client.post( + "/users", + data={ + "csrf_token": csrf, + "name": "Max", + "mywhoosh_email": "max@example.com", + "mywhoosh_password": "mw-secret", + "garmin_email": "max-garmin@example.com", + "garmin_password": "garmin-secret", + "enabled": "on", + }, + follow_redirects=True, + ) + assert response.status_code == 200 + assert "mw-secret" not in response.text + assert "garmin-secret" not in response.text +``` + +- [ ] **Step 2: Run the test and verify failure** + +Run: `pytest tests/web/test_users.py -v` + +Expected: missing routes/form support. + +- [ ] **Step 3: Extend repository update methods** + +```python +# app/db/repositories.py additions + def list_all(self) -> list[SyncUser]: + return list(self.session.scalars(select(SyncUser).order_by(SyncUser.name))) + + def update(self, user: SyncUser, **values) -> SyncUser: + for key, value in values.items(): + setattr(user, key, value) + self.session.commit() + return user +``` + +- [ ] **Step 4: Implement create/edit request handling with encrypted fields** + +```python +# app/web/forms.py +from dataclasses import dataclass + + +@dataclass(frozen=True) +class UserFormData: + name: str + mywhoosh_email: str + mywhoosh_password: str + garmin_email: str + garmin_password: str + enabled: bool +``` + +In `app/web/routes.py`, construct `CredentialCipher(request.app.state.settings.credential_encryption_key)` and encrypt all four service values before repository writes. On edit, only replace an encrypted password if the submitted password is non-empty. Decrypt emails for display; never decrypt passwords for a template. + +Use the exact update payload shape: + +```python +values = { + "name": form.name.strip(), + "enabled": form.enabled, + "mywhoosh_email_enc": cipher.encrypt(form.mywhoosh_email.strip()), + "garmin_email_enc": cipher.encrypt(form.garmin_email.strip()), +} +if form.mywhoosh_password: + values["mywhoosh_password_enc"] = cipher.encrypt(form.mywhoosh_password) +if form.garmin_password: + values["garmin_password_enc"] = cipher.encrypt(form.garmin_password) +``` + +Every POST route must call `validate_csrf(request, csrf_token)` before changing state. + +- [ ] **Step 5: Replace the hard-coded dashboard user list with repository data** + +```python +with request.app.state.session_factory() as session: + users = UserRepository(session).list_all() +return templates.TemplateResponse( + request, + "dashboard.html", + {"users": users, "csrf_token": ensure_csrf_token(request)}, +) +``` + +- [ ] **Step 6: Run web tests** + +Run: `pytest tests/web/test_users.py tests/web/test_auth.py -v` + +Expected: PASS, including explicit assertion that passwords never occur in response HTML. + +- [ ] **Step 7: Commit** + +```bash +git add app/db/repositories.py app/web tests/web/test_users.py +git commit -m "feat: add encrypted sync user management" +``` + +## Task 6: Package the foundation as a local Docker service + +**Files:** +- Create: `Dockerfile` +- Create: `docker-compose.example.yml` +- Create: `.env.example` +- Modify: `app/main.py` + +**Interfaces:** +- Produces a container exposing FastAPI on port `8080` and persisting `/data`. + +- [ ] **Step 1: Add deterministic startup command** + +```dockerfile +# Dockerfile +FROM python:3.12-slim +WORKDIR /app +COPY pyproject.toml /app/ +RUN pip install --no-cache-dir . +COPY app /app/app +RUN mkdir -p /data && chmod 700 /data +ENV DATA_DIR=/data +EXPOSE 8080 +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"] +``` + +- [ ] **Step 2: Add example environment and compose file** + +```env +# .env.example +ADMIN_PASSWORD=change-me +SECRET_KEY=replace-with-at-least-16-random-characters +CREDENTIAL_ENCRYPTION_KEY=replace-with-a-valid-fernet-key +SYNC_INTERVAL_MINUTES=5 +DATA_DIR=/data +DATABASE_URL=sqlite:////data/app.db +``` + +```yaml +# docker-compose.example.yml +services: + sync: + build: . + env_file: .env + ports: + - "8080:8080" + volumes: + - ./data:/data + restart: unless-stopped +``` + +- [ ] **Step 3: Build the image** + +Run: `docker build -t mywhoosh-garmin-sync:test .` + +Expected: successful image build. + +- [ ] **Step 4: Run a container smoke test** + +Run with a real generated Fernet key and test-only secrets: + +```bash +docker run --rm -d --name mywhoosh-garmin-sync-test \ + -p 18080:8080 \ + -e ADMIN_PASSWORD=admin-secret \ + -e SECRET_KEY=0123456789abcdef0123456789abcdef \ + -e CREDENTIAL_ENCRYPTION_KEY="$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')" \ + -v "$PWD/.tmp-data:/data" \ + mywhoosh-garmin-sync:test +``` + +Run: `curl -fsS http://127.0.0.1:18080/healthz` + +Expected: `{"status":"ok"}`. + +Then run: `docker stop mywhoosh-garmin-sync-test` + +- [ ] **Step 5: Run the foundation regression suite** + +Run: `pytest tests/test_config.py tests/db tests/security tests/web -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add Dockerfile docker-compose.example.yml .env.example app/main.py +git commit -m "build: package local admin service" +``` diff --git a/docs/superpowers/plans/2026-08-15-service-clients.md b/docs/superpowers/plans/2026-08-15-service-clients.md new file mode 100644 index 0000000..f374d00 --- /dev/null +++ b/docs/superpowers/plans/2026-08-15-service-clients.md @@ -0,0 +1,716 @@ +# MyWhoosh and Garmin Service Clients Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Implement isolated per-user MyWhoosh and Garmin clients with persistent tokenstores, mockable network boundaries, MyWhoosh direct API login/activity download, Garmin `import_activity()`, duplicate handling, and MFA signaling. + +**Architecture:** Keep external integrations behind narrow protocols so the sync engine never depends directly on `httpx` or `garminconnect`. MyWhoosh uses an injected `httpx.AsyncClient` and a per-user JSON tokenstore. Garmin uses an injected client factory and per-user `python-garminconnect` token directory; blocking Garmin calls are later run through `asyncio.to_thread` by the sync layer. + +**Tech Stack:** Python 3.12, httpx, python-garminconnect, pytest, pytest-asyncio. + +## Global Constraints + +- Do not automate or defeat MyWhoosh CAPTCHA/reCAPTCHA. +- Follow the direct Android-style API login flow used by the reference `jdelrue/mywhoosh2garmin` project. +- Cache MyWhoosh tokens per user under `/data/tokens//mywhoosh.json`. +- Cache Garmin tokens per user under `/data/tokens//garmin/` using the library's tokenstore behavior. +- Authentication/API changes must become explicit integration/authentication errors, not uncontrolled retries. +- Garmin activity transfer must use `import_activity()`, not `upload_activity()`. +- Known duplicate Garmin responses are successful terminal outcomes. +- Garmin MFA must raise a dedicated exception so the UI can collect a one-time code. +- Never log passwords, bearer tokens, Garmin tokens, or MFA codes. + +--- + +## File Structure + +```text +app/mywhoosh/ + __init__.py + models.py + tokenstore.py + client.py +app/garmin/ + __init__.py + uploader.py +tests/mywhoosh/ + test_tokenstore.py + test_client_auth.py + test_client_activities.py +tests/garmin/ + test_uploader.py +``` + +## Task 1: Implement MyWhoosh models and tokenstore + +**Files:** +- Create: `app/mywhoosh/models.py` +- Create: `app/mywhoosh/tokenstore.py` +- Create: `tests/mywhoosh/test_tokenstore.py` + +**Interfaces:** +- Produces `MyWhooshToken`, `MyWhooshActivity`, `MyWhooshTokenStore.load()`, `save()`, `clear()`. + +- [ ] **Step 1: Write tokenstore tests** + +```python +# tests/mywhoosh/test_tokenstore.py +from pathlib import Path + +from app.mywhoosh.models import MyWhooshToken +from app.mywhoosh.tokenstore import MyWhooshTokenStore + + +def test_tokenstore_round_trip_and_permissions(tmp_path: Path) -> None: + store = MyWhooshTokenStore(tmp_path / "tokens" / "7" / "mywhoosh.json") + token = MyWhooshToken(access_token="access", refresh_token="refresh", whoosh_id="whoosh-7") + store.save(token) + + assert store.load() == token + assert oct(store.path.stat().st_mode & 0o777) == "0o600" + + +def test_missing_token_returns_none(tmp_path: Path) -> None: + store = MyWhooshTokenStore(tmp_path / "missing.json") + assert store.load() is None +``` + +- [ ] **Step 2: Run and verify failure** + +Run: `pytest tests/mywhoosh/test_tokenstore.py -v` + +Expected: import failure. + +- [ ] **Step 3: Implement models** + +```python +# app/mywhoosh/models.py +from dataclasses import dataclass +from datetime import datetime + + +@dataclass(frozen=True) +class MyWhooshToken: + access_token: str + refresh_token: str | None + whoosh_id: str | None + + +@dataclass(frozen=True) +class MyWhooshActivity: + id: str + title: str + activity_file_id: str + started_at: datetime | None +``` + +- [ ] **Step 4: Implement atomic JSON token persistence** + +```python +# app/mywhoosh/tokenstore.py +import json +import os +from pathlib import Path + +from app.mywhoosh.models import MyWhooshToken + + +class MyWhooshTokenStore: + def __init__(self, path: Path) -> None: + self.path = path + + def load(self) -> MyWhooshToken | None: + try: + raw = json.loads(self.path.read_text("utf-8")) + except FileNotFoundError: + return None + return MyWhooshToken( + access_token=raw["access_token"], + refresh_token=raw.get("refresh_token"), + whoosh_id=raw.get("whoosh_id"), + ) + + def save(self, token: MyWhooshToken) -> None: + self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + tmp = self.path.with_suffix(".tmp") + tmp.write_text( + json.dumps( + { + "access_token": token.access_token, + "refresh_token": token.refresh_token, + "whoosh_id": token.whoosh_id, + }, + indent=2, + ), + "utf-8", + ) + os.chmod(tmp, 0o600) + tmp.replace(self.path) + os.chmod(self.path, 0o600) + + def clear(self) -> None: + self.path.unlink(missing_ok=True) +``` + +- [ ] **Step 5: Run tests** + +Run: `pytest tests/mywhoosh/test_tokenstore.py -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add app/mywhoosh/models.py app/mywhoosh/tokenstore.py tests/mywhoosh/test_tokenstore.py +git commit -m "feat: add MyWhoosh token persistence" +``` + +## Task 2: Implement MyWhoosh login and cached-session recovery + +**Files:** +- Create: `app/mywhoosh/client.py` +- Create: `tests/mywhoosh/test_client_auth.py` +- Modify: `pyproject.toml` + +**Interfaces:** +- Produces exceptions `MyWhooshAuthError`, `MyWhooshTransientError`, `MyWhooshIntegrationError`. +- Produces `MyWhooshClient.ensure_authenticated(email: str, password: str) -> None`. +- Login endpoint: `https://services.mywhoosh.com/http-service/api/login`. +- Login payload fields: `Username`, `Password`, `Platform="Android"`, `Action=1001`, random `CorrelationId`, random `DeviceId`, `Authorization=""`. + +- [ ] **Step 1: Add test dependencies** + +Add to `pyproject.toml` runtime dependencies: + +```toml +"httpx>=0.27,<1", +``` + +and test dependencies: + +```toml +"pytest-asyncio>=0.24,<1", +``` + +- [ ] **Step 2: Write authentication tests using `httpx.MockTransport`** + +```python +# tests/mywhoosh/test_client_auth.py +import httpx +import pytest + +from app.mywhoosh.client import MyWhooshClient, MyWhooshAuthError +from app.mywhoosh.models import MyWhooshToken +from app.mywhoosh.tokenstore import MyWhooshTokenStore + + +@pytest.mark.asyncio +async def test_login_saves_access_refresh_and_whoosh_id(tmp_path) -> None: + async def handler(request: httpx.Request) -> httpx.Response: + assert request.url.path == "/http-service/api/login" + return httpx.Response( + 200, + json={ + "Success": True, + "AccessToken": "new-access", + "RefreshToken": "new-refresh", + "WhooshId": "w-1", + }, + ) + + store = MyWhooshTokenStore(tmp_path / "mywhoosh.json") + client = MyWhooshClient(store, http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler))) + await client.login("rider@example.com", "secret") + assert store.load() == MyWhooshToken("new-access", "new-refresh", "w-1") + + +@pytest.mark.asyncio +async def test_invalid_credentials_raise_auth_error(tmp_path) -> None: + async def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"Success": False, "Message": "Invalid credentials"}) + + client = MyWhooshClient( + MyWhooshTokenStore(tmp_path / "mywhoosh.json"), + http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)), + ) + with pytest.raises(MyWhooshAuthError): + await client.login("rider@example.com", "bad") +``` + +- [ ] **Step 3: Run and verify failure** + +Run: `pytest tests/mywhoosh/test_client_auth.py -v` + +Expected: missing client implementation. + +- [ ] **Step 4: Implement exception taxonomy and login** + +```python +# app/mywhoosh/client.py +from __future__ import annotations + +import uuid + +import httpx + +from app.mywhoosh.models import MyWhooshToken +from app.mywhoosh.tokenstore import MyWhooshTokenStore + +LOGIN_URL = "https://services.mywhoosh.com/http-service/api/login" +ACTIVITIES_BASE = "https://service14.mywhoosh.com/v2/" + + +class MyWhooshError(RuntimeError): + pass + + +class MyWhooshAuthError(MyWhooshError): + pass + + +class MyWhooshTransientError(MyWhooshError): + pass + + +class MyWhooshIntegrationError(MyWhooshError): + pass + + +class MyWhooshClient: + def __init__(self, token_store: MyWhooshTokenStore, http_client: httpx.AsyncClient | None = None) -> None: + self.token_store = token_store + self.http = http_client or httpx.AsyncClient(timeout=30.0) + self.token = token_store.load() + + async def login(self, email: str, password: str) -> None: + payload = { + "Username": email, + "Password": password, + "Platform": "Android", + "Action": 1001, + "CorrelationId": str(uuid.uuid4()), + "DeviceId": str(uuid.uuid4()), + "Authorization": "", + } + try: + response = await self.http.post(LOGIN_URL, json=payload) + except httpx.TransportError as exc: + raise MyWhooshTransientError("MyWhoosh login request failed") from exc + if response.status_code >= 500: + raise MyWhooshTransientError(f"MyWhoosh login returned HTTP {response.status_code}") + if response.status_code >= 400: + raise MyWhooshAuthError(f"MyWhoosh login returned HTTP {response.status_code}") + try: + body = response.json() + except ValueError as exc: + raise MyWhooshIntegrationError("MyWhoosh login returned invalid JSON") from exc + if body.get("Success") is not True or not body.get("AccessToken"): + raise MyWhooshAuthError(str(body.get("Message") or "MyWhoosh login failed")) + self.token = MyWhooshToken( + access_token=str(body["AccessToken"]), + refresh_token=str(body["RefreshToken"]) if body.get("RefreshToken") else None, + whoosh_id=str(body["WhooshId"]) if body.get("WhooshId") else None, + ) + self.token_store.save(self.token) +``` + +- [ ] **Step 5: Implement `ensure_authenticated` as cache-first validation** + +Do not invent a refresh endpoint. Validate cached tokens using the normal activities request; on `401/403`, clear the cache, login once, and continue. The later `list_activities()` task provides the request method. Expose the intended behavior now: + +```python +async def ensure_authenticated(self, email: str, password: str) -> None: + if self.token is None: + await self.login(email, password) +``` + +Task 3 extends this with one retry after an unauthorized activities response. + +- [ ] **Step 6: Run authentication tests** + +Run: `pytest tests/mywhoosh/test_client_auth.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add pyproject.toml app/mywhoosh/client.py tests/mywhoosh/test_client_auth.py +git commit -m "feat: add MyWhoosh API login" +``` + +## Task 3: Implement MyWhoosh activity listing and FIT download + +**Files:** +- Modify: `app/mywhoosh/client.py` +- Create: `tests/mywhoosh/test_client_activities.py` + +**Interfaces:** +- Produces `list_activities(email: str, password: str) -> list[MyWhooshActivity]`. +- Produces `download_fit(activity_file_id: str, email: str, password: str) -> bytes`. +- Activities endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/activities` with `{"sortDate":"DESC","page":N}`. +- Download endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/download-activity-file` with `{"fileId": activity_file_id}`; response `data` is a pre-signed URL which is fetched with GET. + +- [ ] **Step 1: Write paginated activity-list test** + +```python +@pytest.mark.asyncio +async def test_list_activities_paginates_and_normalizes(tmp_path) -> None: + calls = [] + + async def handler(request: httpx.Request) -> httpx.Response: + calls.append(str(request.url)) + if request.url.path.endswith("/activities"): + payload = json.loads(request.content) + page = payload["page"] + result = { + "data": { + "totalPages": 2, + "results": [{ + "id": f"a-{page}", + "title": f"Ride {page}", + "activityFileId": f"f-{page}", + "startDatetime": "2026-08-15T06:00:00.000Z", + }], + } + } + return httpx.Response(200, json=result) + raise AssertionError(request.url) +``` + +Preload the tokenstore with `access_token="cached"`; assert two normalized `MyWhooshActivity` values are returned. + +- [ ] **Step 2: Write expired-token reauthentication test** + +The mock transport sequence must return `401` for the first activities request, a successful login response, then `200` for the retried activities request. Assert login is attempted exactly once and the tokenstore contains the new access token. + +- [ ] **Step 3: Write FIT download test** + +Mock the download-activity-file endpoint to return `{"data":"https://signed.example/activity.fit"}`, then mock that URL to return bytes beginning with a valid FIT header. Assert `download_fit()` returns those exact bytes. + +- [ ] **Step 4: Implement one authenticated-request retry helper** + +```python +async def _authenticated_post(self, url: str, payload: dict, email: str, password: str) -> httpx.Response: + await self.ensure_authenticated(email, password) + for attempt in range(2): + assert self.token is not None + try: + response = await self.http.post( + url, + json=payload, + headers={"Authorization": f"Bearer {self.token.access_token}"}, + ) + except httpx.TransportError as exc: + raise MyWhooshTransientError("MyWhoosh request failed") from exc + if response.status_code not in {401, 403}: + if response.status_code >= 500: + raise MyWhooshTransientError(f"MyWhoosh returned HTTP {response.status_code}") + return response + if attempt == 0: + self.token_store.clear() + self.token = None + await self.login(email, password) + continue + raise MyWhooshAuthError("MyWhoosh session rejected after reauthentication") + raise AssertionError("unreachable") +``` + +- [ ] **Step 5: Implement list normalization and download** + +Parse `startDatetime` as UTC when present. Skip malformed activity rows only if they lack no stable `id` or `activityFileId`; otherwise surface JSON/schema failures as `MyWhooshIntegrationError` so API changes are visible. + +```python +async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes: + response = await self._authenticated_post( + ACTIVITIES_BASE + "rider/profile/download-activity-file", + {"fileId": activity_file_id}, + email, + password, + ) + if response.status_code >= 400: + raise MyWhooshIntegrationError(f"download metadata returned HTTP {response.status_code}") + url = response.json().get("data") + if not isinstance(url, str) or not url: + raise MyWhooshIntegrationError("MyWhoosh download response has no URL") + try: + fit_response = await self.http.get(url) + except httpx.TransportError as exc: + raise MyWhooshTransientError("FIT download failed") from exc + if fit_response.status_code >= 500: + raise MyWhooshTransientError(f"FIT host returned HTTP {fit_response.status_code}") + if fit_response.status_code >= 400: + raise MyWhooshIntegrationError(f"FIT host returned HTTP {fit_response.status_code}") + return fit_response.content +``` + +- [ ] **Step 6: Run MyWhoosh tests** + +Run: `pytest tests/mywhoosh -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add app/mywhoosh/client.py tests/mywhoosh/test_client_activities.py +git commit -m "feat: fetch MyWhoosh activities and FIT files" +``` + +## Task 4: Implement Garmin import adapter with duplicate and MFA handling + +**Files:** +- Create: `app/garmin/uploader.py` +- Create: `tests/garmin/test_uploader.py` +- Modify: `pyproject.toml` + +**Interfaces:** +- Produces `GarminUploader.import_fit(fit_path: Path, mfa_code: str | None = None) -> UploadResult`. +- Produces exceptions `GarminUploadBlocked`, `GarminAuthError`, `GarminTransientError`. +- Uses `client.login(tokenstore_path)` and `client.import_activity(activity_path)`. + +- [ ] **Step 1: Add Garmin dependency** + +Add to runtime dependencies: + +```toml +"garminconnect>=0.2,<1", +``` + +- [ ] **Step 2: Write fake-client tests based on the previously working uploader pattern** + +```python +# tests/garmin/test_uploader.py +from pathlib import Path + +import pytest + +from app.garmin.uploader import GarminUploadBlocked, GarminUploader + + +class FakeGarmin: + def __init__(self, *args, prompt_mfa=None, import_result=None, login_error=None, import_error=None, **kwargs): + self.prompt_mfa = prompt_mfa + self.import_result = import_result or {"activityId": 42} + self.login_error = login_error + self.import_error = import_error + self.login_path = None + + def login(self, tokenstore=None): + self.login_path = tokenstore + if self.login_error: + raise self.login_error + + def import_activity(self, activity_path: str): + if self.import_error: + raise self.import_error + return self.import_result +``` + +Add these concrete tests below the fake client: + +```python +def test_successful_import_returns_activity_id(tmp_path: Path) -> None: + uploader = GarminUploader( + email="g@example.com", + password="pw", + tokenstore=tmp_path / "garmin", + client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_result={"activityId": 42}), + ) + result = uploader.import_fit(tmp_path / "ride.fit") + assert result.status == "imported" + assert result.garmin_activity_id == "42" + + +def test_duplicate_is_terminal_success(tmp_path: Path) -> None: + uploader = GarminUploader( + email="g@example.com", + password="pw", + tokenstore=tmp_path / "garmin", + client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_error=RuntimeError("409 duplicate")), + ) + result = uploader.import_fit(tmp_path / "ride.fit") + assert result.duplicate is True + assert result.status == "duplicate" + + +def test_mfa_without_code_is_blocked(tmp_path: Path) -> None: + class MfaGarmin(FakeGarmin): + def login(self, tokenstore=None): + self.prompt_mfa() + + uploader = GarminUploader( + email="g@example.com", + password="pw", + tokenstore=tmp_path / "garmin", + client_factory=MfaGarmin, + ) + with pytest.raises(GarminUploadBlocked): + uploader.import_fit(tmp_path / "ride.fit") + + +def test_mfa_code_is_returned_only_to_prompt(tmp_path: Path) -> None: + seen = [] + + class MfaGarmin(FakeGarmin): + def login(self, tokenstore=None): + seen.append(self.prompt_mfa()) + + uploader = GarminUploader( + email="g@example.com", + password="pw", + tokenstore=tmp_path / "garmin", + client_factory=MfaGarmin, + ) + uploader.import_fit(tmp_path / "ride.fit", mfa_code="123456") + assert seen == ["123456"] +``` + +- [ ] **Step 3: Run and verify failure** + +Run: `pytest tests/garmin/test_uploader.py -v` + +Expected: import failure. + +- [ ] **Step 4: Implement the adapter** + +```python +# app/garmin/uploader.py +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Callable, Protocol + + +class GarminClientProtocol(Protocol): + def login(self, tokenstore: str | None = None) -> Any: ... + def import_activity(self, activity_path: str) -> Any: ... + + +@dataclass(frozen=True) +class UploadResult: + status: str + duplicate: bool + garmin_activity_id: str | None + raw_response: Any + + +class GarminUploadBlocked(RuntimeError): + pass + + +class GarminAuthError(RuntimeError): + pass + + +class GarminTransientError(RuntimeError): + pass +``` + +Implement the uploader fully: + +```python +class GarminUploader: + def __init__( + self, + *, + email: str, + password: str, + tokenstore: Path, + client_factory: Callable[..., GarminClientProtocol] | None = None, + ) -> None: + self.email = email + self.password = password + self.tokenstore = tokenstore + self.client_factory = client_factory + self._client: GarminClientProtocol | None = None + self._mfa_code: str | None = None + + def import_fit(self, fit_path: Path, mfa_code: str | None = None) -> UploadResult: + self._mfa_code = mfa_code + try: + client = self._ensure_client() + try: + response = client.import_activity(str(fit_path)) + except Exception as exc: + if _looks_duplicate_error(exc): + return UploadResult("duplicate", True, None, str(exc)) + text = str(exc).lower() + if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")): + raise GarminTransientError("Garmin import failed transiently") from exc + raise + return UploadResult("imported", False, _extract_activity_id(response), response) + finally: + self._mfa_code = None + + def _ensure_client(self) -> GarminClientProtocol: + if self._client is not None: + return self._client + self.tokenstore.mkdir(parents=True, exist_ok=True) + factory = self.client_factory or _default_garmin_factory + client = factory(self.email, self.password, prompt_mfa=self._prompt_mfa) + try: + client.login(str(self.tokenstore)) + except GarminUploadBlocked: + raise + except Exception as exc: + text = str(exc).lower() + if "mfa" in text: + raise GarminUploadBlocked("Garmin MFA is required") from exc + if any(token in text for token in ("password", "credential", "unauthorized", "401")): + raise GarminAuthError("Garmin authentication failed") from exc + if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")): + raise GarminTransientError("Garmin login failed transiently") from exc + raise GarminAuthError("Garmin login failed") from exc + self._client = client + return client + + def _prompt_mfa(self) -> str: + if self._mfa_code: + return self._mfa_code + raise GarminUploadBlocked("Garmin requested MFA") + + +def _default_garmin_factory(*args: Any, **kwargs: Any) -> GarminClientProtocol: + from garminconnect import Garmin + + return Garmin(*args, **kwargs) +``` + +- [ ] **Step 5: Keep duplicate and response-ID extraction deterministic** + +Use these helpers: + +```python +def _looks_duplicate_error(exc: Exception) -> bool: + text = str(exc).lower() + return any(token in text for token in ("duplicate", "already exists", "409")) + + +def _extract_activity_id(response: Any) -> str | None: + if not isinstance(response, dict): + return None + candidates = [response.get("activityId"), response.get("activity_id"), response.get("id")] + detailed = response.get("detailedImportResult") + if isinstance(detailed, dict): + candidates.extend([detailed.get("uploadId"), detailed.get("activityId")]) + for key in ("successes", "success", "importedActivities"): + items = response.get(key) + if isinstance(items, list) and items and isinstance(items[0], dict): + candidates.extend([items[0].get("activityId"), items[0].get("id")]) + return next((str(value) for value in candidates if value is not None), None) +``` + +- [ ] **Step 6: Run Garmin tests** + +Run: `pytest tests/garmin/test_uploader.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add pyproject.toml app/garmin/uploader.py tests/garmin/test_uploader.py +git commit -m "feat: import FIT activities into Garmin" +``` diff --git a/docs/superpowers/plans/2026-08-15-sync-scheduler-web.md b/docs/superpowers/plans/2026-08-15-sync-scheduler-web.md new file mode 100644 index 0000000..4d9d1b2 --- /dev/null +++ b/docs/superpowers/plans/2026-08-15-sync-scheduler-web.md @@ -0,0 +1,804 @@ +# Sync Engine, Scheduler, and Operational UI Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Integrate the database, MyWhoosh client, FIT rewriter, Garmin importer, scheduler, MFA workflow, retry behavior, and operational admin pages into a resilient multi-user sync service. + +**Architecture:** A `SyncManager` owns per-user `asyncio.Lock` instances and executes a durable activity state machine. External clients are injected via factories for tests. A lightweight FastAPI lifespan scheduler triggers syncs at the configured interval; different users run concurrently, while each user's pipeline is serialized. + +**Tech Stack:** Python 3.12, asyncio, FastAPI lifespan, SQLAlchemy, HTMX, Jinja2, pytest/pytest-asyncio. + +## Global Constraints + +- Multiple users sync independently and may run concurrently. +- At most one sync may run for a given user at a time. +- Manual sync and scheduled sync use the same pipeline and lock. +- Durable activity stages are `discovered`, `downloaded`, `converted`, `imported`, `duplicate`, `failed` with `last_completed_stage` retained on failure. +- `imported` and `duplicate` are terminal. +- Transient network/server failures retry at most once in a run; later attempts occur on future scheduler ticks. +- Invalid MyWhoosh/Garmin credentials and Garmin MFA set `action_required`. +- Corrupt/unsupported FIT is non-retryable per activity. +- Failure of one user or one activity must never stop other users. +- Original and converted FIT files remain on disk in v1. +- MFA codes are never persisted or logged. + +--- + +## File Structure + +```text +app/sync/ + __init__.py + states.py + manager.py + scheduler.py +app/web/ + operations.py + templates/ + dashboard.html + users/detail.html + system.html + fragments/user_card.html + fragments/sync_result.html + fragments/mfa_form.html +app/db/ + repositories.py +tests/sync/ + fakes.py + test_manager.py + test_concurrency.py + test_scheduler.py +tests/web/ + test_operations.py + test_mfa.py +``` + +## Task 1: Add durable activity/sync-run repository operations + +**Files:** +- Modify: `app/db/repositories.py` +- Create: `tests/db/test_sync_state.py` + +**Interfaces:** +- Produces methods to advance activity stages, mark failures without losing `last_completed_stage`, list pending activities, and create/finalize sync runs. + +- [ ] **Step 1: Write failing state-transition tests** + +```python +# tests/db/test_sync_state.py +from app.db.models import ActivityStatus + + +def test_failure_retains_last_completed_stage(activity_repository, seeded_activity) -> None: + activity_repository.mark_downloaded(seeded_activity.id, "/data/activities/1/a/source.fit") + activity_repository.mark_failed(seeded_activity.id, "Garmin timeout", retryable=True) + activity = activity_repository.get(seeded_activity.id) + + assert activity.status == ActivityStatus.FAILED + assert activity.last_completed_stage == ActivityStatus.DOWNLOADED + assert activity.retryable is True + + +def test_converted_activity_is_pending_until_terminal(activity_repository, seeded_activity) -> None: + activity_repository.mark_converted(seeded_activity.id, "/data/activities/1/a/converted.fit") + ids = [item.id for item in activity_repository.list_pending_for_user(seeded_activity.user_id)] + assert seeded_activity.id in ids +``` + +- [ ] **Step 2: Run and verify failure** + +Run: `pytest tests/db/test_sync_state.py -v` + +Expected: missing repository methods. + +- [ ] **Step 3: Implement explicit transition methods** + +Add methods with these exact effects: + +```python +def mark_downloaded(self, activity_id: int, path: str) -> Activity: + activity = self._require(activity_id) + activity.source_fit_path = path + activity.status = ActivityStatus.DOWNLOADED + activity.last_completed_stage = ActivityStatus.DOWNLOADED + activity.last_error = None + activity.retryable = True + self.session.commit() + return activity + + +def mark_converted(self, activity_id: int, path: str) -> Activity: + activity = self._require(activity_id) + activity.converted_fit_path = path + activity.status = ActivityStatus.CONVERTED + activity.last_completed_stage = ActivityStatus.CONVERTED + activity.last_error = None + activity.retryable = True + self.session.commit() + return activity + + +def mark_imported(self, activity_id: int, garmin_activity_id: str | None) -> Activity: + activity = self._require(activity_id) + activity.status = ActivityStatus.IMPORTED + activity.last_completed_stage = ActivityStatus.IMPORTED + activity.garmin_activity_id = garmin_activity_id + activity.last_error = None + activity.retryable = False + self.session.commit() + return activity + + +def mark_duplicate(self, activity_id: int) -> Activity: + activity = self._require(activity_id) + activity.status = ActivityStatus.DUPLICATE + activity.last_completed_stage = ActivityStatus.DUPLICATE + activity.last_error = None + activity.retryable = False + self.session.commit() + return activity + + +def mark_failed(self, activity_id: int, error: str, *, retryable: bool) -> Activity: + activity = self._require(activity_id) + activity.status = ActivityStatus.FAILED + activity.last_error = error[:2000] + activity.retryable = retryable + self.session.commit() + return activity +``` + +`list_pending_for_user()` must exclude terminal states and include failed rows only when `retryable=True`. + +- [ ] **Step 4: Add `SyncRunRepository` create/finalize methods** + +`start(user_id)` creates `RUNNING`; `finish(...)` sets counts, `finished_at`, status, and optional summary error. Do not store exception tracebacks in SQLite. + +- [ ] **Step 5: Run DB state tests** + +Run: `pytest tests/db/test_sync_state.py -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add app/db/repositories.py tests/db/test_sync_state.py +git commit -m "feat: add durable sync state transitions" +``` + +## Task 2: Implement the single-user sync state machine + +**Files:** +- Create: `app/sync/states.py` +- Create: `app/sync/manager.py` +- Create: `tests/sync/fakes.py` +- Create: `tests/sync/test_manager.py` + +**Interfaces:** +- Produces `SyncManager.sync_user(user_id: int, mfa_code: str | None = None) -> SyncOutcome`. +- Constructor receives `session_factory`, `credential_cipher`, `settings`, `mywhoosh_factory`, `garmin_factory`, and `fit_converter`. +- `mywhoosh_factory(token_store: MyWhooshTokenStore) -> MyWhooshClient`. +- `garmin_factory(email: str, password: str, tokenstore: Path) -> GarminUploader`. +- `fit_converter(source_path: Path, output_path: Path) -> FitConversionResult`. + +- [ ] **Step 1: Define result models and fake integration factories** + +```python +# app/sync/states.py +from dataclasses import dataclass + + +@dataclass(frozen=True) +class SyncOutcome: + user_id: int + status: str + discovered: int + imported: int + skipped: int + failed: int + message: str | None = None +``` + +Implement concrete fakes: + +```python +# tests/sync/fakes.py +from app.garmin.uploader import UploadResult + + +class FakeMyWhooshClient: + def __init__(self, activities, fit_bytes: bytes) -> None: + self.activities = activities + self.fit_bytes = fit_bytes + self.list_calls = 0 + self.download_calls = 0 + + async def list_activities(self, email: str, password: str): + self.list_calls += 1 + return list(self.activities) + + async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes: + self.download_calls += 1 + return self.fit_bytes + + +class FakeGarminUploader: + def __init__(self, result: UploadResult | None = None, error: Exception | None = None) -> None: + self.result = result or UploadResult("imported", False, "g-1", {"activityId": "g-1"}) + self.error = error + self.calls = 0 + + def import_fit(self, fit_path, mfa_code=None): + self.calls += 1 + if self.error is not None: + raise self.error + return self.result +``` + +- [ ] **Step 2: Write the happy-path test** + +```python +@pytest.mark.asyncio +async def test_new_activity_downloads_converts_and_imports(manager, seeded_user, tmp_path) -> None: + outcome = await manager.sync_user(seeded_user.id) + + assert outcome.discovered == 1 + assert outcome.imported == 1 + assert outcome.failed == 0 + activity = load_only_activity(seeded_user.id) + assert activity.status == ActivityStatus.IMPORTED + assert Path(activity.source_fit_path).exists() + assert Path(activity.converted_fit_path).exists() +``` + +- [ ] **Step 3: Write resume tests before implementation** + +```python +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("status", "last_stage", "expected_downloads", "expected_conversions", "expected_imports"), + [ + (ActivityStatus.DOWNLOADED, ActivityStatus.DOWNLOADED, 0, 1, 1), + (ActivityStatus.CONVERTED, ActivityStatus.CONVERTED, 0, 0, 1), + (ActivityStatus.IMPORTED, ActivityStatus.IMPORTED, 0, 0, 0), + (ActivityStatus.FAILED, ActivityStatus.CONVERTED, 0, 0, 1), + ], +) +async def test_resume_from_durable_stage( + manager_factory, seeded_activity_factory, status, last_stage, + expected_downloads, expected_conversions, expected_imports, +) -> None: + activity = seeded_activity_factory(status=status, last_completed_stage=last_stage, retryable=True) + manager, mywhoosh, converter, garmin = manager_factory(activity) + await manager.sync_user(activity.user_id) + assert mywhoosh.download_calls == expected_downloads + assert converter.calls == expected_conversions + assert garmin.calls == expected_imports +``` + +- [ ] **Step 4: Run and verify failure** + +Run: `pytest tests/sync/test_manager.py -v` + +Expected: missing manager. + +- [ ] **Step 5: Implement per-activity filesystem layout and state machine** + +Use paths: + +```python +activity_dir = settings.activities_dir / str(user.id) / activity.mywhoosh_activity_id +source_path = activity_dir / "source.fit" +converted_path = activity_dir / "edge-1030-plus.fit" +``` + +Create per-user integration instances from decrypted credentials and isolated token paths: + +```python +mw_email = self.credential_cipher.decrypt(user.mywhoosh_email_enc) +mw_password = self.credential_cipher.decrypt(user.mywhoosh_password_enc) +garmin_email = self.credential_cipher.decrypt(user.garmin_email_enc) +garmin_password = self.credential_cipher.decrypt(user.garmin_password_enc) + +token_dir = self.settings.tokens_dir / str(user.id) +mywhoosh = self.mywhoosh_factory(MyWhooshTokenStore(token_dir / "mywhoosh.json")) +garmin = self.garmin_factory(garmin_email, garmin_password, token_dir / "garmin") +remote_activities = await mywhoosh.list_activities(mw_email, mw_password) +``` + +For each remote activity, call `get_or_create_discovered(...)`, then resume from `activity.last_completed_stage` when `activity.status == FAILED`; otherwise use `activity.status`. + +Core sequence: + +```python +if stage == ActivityStatus.DISCOVERED: + fit_bytes = await mywhoosh.download_fit(remote.activity_file_id, mw_email, mw_password) + activity_dir.mkdir(parents=True, exist_ok=True) + source_path.write_bytes(fit_bytes) + repo.mark_downloaded(activity.id, str(source_path)) + +if stage in {ActivityStatus.DOWNLOADED}: + fit_converter(source_path, converted_path) + repo.mark_converted(activity.id, str(converted_path)) + +if stage in {ActivityStatus.CONVERTED}: + upload = await asyncio.to_thread(garmin.import_fit, converted_path, mfa_code) + if upload.duplicate: + repo.mark_duplicate(activity.id) + else: + repo.mark_imported(activity.id, upload.garmin_activity_id) +``` + +After each repository transition, update the local `stage` variable from the returned record so resume behavior is deterministic. + +- [ ] **Step 6: Implement exception mapping** + +Map exceptions with explicit user connection-state updates: + +```python +except MyWhooshTransientError as exc: + user.health_state = HealthState.DEGRADED + user.mywhoosh_state = "error" + repo.mark_failed(activity.id, str(exc), retryable=True) +except MyWhooshAuthError as exc: + user.health_state = HealthState.ACTION_REQUIRED + user.mywhoosh_state = "auth_required" + user.action_reason = "mywhoosh_auth_required" + stop_user_run = True +except MyWhooshIntegrationError as exc: + user.health_state = HealthState.ACTION_REQUIRED + user.mywhoosh_state = "integration_error" + user.action_reason = "mywhoosh_integration_changed" + stop_user_run = True +except GarminUploadBlocked: + user.health_state = HealthState.ACTION_REQUIRED + user.garmin_state = "mfa_required" + user.action_reason = "garmin_mfa_required" + stop_user_run = True +except GarminAuthError as exc: + user.health_state = HealthState.ACTION_REQUIRED + user.garmin_state = "auth_required" + user.action_reason = "garmin_auth_required" + stop_user_run = True +except GarminTransientError as exc: + user.health_state = HealthState.DEGRADED + user.garmin_state = "error" + repo.mark_failed(activity.id, str(exc), retryable=True) +except FitFormatError as exc: + repo.mark_failed(activity.id, str(exc), retryable=False) +``` + +On successful MyWhoosh listing set `mywhoosh_state="connected"`; on successful Garmin import set `garmin_state="connected"`. Persist the user after each connection-state change. Unexpected exceptions mark the run/user `degraded` and log only exception class plus sanitized message. + +- [ ] **Step 7: Run manager tests** + +Run: `pytest tests/sync/test_manager.py -v` + +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add app/sync/states.py app/sync/manager.py tests/sync +git commit -m "feat: add resumable per-user sync pipeline" +``` + +## Task 3: Add per-user locks and cross-user isolation + +**Files:** +- Modify: `app/sync/manager.py` +- Create: `tests/sync/test_concurrency.py` + +**Interfaces:** +- Produces `SyncAlreadyRunning` and ensures only one active `sync_user()` call per user. + +- [ ] **Step 1: Write concurrency tests** + +```python +@pytest.mark.asyncio +async def test_same_user_cannot_run_twice(manager, seeded_user) -> None: + first_started = asyncio.Event() + release_first = asyncio.Event() + manager.test_hooks = SyncTestHooks(first_started=first_started, release=release_first) + + first = asyncio.create_task(manager.sync_user(seeded_user.id)) + await first_started.wait() + + with pytest.raises(SyncAlreadyRunning): + await manager.sync_user(seeded_user.id) + + release_first.set() + await first + + +@pytest.mark.asyncio +async def test_different_users_can_run_concurrently(manager, user_a, user_b) -> None: + results = await asyncio.gather(manager.sync_user(user_a.id), manager.sync_user(user_b.id)) + assert {result.user_id for result in results} == {user_a.id, user_b.id} +``` + +Do not leave production-only `test_hooks`; instead inject a fake MyWhoosh client whose `list_activities()` blocks on test events. + +- [ ] **Step 2: Run and verify failure** + +Run: `pytest tests/sync/test_concurrency.py -v` + +Expected: same-user duplicate execution is not yet blocked. + +- [ ] **Step 3: Implement lock registry** + +```python +class SyncAlreadyRunning(RuntimeError): + pass + + +class SyncManager: + def __init__(...): + self._locks: dict[int, asyncio.Lock] = {} + self._locks_guard = asyncio.Lock() + + async def _lock_for(self, user_id: int) -> asyncio.Lock: + async with self._locks_guard: + return self._locks.setdefault(user_id, asyncio.Lock()) + + async def sync_user(self, user_id: int, mfa_code: str | None = None) -> SyncOutcome: + lock = await self._lock_for(user_id) + if lock.locked(): + raise SyncAlreadyRunning(f"sync already running for user {user_id}") + async with lock: + return await self._sync_user_locked(user_id, mfa_code) +``` + +- [ ] **Step 4: Add a `sync_all_enabled()` isolation method** + +```python +async def sync_all_enabled(self) -> list[SyncOutcome | Exception]: + user_ids = self._load_enabled_user_ids() + return await asyncio.gather( + *(self.sync_user(user_id) for user_id in user_ids), + return_exceptions=True, + ) +``` + +A failure for one user must appear as one list element and must not cancel sibling jobs. + +- [ ] **Step 5: Run concurrency tests** + +Run: `pytest tests/sync/test_concurrency.py -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add app/sync/manager.py tests/sync/test_concurrency.py +git commit -m "feat: isolate concurrent user syncs" +``` + +## Task 4: Add the periodic scheduler through FastAPI lifespan + +**Files:** +- Create: `app/sync/scheduler.py` +- Modify: `app/main.py` +- Create: `tests/sync/test_scheduler.py` + +**Interfaces:** +- Produces `SyncScheduler.start()`, `stop()`, `run_once()`, `last_tick`, `next_tick`. +- Scheduler interval is `Settings.sync_interval_minutes`. + +- [ ] **Step 1: Write scheduler test with a short injected interval** + +```python +@pytest.mark.asyncio +async def test_scheduler_calls_sync_all_and_survives_failure() -> None: + fake = FakeSyncManager(results=[RuntimeError("one user failed")]) + scheduler = SyncScheduler(fake, interval_seconds=0.01) + await scheduler.start() + await asyncio.sleep(0.035) + await scheduler.stop() + assert fake.calls >= 2 + assert scheduler.last_tick is not None +``` + +- [ ] **Step 2: Run and verify failure** + +Run: `pytest tests/sync/test_scheduler.py -v` + +Expected: missing scheduler. + +- [ ] **Step 3: Implement scheduler loop** + +```python +class SyncScheduler: + def __init__(self, manager, *, interval_seconds: float) -> None: + self.manager = manager + self.interval_seconds = interval_seconds + self._task: asyncio.Task | None = None + self._stop = asyncio.Event() + self.last_tick = None + self.next_tick = None + + async def run_once(self) -> None: + self.last_tick = datetime.now(timezone.utc) + await self.manager.sync_all_enabled() + self.next_tick = datetime.now(timezone.utc) + timedelta(seconds=self.interval_seconds) + + async def _run(self) -> None: + while not self._stop.is_set(): + await self.run_once() + try: + await asyncio.wait_for(self._stop.wait(), timeout=self.interval_seconds) + except TimeoutError: + pass +``` + +`stop()` sets the event and awaits the task. Never allow one `sync_all_enabled()` exception to kill the loop; log it and continue. + +- [ ] **Step 4: Wire into FastAPI lifespan** + +Build the concrete `SyncManager` once during app startup, store it on `app.state.sync_manager`, create `SyncScheduler(... interval_minutes * 60)`, start it, and stop it during lifespan shutdown. + +- [ ] **Step 5: Run scheduler tests** + +Run: `pytest tests/sync/test_scheduler.py -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add app/sync/scheduler.py app/main.py tests/sync/test_scheduler.py +git commit -m "feat: schedule periodic user synchronization" +``` + +## Task 5: Add dashboard/manual sync/system operational routes + +**Files:** +- Create: `app/web/operations.py` +- Modify: `app/web/routes.py` +- Modify: `app/web/templates/dashboard.html` +- Create: `app/web/templates/system.html` +- Create: `app/web/templates/fragments/sync_result.html` +- Create: `tests/web/test_operations.py` + +**Interfaces:** +- Routes: `POST /users/{id}/sync`, `POST /sync-all`, `GET /system`. +- Manual actions use the same `SyncManager` instance and lock as the scheduler. + +- [ ] **Step 1: Write manual-sync tests** + +```python +def test_manual_sync_calls_shared_manager(authenticated_client, fake_sync_manager) -> None: + response = authenticated_client.post( + "/users/1/sync", + data={"csrf_token": authenticated_client.csrf_token}, + ) + assert response.status_code == 200 + assert fake_sync_manager.user_calls == [1] + + +def test_manual_sync_reports_already_running(authenticated_client, fake_sync_manager) -> None: + fake_sync_manager.raise_already_running = True + response = authenticated_client.post( + "/users/1/sync", + data={"csrf_token": authenticated_client.csrf_token}, + ) + assert response.status_code == 409 + assert "already running" in response.text.lower() +``` + +- [ ] **Step 2: Run and verify failure** + +Run: `pytest tests/web/test_operations.py -v` + +Expected: routes missing. + +- [ ] **Step 3: Implement routes with admin and CSRF checks** + +Each state-changing route must execute in this order: + +```python +require_admin(request) +validate_csrf(request, csrf_token) +``` + +Then call `await request.app.state.sync_manager.sync_user(user_id)` or `sync_all_enabled()`. + +- [ ] **Step 4: Expand dashboard data** + +Add a repository projection that contains only safe display fields: + +```python +@dataclass(frozen=True) +class UserDashboardRow: + id: int + name: str + enabled: bool + health_state: str + action_reason: str | None + last_sync_at: datetime | None + last_activity_name: str | None + last_activity_status: str | None + + +def dashboard_rows(self) -> list[UserDashboardRow]: + users = self.list_all() + rows = [] + for user in users: + last_run = self.session.scalar( + select(SyncRun).where(SyncRun.user_id == user.id).order_by(SyncRun.started_at.desc()).limit(1) + ) + last_activity = self.session.scalar( + select(Activity).where(Activity.user_id == user.id).order_by(Activity.created_at.desc()).limit(1) + ) + rows.append(UserDashboardRow( + id=user.id, + name=user.name, + enabled=user.enabled, + health_state=user.health_state.value, + action_reason=user.action_reason, + last_sync_at=last_run.finished_at if last_run else None, + last_activity_name=last_activity.activity_name if last_activity else None, + last_activity_status=last_activity.status.value if last_activity else None, + )) + return rows +``` + +Pass only these rows to `dashboard.html`. Render the MFA action only when `row.action_reason == "garmin_mfa_required"`. No decrypted credential is part of this projection. + +- [ ] **Step 5: Implement read-only system page** + +Expose application version, configured interval, scheduler `last_tick` and `next_tick`, user count, and activity count. The only action is a CSRF-protected `sync all now` POST. + +- [ ] **Step 6: Run tests** + +Run: `pytest tests/web/test_operations.py -v` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add app/web tests/web/test_operations.py +git commit -m "feat: add operational sync controls" +``` + +## Task 6: Add Garmin MFA lifecycle and failed-activity retry + +**Files:** +- Modify: `app/web/routes.py` +- Modify: `app/web/templates/users/detail.html` +- Create: `app/web/templates/fragments/mfa_form.html` +- Create: `tests/web/test_mfa.py` +- Modify: `app/sync/manager.py` + +**Interfaces:** +- Route: `POST /users/{id}/garmin-mfa` with one-time `code`. +- Route: `POST /activities/{id}/retry`. +- MFA code exists only in request memory and the immediate `sync_user(user_id, mfa_code=code)` call. + +- [ ] **Step 1: Write MFA lifecycle test** + +```python +def test_mfa_code_is_used_once_and_not_persisted(authenticated_client, fake_sync_manager, db_session) -> None: + response = authenticated_client.post( + "/users/1/garmin-mfa", + data={"csrf_token": authenticated_client.csrf_token, "code": "123456"}, + ) + assert response.status_code == 200 + assert fake_sync_manager.mfa_calls == [(1, "123456")] + + persisted_text = " ".join(str(row) for row in db_session.execute(text("select * from sync_runs")).all()) + assert "123456" not in persisted_text +``` + +- [ ] **Step 2: Write retry test for non-terminal failed activity** + +Assert the route changes a retryable failed activity back to `status=last_completed_stage`, clears `last_error`, then calls the user's normal sync. Reject retry for `retryable=False` with HTTP 409. + +- [ ] **Step 3: Implement MFA route** + +Validate code as a non-empty short string, never log it, and call: + +```python +outcome = await request.app.state.sync_manager.sync_user(user_id, mfa_code=code.strip()) +``` + +After a successful Garmin login/import, clear `action_reason` and restore health to `healthy` or `degraded` according to the resulting sync outcome. + +- [ ] **Step 4: Implement failed-activity reset operation** + +Repository method: + +```python +def reset_retryable_failure(self, activity_id: int) -> Activity: + activity = self._require(activity_id) + if activity.status != ActivityStatus.FAILED or not activity.retryable: + raise ValueError("activity is not retryable") + activity.status = activity.last_completed_stage + activity.last_error = None + self.session.commit() + return activity +``` + +- [ ] **Step 5: Run MFA/retry tests** + +Run: `pytest tests/web/test_mfa.py tests/web/test_operations.py -v` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add app/web app/sync/manager.py app/db/repositories.py tests/web/test_mfa.py +git commit -m "feat: handle Garmin MFA and activity retries" +``` + +## Task 7: End-to-end regression and Docker acceptance + +**Files:** +- Modify: `docker-compose.example.yml` only if integration exposes a missing runtime configuration +- Create: `tests/test_acceptance.py` + +**Interfaces:** +- No new interface; verifies the v1 acceptance criteria with fake external services. + +- [ ] **Step 1: Add an application-level acceptance test with two users** + +Build the app with temporary SQLite/data directories and injected fake MyWhoosh/Garmin factories. Seed two enabled users, give each one distinct remote activity IDs, run `sync_all_enabled()`, and assert: + +```python +assert all(result.status == "success" for result in results) +assert count_terminal_activities(user_a.id) == 1 +assert count_terminal_activities(user_b.id) == 1 +assert user_a_source_path.parent != user_b_source_path.parent +assert user_a_garmin_factory.tokenstore != user_b_garmin_factory.tokenstore +``` + +- [ ] **Step 2: Add isolation acceptance test** + +Configure User B to raise `GarminUploadBlocked`; assert User A still imports and User B ends `action_required` with no impact on User A. + +- [ ] **Step 3: Run the full suite** + +Run: `pytest -v` + +Expected: PASS. + +- [ ] **Step 4: Build Docker image again** + +Run: `docker build -t mywhoosh-garmin-sync:test .` + +Expected: successful build with the complete dependency set. + +- [ ] **Step 5: Start local container and exercise smoke paths** + +Start with a temporary bind-mounted `/data`, then verify: + +```bash +curl -fsS http://127.0.0.1:18080/healthz +curl -I http://127.0.0.1:18080/ +``` + +Expected: health JSON and dashboard redirect to `/login` when unauthenticated. + +- [ ] **Step 6: Verify secrets are absent from captured test logs** + +Run: + +```bash +pytest -v 2>&1 | tee /tmp/mywhoosh-garmin-test.log +! grep -F "mw-secret" /tmp/mywhoosh-garmin-test.log +! grep -F "garmin-secret" /tmp/mywhoosh-garmin-test.log +! grep -F "123456" /tmp/mywhoosh-garmin-test.log +``` + +Expected: all three negated `grep` commands succeed. + +- [ ] **Step 7: Commit** + +```bash +git add tests/test_acceptance.py docker-compose.example.yml +git commit -m "test: cover multi-user sync acceptance" +``` diff --git a/docs/superpowers/specs/2026-08-15-mywhoosh-garmin-sync-design.md b/docs/superpowers/specs/2026-08-15-mywhoosh-garmin-sync-design.md new file mode 100644 index 0000000..a7a9749 --- /dev/null +++ b/docs/superpowers/specs/2026-08-15-mywhoosh-garmin-sync-design.md @@ -0,0 +1,620 @@ +# MyWhoosh -> Garmin Sync Service — Design + +Date: 2026-08-15 +Status: Draft for user review + +## 1. Goal + +Build a self-hosted Docker service that periodically checks MyWhoosh for new cycling activities for multiple configured users, downloads each new FIT file, rewrites the device metadata to a Garmin Edge 1030 Plus, and imports the activity into Garmin Connect without forwarding it to Strava. + +The service is administered through a local-only web interface protected by a single admin password. + +## 2. Scope + +### In scope for v1 + +- Multiple independent sync users. +- Local-only admin UI. +- Admin password supplied via environment variable. +- SQLite persistence. +- Encrypted MyWhoosh and Garmin credentials at rest. +- Persistent per-user MyWhoosh and Garmin token stores. +- Periodic background sync at a configurable interval. +- Manual "sync now" actions. +- MyWhoosh activity discovery and FIT download. +- Binary FIT metadata patching to Garmin Edge 1030 Plus. +- FIT CRC validation and repair. +- Garmin Connect import via `import_activity()`. +- No intentional forwarding to Strava. +- Per-user error isolation, status and retry handling. +- Garmin MFA handling through the admin UI when required. +- Activity history and sync-run history. + +### Out of scope for v1 + +- Public Internet exposure. +- Multi-admin or per-user web logins. +- OAuth/OIDC for the admin UI. +- Editing deployment configuration from the web UI. +- Strava integration. +- Mobile app. +- Distributed workers or external queues. +- Automatic CAPTCHA solving or browser automation for MyWhoosh login. + +## 3. High-level architecture + +The application runs as one Docker container with a persistent `/data` volume. + +Components: + +1. FastAPI application. +2. Server-rendered Jinja2 admin UI with HTMX for small interactive actions. +3. SQLite database. +4. Scheduler. +5. Sync manager. +6. Per-user MyWhoosh client. +7. FIT rewriter. +8. Per-user Garmin client/uploader. +9. Credential encryption service. + +Data flow: + +MyWhoosh -> download FIT -> validate FIT -> patch device metadata -> rewrite CRC -> Garmin `import_activity()` -> persist result. + +## 4. Deployment configuration + +Configuration is supplied through environment variables, for example: + +- `ADMIN_PASSWORD` +- `SECRET_KEY` +- `CREDENTIAL_ENCRYPTION_KEY` +- `SYNC_INTERVAL_MINUTES` +- `DATABASE_URL=sqlite:////data/app.db` +- `DATA_DIR=/data` + +The web server binds inside the container and is published only to the trusted local network by Docker configuration. + +The web UI does not modify these deployment-level settings. + +## 5. Admin authentication + +The application has one admin login with no username. + +- The password is read from `ADMIN_PASSWORD`. +- The password is never persisted in SQLite. +- Successful login establishes a signed session using `SECRET_KEY`. +- Authentication failures reveal no account details. +- Session cookies should be `HttpOnly` and `SameSite=Lax`. +- If TLS is later placed in front of the service, `Secure` should be enabled for the cookie. + +Because the service is intended for LAN-only use, v1 does not introduce a separate identity provider. + +## 6. User model + +Each sync user is independent. + +A user contains: + +- id +- display name +- enabled flag +- health state +- encrypted MyWhoosh email/password +- encrypted Garmin email/password +- created/updated timestamps + +Credentials are encrypted before being written to SQLite. The encryption key comes exclusively from `CREDENTIAL_ENCRYPTION_KEY`. + +Stored passwords are never returned to the browser. When editing a user, an empty password field means "keep the existing password". + +## 7. Token storage + +Authentication/session tokens are separated per user. + +Suggested filesystem layout: + +```text +/data/ + app.db + tokens/ + / + mywhoosh.json + garmin/ + activities/ + / +``` + +The Garmin tokenstore mechanism from `python-garminconnect` should be reused rather than reimplemented. + +The MyWhoosh token cache stores the access token and, where usable, the refresh token and associated account metadata. + +No user may read or reuse another user's tokenstore. + +## 8. MyWhoosh authentication + +The service should follow the same direct API-login pattern used by `jdelrue/mywhoosh2garmin` rather than automating the MyWhoosh web login page. + +The intended flow is: + +1. Load cached per-user token. +2. Attempt an authenticated activities request. +3. If accepted, continue. +4. If unauthorized, attempt API login using the configured MyWhoosh credentials and the Android-style login payload used by the reference project. +5. Persist fresh token data. +6. Retry the operation once. + +The implementation must not attempt to defeat or automate CAPTCHA/reCAPTCHA challenges. + +The MyWhoosh endpoints are not treated as a stable public API. Changes in these endpoints should surface as a clear `action_required`/authentication or integration failure rather than causing uncontrolled retries. + +## 9. MyWhoosh activity discovery + +For each enabled user, the MyWhoosh client retrieves recent activities using the authenticated bearer token. + +Each MyWhoosh activity must have a stable external activity identifier. The pair `(user_id, mywhoosh_activity_id)` is unique in SQLite. + +This makes discovery idempotent: the same activity may be returned on every scheduler run but is processed only once unless it previously failed at a retryable stage. + +The client is responsible only for: + +- authentication, +- listing activities, +- normalizing metadata, +- downloading the original FIT file. + +It has no knowledge of Garmin or FIT rewriting. + +## 10. FIT rewriting + +The FIT rewriter follows the binary-patching approach from the existing working Python implementation rather than fully decoding and re-encoding the activity. + +### 10.1 Device identity + +Target device: + +- manufacturer: Garmin (`1`) +- product: Edge 1030 Plus (`3570`) +- product name: `Edge 1030 Plus` +- serial number: optional; if absent, the original serial field is left unchanged unless a later compatibility requirement proves otherwise + +### 10.2 Patched messages + +`file_id` fields when present: + +- manufacturer +- product +- optional serial number +- product name + +`device_info` fields are patched only for the creator device (`device_index == 0`) when a usable device index is present. + +Other sensor/device records should remain unchanged so a trainer, HR sensor or power meter does not become an Edge 1030 Plus accidentally. + +If the source FIT lacks enough information to identify creator-specific `device_info` safely, `file_id` remains mandatory and `device_info` patching should be conservative rather than rewriting all device messages. + +### 10.3 Binary preservation + +The rewriter must preserve all bytes not deliberately changed, except FIT CRC fields. + +It must support: + +- 12-byte and 14-byte FIT headers, +- little- and big-endian definition architectures, +- compressed timestamp records, +- developer fields, +- changing local message definitions. + +### 10.4 Validation + +Before patching: + +- validate `.FIT` signature, +- validate declared length, +- validate header CRC when present, +- validate file CRC. + +After patching: + +- rewrite header CRC when present, +- rewrite file CRC, +- validate the output again, +- verify expected target metadata is readable. + +Invalid FIT input is a non-retryable activity error unless the original file is later replaced/redownloaded. + +## 11. Garmin import + +The existing Garmin uploader pattern is reused with `python-garminconnect`. + +The final activity is sent using `import_activity()` rather than `upload_activity()` because the desired behavior is to import into Garmin Connect without intentional onward synchronization to Strava. + +Per user: + +1. Reuse Garmin tokenstore where possible. +2. Login/refresh when required. +3. Call `import_activity()` with the converted FIT path. +4. Record the returned Garmin activity/import identifier if available. +5. Treat known duplicate responses as completed `duplicate`, not as fatal failures. + +## 12. Garmin MFA + +MFA is modeled as an explicit user state. + +If Garmin requires MFA and there is no one-time code available: + +- the user's health becomes `action_required`, +- that user's Garmin import attempts pause, +- other users continue syncing normally, +- the dashboard shows that MFA is required. + +The admin can submit the one-time MFA code through the local UI. + +The code: + +- is used only for that login attempt, +- is never written to SQLite, +- is never written to logs, +- is discarded immediately after use. + +On successful authentication the Garmin tokenstore is persisted and the user returns to normal sync behavior. + +## 13. Activity state machine + +An activity progresses through durable stages: + +- `discovered` +- `downloaded` +- `converted` +- `imported` +- `duplicate` +- `failed` + +Persisted activity fields include: + +- internal id +- user id +- MyWhoosh activity id +- activity date/time +- activity name +- original FIT path +- converted FIT path +- current status +- Garmin activity/import id when known +- last error +- created/updated timestamps + +Completed terminal states are `imported` and `duplicate`. + +A failure must retain the latest successfully completed stage so a retry can resume without repeating unnecessary work. + +## 14. Sync-run model + +Each user sync invocation creates a sync-run record containing: + +- id +- user id +- start time +- finish time +- status (`running`, `success`, `partial`, `failed`) +- discovered count +- imported count +- skipped count +- failed count +- summary error when relevant + +Detailed application logs remain on stdout; SQLite stores only UI-relevant summaries. + +## 15. Scheduler and concurrency + +A central scheduler triggers every `SYNC_INTERVAL_MINUTES`. + +On each tick: + +1. Load enabled users. +2. Schedule one sync job per user. +3. Allow different users to run concurrently. +4. Enforce at most one active sync per user with a per-user lock. + +Manual "sync now" uses exactly the same sync pipeline and the same lock. + +If a manual request arrives while that user is already syncing, the application should return a clear "already running" result rather than start another run. + +The scheduler must not block because one account is slow, broken or waiting for user action. + +## 16. Retry policy + +Retries depend on failure type. + +### Retryable automatically + +- transient network errors +- timeouts +- temporary MyWhoosh/Garmin server errors +- expired session/token after one reauthentication attempt + +Within one sync run, use at most a small bounded retry (for example one retry). Further retry occurs on the next scheduler tick. + +### Action required + +- invalid MyWhoosh credentials +- MyWhoosh login/API behavior changed in a way that prevents authentication +- Garmin MFA required +- invalid Garmin credentials + +### Non-retryable per activity + +- corrupt/invalid FIT file +- unsupported FIT structure that cannot be safely patched + +The admin UI can expose an explicit "retry" action for failed activities after the underlying issue is fixed. + +## 17. User health state + +Each user has a concise operational state: + +- `healthy` +- `syncing` +- `degraded` +- `action_required` +- `disabled` + +This state is derived from configuration and recent sync/authentication outcomes and is shown prominently on the dashboard. + +## 18. Admin UI + +### 18.1 Login + +Single password field and submit action. + +### 18.2 Dashboard + +Shows all users with: + +- name +- health state +- MyWhoosh connection state +- Garmin connection state +- last sync +- last imported activity +- primary error/action if any +- "sync now" +- "details" +- MFA action when needed + +Includes "add account". + +### 18.3 User create/edit + +Fields: + +- display name +- MyWhoosh email +- MyWhoosh password +- Garmin email +- Garmin password +- enabled flag + +Actions: + +- save +- test connection + +Existing passwords are never rendered back to the browser. + +### 18.4 User details + +Shows: + +- current connection and health states +- most recent sync-run summary +- recent activities and status +- latest errors + +Actions: + +- sync now +- retry failed activity +- enter Garmin MFA when required + +### 18.5 System page + +Read-only operational information: + +- application version +- configured sync interval +- last scheduler tick +- next expected tick +- account count +- activity count + +Action: + +- sync all now + +## 19. UI technology + +Use: + +- FastAPI +- Jinja2 +- HTMX +- small application-specific CSS + +Do not introduce Angular, React, Tailwind or Bootstrap for v1 unless requirements change. + +HTMX is used for bounded actions such as: + +- sync now +- test connection +- submit MFA +- retry activity + +The application remains server-rendered and easy to operate as one container. + +## 20. Security requirements + +- Never log passwords, bearer tokens, session tokens, encryption keys or MFA codes. +- Encrypt stored MyWhoosh and Garmin credentials. +- Keep tokenstores under the persistent data directory with restrictive filesystem permissions where possible. +- Escape all user-visible data rendered into HTML. +- Protect state-changing web requests against CSRF. +- Validate all IDs against the authenticated admin session rather than trusting client-provided paths blindly. +- Use prepared/ORM parameterized database access. +- Do not expose decrypted credentials through API responses or templates. + +## 21. Cleanup and retention + +For v1, original and converted FIT files are retained because they are valuable for debugging failed imports. + +Automated retention/cleanup can be added later after operating behavior is known. + +## 22. Error isolation + +Failure of one user must never prevent other users from syncing. + +Examples: + +- User A imports successfully while User B requires Garmin MFA. +- User C may have invalid MyWhoosh credentials without affecting scheduler execution for A or B. +- A corrupt activity file affects only that activity and user. + +## 23. Testing strategy + +### FIT rewriter tests + +- valid 12-byte header FIT +- valid 14-byte header FIT +- invalid header CRC +- invalid file CRC +- malformed/truncated definitions +- developer fields preserved +- compressed timestamp records handled +- Edge 1030 Plus manufacturer/product patched correctly +- non-creator `device_info` unchanged +- output CRC valid +- bytes outside expected metadata and CRC locations unchanged + +### MyWhoosh client tests + +Use mocked HTTP responses for: + +- valid cached token +- expired token followed by successful login +- invalid credentials +- transient server error +- activity listing +- FIT download + +Do not make live MyWhoosh requests in the normal unit test suite. + +### Garmin uploader tests + +Use a fake/protocol-compatible Garmin client for: + +- tokenstore login +- successful import +- duplicate +- MFA required +- invalid login +- transient import failure + +### Sync manager tests + +- new activity full happy path +- discovered activity is not duplicated +- resume from downloaded +- resume from converted +- retry after transient Garmin failure +- one user's failure does not affect another +- per-user lock prevents concurrent duplicate sync + +### Web tests + +- admin login success/failure +- unauthenticated routes redirect/reject +- create/edit/disable user +- password never returned +- manual sync action +- MFA submission lifecycle +- CSRF on mutating requests + +## 24. Suggested module boundaries + +```text +app/ + main.py + config.py + + auth/ + admin.py + + db/ + models.py + session.py + repositories.py + + security/ + credentials.py + + mywhoosh/ + client.py + models.py + tokenstore.py + + fit/ + rewriter.py + crc.py + models.py + + garmin/ + uploader.py + + sync/ + manager.py + scheduler.py + states.py + + web/ + routes.py + forms.py + templates/ + static/ + +tests/ +``` + +Each unit should depend on explicit interfaces/protocols where external services are involved so tests do not require live accounts. + +## 25. Primary design decisions + +1. One local admin instead of user-facing authentication. +2. Multiple independent sync accounts. +3. SQLite for durable application state. +4. Environment variables for deployment secrets/configuration. +5. Encrypted service credentials at rest. +6. Per-user tokenstores. +7. Direct MyWhoosh API login pattern; no CAPTCHA automation. +8. Binary FIT patching rather than decode/re-encode. +9. Garmin Edge 1030 Plus product ID `3570`. +10. Conservative creator-device patching. +11. Garmin `import_activity()` for Garmin-only import behavior. +12. FastAPI + Jinja2 + HTMX for a small single-container admin UI. +13. Parallel sync across users, serialized sync within each user. +14. Durable activity stages for resumable/idempotent sync. + +## 26. Acceptance criteria for v1 + +The system is ready for v1 when: + +1. It runs from Docker with persistent `/data` storage. +2. The admin can log in locally using the environment-configured password. +3. The admin can add at least two independent users. +4. Each user can authenticate independently to MyWhoosh and Garmin. +5. New MyWhoosh activities are discovered automatically on schedule. +6. FIT files are downloaded and patched to Garmin Edge 1030 Plus metadata with valid CRCs. +7. Converted activities are imported into the corresponding Garmin Connect account using `import_activity()`. +8. Already processed activities are not imported again. +9. Garmin MFA for one user can be resolved through the UI and does not block other users. +10. A failure for one user or one activity does not stop the scheduler. +11. The dashboard shows current state, recent syncs and actionable errors. +12. Secrets and MFA codes do not appear in logs or browser responses.