bastian/mywhoosh2garmin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plans + specs

Browse Source
This commit is contained in:
Bastian Wagner
2026-08-15 09:00:41 +02:00
commit f6da346e18
5 changed files with 3804 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

577
docs/superpowers/plans/2026-08-15-fit-rewriter.md Normal file
View File

@@ -0,0 +1,577 @@
# FIT Edge 1030 Plus Rewriter Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement a binary-preserving FIT metadata rewriter that changes only Garmin creator-device metadata plus required CRC bytes, targeting Garmin Edge 1030 Plus product ID `3570`.
**Architecture:** Parse FIT definition/data records only far enough to locate `file_id` and creator `device_info` fields, patch bytes in place, recalculate header/file CRC, and validate the output. Do not fully decode/re-encode the activity, so unknown/developer fields and all non-target bytes remain untouched.
**Tech Stack:** Python 3.12 standard library (`struct`, `dataclasses`, `pathlib`), pytest.
## Global Constraints
- Garmin manufacturer ID is `1`.
- Garmin Edge 1030 Plus product ID is `3570`.
- Product name is `Edge 1030 Plus`.
- Support FIT header sizes 12 and 14.
- Validate declared data size and `.FIT` signature.
- Validate header CRC when a 14-byte header is present.
- Validate file CRC before and after modification.
- Support little- and big-endian definition architectures, compressed timestamp records, developer fields, and changing local message definitions.
- Patch `file_id` device fields when present.
- Patch `device_info` only when it is safely identified as the creator (`device_index == 0`).
- If creator-specific `device_info` cannot be identified safely, leave it unchanged rather than rewriting all device records.
- Preserve every non-target byte except CRC fields.
---
## File Structure
```text
app/fit/
__init__.py
crc.py
models.py
rewriter.py
tests/fit/
builders.py
test_crc.py
test_rewriter_validation.py
test_rewriter_patching.py
test_rewriter_preservation.py
```
## Task 1: Implement Garmin FIT CRC
**Files:**
- Create: `app/fit/crc.py`
- Create: `tests/fit/test_crc.py`
**Interfaces:**
- Produces: `fit_crc(data: bytes | bytearray | memoryview) -> int`.
- [ ] **Step 1: Write failing CRC vector tests**
```python
# tests/fit/test_crc.py
from app.fit.crc import fit_crc
def test_empty_crc_is_zero() -> None:
assert fit_crc(b"") == 0
def test_crc_is_incremental_equivalent() -> None:
payload = b".FIT-device-metadata"
assert fit_crc(payload) == fit_crc(memoryview(payload))
assert 0 <= fit_crc(payload) <= 0xFFFF
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/fit/test_crc.py -v`
Expected: import failure.
- [ ] **Step 3: Implement the FIT nibble-table CRC algorithm**
```python
# app/fit/crc.py
CRC_TABLE = (
0x0000, 0xCC01, 0xD801, 0x1400,
0xF001, 0x3C00, 0x2800, 0xE401,
0xA001, 0x6C00, 0x7800, 0xB401,
0x5000, 0x9C01, 0x8801, 0x4400,
)
def fit_crc(data: bytes | bytearray | memoryview) -> int:
crc = 0
for byte in data:
tmp = CRC_TABLE[crc & 0xF]
crc = (crc >> 4) & 0x0FFF
crc ^= tmp ^ CRC_TABLE[byte & 0xF]
tmp = CRC_TABLE[crc & 0xF]
crc = (crc >> 4) & 0x0FFF
crc ^= tmp ^ CRC_TABLE[(byte >> 4) & 0xF]
return crc & 0xFFFF
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/fit/test_crc.py -v`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add app/fit/crc.py tests/fit/test_crc.py
git commit -m "feat: add FIT CRC calculation"
```
## Task 2: Parse and validate the FIT container safely
**Files:**
- Create: `app/fit/models.py`
- Create: `app/fit/rewriter.py`
- Create: `tests/fit/builders.py`
- Create: `tests/fit/test_rewriter_validation.py`
**Interfaces:**
- Produces: `FitFormatError`, `FieldDefinition`, `LocalDefinition`, `_validate_fit_container()`, `_iter_data_fields()`.
- `_iter_data_fields(data)` returns data records with exact field offsets without decoding unrelated values.
- [ ] **Step 1: Create a deterministic minimal FIT fixture builder**
```python
# tests/fit/builders.py
import struct
from app.fit.crc import fit_crc
def make_fit(data_records: bytes, *, header_size: int = 14) -> bytes:
if header_size not in {12, 14}:
raise ValueError(header_size)
header = bytearray(header_size)
header[0] = header_size
header[1] = 0x20
struct.pack_into("<H", header, 2, 0x0100)
struct.pack_into("<I", header, 4, len(data_records))
header[8:12] = b".FIT"
if header_size == 14:
struct.pack_into("<H", header, 12, fit_crc(header[:12]))
body = header + data_records
return bytes(body + struct.pack("<H", fit_crc(body)))
def definition(local: int, global_num: int, fields: list[tuple[int, int, int]], *, endian: str = "<") -> bytes:
architecture = 1 if endian == ">" else 0
payload = bytearray([0, architecture])
payload.extend(struct.pack(f"{endian}H", global_num))
payload.append(len(fields))
for num, size, base_type in fields:
payload.extend(bytes([num, size, base_type]))
return bytes([0x40 | local]) + bytes(payload)
def data(local: int, payload: bytes) -> bytes:
return bytes([local]) + payload
```
- [ ] **Step 2: Write validation tests**
```python
# tests/fit/test_rewriter_validation.py
from pathlib import Path
import pytest
from app.fit.rewriter import FitFormatError, is_fit_file
from tests.fit.builders import make_fit
def test_valid_12_and_14_byte_headers(tmp_path: Path) -> None:
for size in (12, 14):
path = tmp_path / f"valid-{size}.fit"
path.write_bytes(make_fit(b"", header_size=size))
assert is_fit_file(path) is True
def test_bad_file_crc_is_rejected(tmp_path: Path) -> None:
payload = bytearray(make_fit(b""))
payload[-1] ^= 0xFF
path = tmp_path / "bad.fit"
path.write_bytes(payload)
assert is_fit_file(path) is False
```
- [ ] **Step 3: Run and verify failure**
Run: `pytest tests/fit/test_rewriter_validation.py -v`
Expected: import failure.
- [ ] **Step 4: Implement the parser structures and validation**
```python
# app/fit/models.py
from dataclasses import dataclass
@dataclass(frozen=True)
class FieldDefinition:
num: int
size: int
base_type: int
@dataclass(frozen=True)
class LocalDefinition:
global_message_num: int
endian: str
fields: tuple[FieldDefinition, ...]
developer_field_size: int
```
Implement in `app/fit/rewriter.py` the validated logic from the known working implementation:
```python
class FitFormatError(ValueError):
pass
def _validate_fit_container(data: bytearray) -> None:
if len(data) < 14:
raise FitFormatError("FIT file is too small")
header_size = data[0]
if header_size not in {12, 14}:
raise FitFormatError(f"Unsupported FIT header size: {header_size}")
if len(data) < header_size + 2:
raise FitFormatError("FIT file is shorter than its header")
if bytes(data[8:12]) != b".FIT":
raise FitFormatError("Missing .FIT signature")
data_size = struct.unpack_from("<I", data, 4)[0]
expected_size = header_size + data_size + 2
if len(data) != expected_size:
raise FitFormatError(f"FIT size mismatch: header says {expected_size} bytes, file has {len(data)}")
if header_size == 14:
expected_header_crc = struct.unpack_from("<H", data, 12)[0]
if expected_header_crc != fit_crc(data[:12]):
raise FitFormatError("FIT header CRC check failed")
expected_file_crc = struct.unpack_from("<H", data, len(data) - 2)[0]
if expected_file_crc != fit_crc(data[:-2]):
raise FitFormatError("FIT file CRC check failed")
```
Implement the parser helpers explicitly:
```python
def _read_definition(
data: bytearray,
offset: int,
has_developer_fields: bool,
end_offset: int,
) -> tuple[LocalDefinition, int]:
if offset + 5 > end_offset:
raise FitFormatError("Truncated FIT definition message")
offset += 1 # reserved byte
architecture = data[offset]
offset += 1
if architecture not in {0, 1}:
raise FitFormatError(f"Unsupported FIT architecture: {architecture}")
endian = ">" if architecture == 1 else "<"
global_message_num = struct.unpack_from(f"{endian}H", data, offset)[0]
offset += 2
field_count = data[offset]
offset += 1
fields: list[FieldDefinition] = []
for _ in range(field_count):
if offset + 3 > end_offset:
raise FitFormatError("Truncated FIT field definition")
fields.append(FieldDefinition(data[offset], data[offset + 1], data[offset + 2]))
offset += 3
developer_field_size = 0
if has_developer_fields:
if offset >= end_offset:
raise FitFormatError("Truncated FIT developer field count")
developer_count = data[offset]
offset += 1
for _ in range(developer_count):
if offset + 3 > end_offset:
raise FitFormatError("Truncated FIT developer field definition")
developer_field_size += data[offset + 1]
offset += 3
return LocalDefinition(global_message_num, endian, tuple(fields), developer_field_size), offset
def _collect_field_offsets(
definition: LocalDefinition,
offset: int,
end_offset: int,
) -> tuple[list[tuple[FieldDefinition, int]], int]:
result: list[tuple[FieldDefinition, int]] = []
current = offset
for field in definition.fields:
if current + field.size > end_offset:
raise FitFormatError("Truncated FIT data record")
result.append((field, current))
current += field.size
if current + definition.developer_field_size > end_offset:
raise FitFormatError("Truncated FIT developer field payload")
current += definition.developer_field_size
return result, current
def _iter_data_fields(
data: bytearray,
) -> list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]]:
header_size = data[0]
data_size = struct.unpack_from("<I", data, 4)[0]
offset = header_size
end_offset = header_size + data_size
definitions: dict[int, LocalDefinition] = {}
records: list[tuple[LocalDefinition, list[tuple[FieldDefinition, int]]]] = []
while offset < end_offset:
record_header = data[offset]
offset += 1
if record_header & 0x80:
local = (record_header >> 5) & 0x03
definition = definitions.get(local)
if definition is None:
raise FitFormatError(f"Compressed timestamp record used unknown local definition {local}")
field_offsets, offset = _collect_field_offsets(definition, offset, end_offset)
records.append((definition, field_offsets))
continue
local = record_header & 0x0F
is_definition = bool(record_header & 0x40)
has_developer_fields = bool(record_header & 0x20)
if is_definition:
definition, offset = _read_definition(data, offset, has_developer_fields, end_offset)
definitions[local] = definition
continue
definition = definitions.get(local)
if definition is None:
raise FitFormatError(f"Data record used unknown local definition {local}")
field_offsets, offset = _collect_field_offsets(definition, offset, end_offset)
records.append((definition, field_offsets))
if offset != end_offset:
raise FitFormatError("FIT parser did not end on data boundary")
return records
```
- [ ] **Step 5: Expose `is_fit_file` and verify parser boundary failures**
```python
def is_fit_file(path: Path) -> bool:
try:
data = bytearray(path.read_bytes())
_validate_fit_container(data)
_iter_data_fields(data)
except (OSError, FitFormatError):
return False
return True
```
- [ ] **Step 6: Run validation tests**
Run: `pytest tests/fit/test_rewriter_validation.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/fit/models.py app/fit/rewriter.py tests/fit/builders.py tests/fit/test_rewriter_validation.py
git commit -m "feat: parse and validate FIT containers"
```
## Task 3: Patch `file_id` and only creator `device_info`
**Files:**
- Modify: `app/fit/models.py`
- Modify: `app/fit/rewriter.py`
- Create: `tests/fit/test_rewriter_patching.py`
**Interfaces:**
- Produces: `GarminDevice`, `FitConversionResult`, `DeviceFieldValue`, `convert_fit_device()`, `read_device_field_values()`.
- Default device is Garmin manufacturer `1`, Edge 1030 Plus product `3570`, name `Edge 1030 Plus`.
- [ ] **Step 1: Add model types**
```python
# app/fit/models.py additions
from pathlib import Path
GARMIN_MANUFACTURER_ID = 1
@dataclass(frozen=True)
class GarminDevice:
manufacturer_id: int = GARMIN_MANUFACTURER_ID
product_id: int = 3570
product_name: str = "Edge 1030 Plus"
serial_number: int | None = None
@dataclass(frozen=True)
class FitConversionResult:
source_path: Path
output_path: Path
patched_field_count: int
header_crc: int | None
file_crc: int
@dataclass(frozen=True)
class DeviceFieldValue:
global_message_num: int
field_num: int
value: int | str
```
- [ ] **Step 2: Write a fixture containing one creator and one sensor `device_info`**
Use these FIT field numbers in `tests/fit/test_rewriter_patching.py`:
```python
FILE_ID_MESG_NUM = 0
DEVICE_INFO_MESG_NUM = 23
# file_id: manufacturer(1/u16), product(2/u16)
file_def = definition(0, FILE_ID_MESG_NUM, [(1, 2, 0x84), (2, 2, 0x84)])
file_data = data(0, struct.pack("<HH", 255, 999))
# device_info: device_index(0/u8), manufacturer(2/u16), product(4/u16)
device_def = definition(1, DEVICE_INFO_MESG_NUM, [(0, 1, 0x02), (2, 2, 0x84), (4, 2, 0x84)])
creator = data(1, struct.pack("<BHH", 0, 255, 999))
sensor = data(1, struct.pack("<BHH", 1, 32, 1234))
```
Write assertions that `file_id` and creator become `(1, 3570)` while sensor remains `(32, 1234)`.
- [ ] **Step 3: Run the test and verify failure**
Run: `pytest tests/fit/test_rewriter_patching.py -v`
Expected: missing conversion functions.
- [ ] **Step 4: Implement conservative creator detection**
In `_patch_device_metadata`, process each `device_info` record as a unit:
```python
field_map = {field.num: (field, offset) for field, offset in field_offsets}
if definition.global_message_num == DEVICE_INFO_MESG_NUM:
device_index_entry = field_map.get(0)
if device_index_entry is None:
continue
index_field, index_offset = device_index_entry
device_index = _read_field_value(data, index_offset, index_field, definition.endian)
if device_index != 0:
continue
```
Only after this check may fields `2`, `3`, `4`, and `27` be patched. Do not patch a `device_info` message that lacks field `0`.
Always patch eligible `file_id` fields `1`, `2`, optional `3`, and `8`.
- [ ] **Step 5: Implement numeric/string read-write helpers and conversion entry point**
Use the supplied working `_read_field_value()` and `_write_field_value()` behavior, then expose:
```python
def convert_fit_device(source_path: Path, output_path: Path, device: GarminDevice | None = None) -> FitConversionResult:
resolved = device or GarminDevice()
data = bytearray(source_path.read_bytes())
_validate_fit_container(data)
patched_count = _patch_device_metadata(data, resolved)
header_crc = _rewrite_header_crc(data)
file_crc = _rewrite_file_crc(data)
_validate_fit_container(data)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_bytes(data)
return FitConversionResult(source_path, output_path, patched_count, header_crc, file_crc)
```
- [ ] **Step 6: Run patching tests**
Run: `pytest tests/fit/test_rewriter_patching.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/fit tests/fit/test_rewriter_patching.py
git commit -m "feat: patch FIT creator as Edge 1030 Plus"
```
## Task 4: Prove binary preservation and advanced record support
**Files:**
- Create: `tests/fit/test_rewriter_preservation.py`
- Modify: `tests/fit/builders.py`
- Modify: `app/fit/rewriter.py` only if a test exposes an actual parser defect
**Interfaces:**
- No new public interface; this task hardens `convert_fit_device()`.
- [ ] **Step 1: Write a preservation test that records changed byte positions**
```python
# tests/fit/test_rewriter_preservation.py
from pathlib import Path
from app.fit.rewriter import convert_fit_device
def test_only_target_fields_and_crcs_change(tmp_path: Path, complex_fit_bytes: bytes) -> None:
source = tmp_path / "source.fit"
output = tmp_path / "output.fit"
source.write_bytes(complex_fit_bytes)
convert_fit_device(source, output)
before = source.read_bytes()
after = output.read_bytes()
assert len(before) == len(after)
changed = {index for index, (a, b) in enumerate(zip(before, after)) if a != b}
expected_metadata_offsets = set(find_expected_device_metadata_offsets(before))
crc_offsets = {12, 13, len(before) - 2, len(before) - 1}
assert changed <= expected_metadata_offsets | crc_offsets
```
The fixture helper `find_expected_device_metadata_offsets()` must use the test fixture's known construction offsets, not production parser code, so the test is independent.
- [ ] **Step 2: Add fixtures for compressed timestamps, developer fields, and local-definition replacement**
Construct one synthetic FIT data section containing:
1. a normal `file_id` definition/data pair;
2. a `device_info` definition with creator record;
3. a record definition with one developer field and one data record;
4. a compressed-timestamp data header referring to a known local definition;
5. a later replacement definition for the same local message number.
The test passes if conversion completes, output CRC validates, and non-target payload bytes remain identical.
- [ ] **Step 3: Run the focused advanced tests**
Run: `pytest tests/fit/test_rewriter_preservation.py -v`
Expected: PASS. If a parser guard fails, fix only the smallest parser defect required by the test.
- [ ] **Step 4: Add rejection tests for malformed/truncated definitions**
```python
def test_truncated_definition_is_non_recoverable(tmp_path: Path) -> None:
path = tmp_path / "truncated.fit"
path.write_bytes(make_fit(bytes([0x40, 0x00, 0x00])))
assert is_fit_file(path) is False
```
Also assert that `convert_fit_device()` raises `FitFormatError` for the same input.
- [ ] **Step 5: Run the entire FIT suite**
Run: `pytest tests/fit -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add tests/fit app/fit/rewriter.py
git commit -m "test: verify FIT binary preservation"
```

1087
docs/superpowers/plans/2026-08-15-foundation-admin-data.md Normal file
View File

File diff suppressed because it is too large Load Diff

716
docs/superpowers/plans/2026-08-15-service-clients.md Normal file
View File

@@ -0,0 +1,716 @@
# MyWhoosh and Garmin Service Clients Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement isolated per-user MyWhoosh and Garmin clients with persistent tokenstores, mockable network boundaries, MyWhoosh direct API login/activity download, Garmin `import_activity()`, duplicate handling, and MFA signaling.
**Architecture:** Keep external integrations behind narrow protocols so the sync engine never depends directly on `httpx` or `garminconnect`. MyWhoosh uses an injected `httpx.AsyncClient` and a per-user JSON tokenstore. Garmin uses an injected client factory and per-user `python-garminconnect` token directory; blocking Garmin calls are later run through `asyncio.to_thread` by the sync layer.
**Tech Stack:** Python 3.12, httpx, python-garminconnect, pytest, pytest-asyncio.
## Global Constraints
- Do not automate or defeat MyWhoosh CAPTCHA/reCAPTCHA.
- Follow the direct Android-style API login flow used by the reference `jdelrue/mywhoosh2garmin` project.
- Cache MyWhoosh tokens per user under `/data/tokens/<user-id>/mywhoosh.json`.
- Cache Garmin tokens per user under `/data/tokens/<user-id>/garmin/` using the library's tokenstore behavior.
- Authentication/API changes must become explicit integration/authentication errors, not uncontrolled retries.
- Garmin activity transfer must use `import_activity()`, not `upload_activity()`.
- Known duplicate Garmin responses are successful terminal outcomes.
- Garmin MFA must raise a dedicated exception so the UI can collect a one-time code.
- Never log passwords, bearer tokens, Garmin tokens, or MFA codes.
---
## File Structure
```text
app/mywhoosh/
__init__.py
models.py
tokenstore.py
client.py
app/garmin/
__init__.py
uploader.py
tests/mywhoosh/
test_tokenstore.py
test_client_auth.py
test_client_activities.py
tests/garmin/
test_uploader.py
```
## Task 1: Implement MyWhoosh models and tokenstore
**Files:**
- Create: `app/mywhoosh/models.py`
- Create: `app/mywhoosh/tokenstore.py`
- Create: `tests/mywhoosh/test_tokenstore.py`
**Interfaces:**
- Produces `MyWhooshToken`, `MyWhooshActivity`, `MyWhooshTokenStore.load()`, `save()`, `clear()`.
- [ ] **Step 1: Write tokenstore tests**
```python
# tests/mywhoosh/test_tokenstore.py
from pathlib import Path
from app.mywhoosh.models import MyWhooshToken
from app.mywhoosh.tokenstore import MyWhooshTokenStore
def test_tokenstore_round_trip_and_permissions(tmp_path: Path) -> None:
store = MyWhooshTokenStore(tmp_path / "tokens" / "7" / "mywhoosh.json")
token = MyWhooshToken(access_token="access", refresh_token="refresh", whoosh_id="whoosh-7")
store.save(token)
assert store.load() == token
assert oct(store.path.stat().st_mode & 0o777) == "0o600"
def test_missing_token_returns_none(tmp_path: Path) -> None:
store = MyWhooshTokenStore(tmp_path / "missing.json")
assert store.load() is None
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/mywhoosh/test_tokenstore.py -v`
Expected: import failure.
- [ ] **Step 3: Implement models**
```python
# app/mywhoosh/models.py
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class MyWhooshToken:
access_token: str
refresh_token: str | None
whoosh_id: str | None
@dataclass(frozen=True)
class MyWhooshActivity:
id: str
title: str
activity_file_id: str
started_at: datetime | None
```
- [ ] **Step 4: Implement atomic JSON token persistence**
```python
# app/mywhoosh/tokenstore.py
import json
import os
from pathlib import Path
from app.mywhoosh.models import MyWhooshToken
class MyWhooshTokenStore:
def __init__(self, path: Path) -> None:
self.path = path
def load(self) -> MyWhooshToken | None:
try:
raw = json.loads(self.path.read_text("utf-8"))
except FileNotFoundError:
return None
return MyWhooshToken(
access_token=raw["access_token"],
refresh_token=raw.get("refresh_token"),
whoosh_id=raw.get("whoosh_id"),
)
def save(self, token: MyWhooshToken) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
tmp = self.path.with_suffix(".tmp")
tmp.write_text(
json.dumps(
{
"access_token": token.access_token,
"refresh_token": token.refresh_token,
"whoosh_id": token.whoosh_id,
},
indent=2,
),
"utf-8",
)
os.chmod(tmp, 0o600)
tmp.replace(self.path)
os.chmod(self.path, 0o600)
def clear(self) -> None:
self.path.unlink(missing_ok=True)
```
- [ ] **Step 5: Run tests**
Run: `pytest tests/mywhoosh/test_tokenstore.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/mywhoosh/models.py app/mywhoosh/tokenstore.py tests/mywhoosh/test_tokenstore.py
git commit -m "feat: add MyWhoosh token persistence"
```
## Task 2: Implement MyWhoosh login and cached-session recovery
**Files:**
- Create: `app/mywhoosh/client.py`
- Create: `tests/mywhoosh/test_client_auth.py`
- Modify: `pyproject.toml`
**Interfaces:**
- Produces exceptions `MyWhooshAuthError`, `MyWhooshTransientError`, `MyWhooshIntegrationError`.
- Produces `MyWhooshClient.ensure_authenticated(email: str, password: str) -> None`.
- Login endpoint: `https://services.mywhoosh.com/http-service/api/login`.
- Login payload fields: `Username`, `Password`, `Platform="Android"`, `Action=1001`, random `CorrelationId`, random `DeviceId`, `Authorization=""`.
- [ ] **Step 1: Add test dependencies**
Add to `pyproject.toml` runtime dependencies:
```toml
"httpx>=0.27,<1",
```
and test dependencies:
```toml
"pytest-asyncio>=0.24,<1",
```
- [ ] **Step 2: Write authentication tests using `httpx.MockTransport`**
```python
# tests/mywhoosh/test_client_auth.py
import httpx
import pytest
from app.mywhoosh.client import MyWhooshClient, MyWhooshAuthError
from app.mywhoosh.models import MyWhooshToken
from app.mywhoosh.tokenstore import MyWhooshTokenStore
@pytest.mark.asyncio
async def test_login_saves_access_refresh_and_whoosh_id(tmp_path) -> None:
async def handler(request: httpx.Request) -> httpx.Response:
assert request.url.path == "/http-service/api/login"
return httpx.Response(
200,
json={
"Success": True,
"AccessToken": "new-access",
"RefreshToken": "new-refresh",
"WhooshId": "w-1",
},
)
store = MyWhooshTokenStore(tmp_path / "mywhoosh.json")
client = MyWhooshClient(store, http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)))
await client.login("rider@example.com", "secret")
assert store.load() == MyWhooshToken("new-access", "new-refresh", "w-1")
@pytest.mark.asyncio
async def test_invalid_credentials_raise_auth_error(tmp_path) -> None:
async def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(200, json={"Success": False, "Message": "Invalid credentials"})
client = MyWhooshClient(
MyWhooshTokenStore(tmp_path / "mywhoosh.json"),
http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)),
)
with pytest.raises(MyWhooshAuthError):
await client.login("rider@example.com", "bad")
```
- [ ] **Step 3: Run and verify failure**
Run: `pytest tests/mywhoosh/test_client_auth.py -v`
Expected: missing client implementation.
- [ ] **Step 4: Implement exception taxonomy and login**
```python
# app/mywhoosh/client.py
from __future__ import annotations
import uuid
import httpx
from app.mywhoosh.models import MyWhooshToken
from app.mywhoosh.tokenstore import MyWhooshTokenStore
LOGIN_URL = "https://services.mywhoosh.com/http-service/api/login"
ACTIVITIES_BASE = "https://service14.mywhoosh.com/v2/"
class MyWhooshError(RuntimeError):
pass
class MyWhooshAuthError(MyWhooshError):
pass
class MyWhooshTransientError(MyWhooshError):
pass
class MyWhooshIntegrationError(MyWhooshError):
pass
class MyWhooshClient:
def __init__(self, token_store: MyWhooshTokenStore, http_client: httpx.AsyncClient | None = None) -> None:
self.token_store = token_store
self.http = http_client or httpx.AsyncClient(timeout=30.0)
self.token = token_store.load()
async def login(self, email: str, password: str) -> None:
payload = {
"Username": email,
"Password": password,
"Platform": "Android",
"Action": 1001,
"CorrelationId": str(uuid.uuid4()),
"DeviceId": str(uuid.uuid4()),
"Authorization": "",
}
try:
response = await self.http.post(LOGIN_URL, json=payload)
except httpx.TransportError as exc:
raise MyWhooshTransientError("MyWhoosh login request failed") from exc
if response.status_code >= 500:
raise MyWhooshTransientError(f"MyWhoosh login returned HTTP {response.status_code}")
if response.status_code >= 400:
raise MyWhooshAuthError(f"MyWhoosh login returned HTTP {response.status_code}")
try:
body = response.json()
except ValueError as exc:
raise MyWhooshIntegrationError("MyWhoosh login returned invalid JSON") from exc
if body.get("Success") is not True or not body.get("AccessToken"):
raise MyWhooshAuthError(str(body.get("Message") or "MyWhoosh login failed"))
self.token = MyWhooshToken(
access_token=str(body["AccessToken"]),
refresh_token=str(body["RefreshToken"]) if body.get("RefreshToken") else None,
whoosh_id=str(body["WhooshId"]) if body.get("WhooshId") else None,
)
self.token_store.save(self.token)
```
- [ ] **Step 5: Implement `ensure_authenticated` as cache-first validation**
Do not invent a refresh endpoint. Validate cached tokens using the normal activities request; on `401/403`, clear the cache, login once, and continue. The later `list_activities()` task provides the request method. Expose the intended behavior now:
```python
async def ensure_authenticated(self, email: str, password: str) -> None:
if self.token is None:
await self.login(email, password)
```
Task 3 extends this with one retry after an unauthorized activities response.
- [ ] **Step 6: Run authentication tests**
Run: `pytest tests/mywhoosh/test_client_auth.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add pyproject.toml app/mywhoosh/client.py tests/mywhoosh/test_client_auth.py
git commit -m "feat: add MyWhoosh API login"
```
## Task 3: Implement MyWhoosh activity listing and FIT download
**Files:**
- Modify: `app/mywhoosh/client.py`
- Create: `tests/mywhoosh/test_client_activities.py`
**Interfaces:**
- Produces `list_activities(email: str, password: str) -> list[MyWhooshActivity]`.
- Produces `download_fit(activity_file_id: str, email: str, password: str) -> bytes`.
- Activities endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/activities` with `{"sortDate":"DESC","page":N}`.
- Download endpoint: `POST https://service14.mywhoosh.com/v2/rider/profile/download-activity-file` with `{"fileId": activity_file_id}`; response `data` is a pre-signed URL which is fetched with GET.
- [ ] **Step 1: Write paginated activity-list test**
```python
@pytest.mark.asyncio
async def test_list_activities_paginates_and_normalizes(tmp_path) -> None:
calls = []
async def handler(request: httpx.Request) -> httpx.Response:
calls.append(str(request.url))
if request.url.path.endswith("/activities"):
payload = json.loads(request.content)
page = payload["page"]
result = {
"data": {
"totalPages": 2,
"results": [{
"id": f"a-{page}",
"title": f"Ride {page}",
"activityFileId": f"f-{page}",
"startDatetime": "2026-08-15T06:00:00.000Z",
}],
}
}
return httpx.Response(200, json=result)
raise AssertionError(request.url)
```
Preload the tokenstore with `access_token="cached"`; assert two normalized `MyWhooshActivity` values are returned.
- [ ] **Step 2: Write expired-token reauthentication test**
The mock transport sequence must return `401` for the first activities request, a successful login response, then `200` for the retried activities request. Assert login is attempted exactly once and the tokenstore contains the new access token.
- [ ] **Step 3: Write FIT download test**
Mock the download-activity-file endpoint to return `{"data":"https://signed.example/activity.fit"}`, then mock that URL to return bytes beginning with a valid FIT header. Assert `download_fit()` returns those exact bytes.
- [ ] **Step 4: Implement one authenticated-request retry helper**
```python
async def _authenticated_post(self, url: str, payload: dict, email: str, password: str) -> httpx.Response:
await self.ensure_authenticated(email, password)
for attempt in range(2):
assert self.token is not None
try:
response = await self.http.post(
url,
json=payload,
headers={"Authorization": f"Bearer {self.token.access_token}"},
)
except httpx.TransportError as exc:
raise MyWhooshTransientError("MyWhoosh request failed") from exc
if response.status_code not in {401, 403}:
if response.status_code >= 500:
raise MyWhooshTransientError(f"MyWhoosh returned HTTP {response.status_code}")
return response
if attempt == 0:
self.token_store.clear()
self.token = None
await self.login(email, password)
continue
raise MyWhooshAuthError("MyWhoosh session rejected after reauthentication")
raise AssertionError("unreachable")
```
- [ ] **Step 5: Implement list normalization and download**
Parse `startDatetime` as UTC when present. Skip malformed activity rows only if they lack no stable `id` or `activityFileId`; otherwise surface JSON/schema failures as `MyWhooshIntegrationError` so API changes are visible.
```python
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
response = await self._authenticated_post(
ACTIVITIES_BASE + "rider/profile/download-activity-file",
{"fileId": activity_file_id},
email,
password,
)
if response.status_code >= 400:
raise MyWhooshIntegrationError(f"download metadata returned HTTP {response.status_code}")
url = response.json().get("data")
if not isinstance(url, str) or not url:
raise MyWhooshIntegrationError("MyWhoosh download response has no URL")
try:
fit_response = await self.http.get(url)
except httpx.TransportError as exc:
raise MyWhooshTransientError("FIT download failed") from exc
if fit_response.status_code >= 500:
raise MyWhooshTransientError(f"FIT host returned HTTP {fit_response.status_code}")
if fit_response.status_code >= 400:
raise MyWhooshIntegrationError(f"FIT host returned HTTP {fit_response.status_code}")
return fit_response.content
```
- [ ] **Step 6: Run MyWhoosh tests**
Run: `pytest tests/mywhoosh -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/mywhoosh/client.py tests/mywhoosh/test_client_activities.py
git commit -m "feat: fetch MyWhoosh activities and FIT files"
```
## Task 4: Implement Garmin import adapter with duplicate and MFA handling
**Files:**
- Create: `app/garmin/uploader.py`
- Create: `tests/garmin/test_uploader.py`
- Modify: `pyproject.toml`
**Interfaces:**
- Produces `GarminUploader.import_fit(fit_path: Path, mfa_code: str | None = None) -> UploadResult`.
- Produces exceptions `GarminUploadBlocked`, `GarminAuthError`, `GarminTransientError`.
- Uses `client.login(tokenstore_path)` and `client.import_activity(activity_path)`.
- [ ] **Step 1: Add Garmin dependency**
Add to runtime dependencies:
```toml
"garminconnect>=0.2,<1",
```
- [ ] **Step 2: Write fake-client tests based on the previously working uploader pattern**
```python
# tests/garmin/test_uploader.py
from pathlib import Path
import pytest
from app.garmin.uploader import GarminUploadBlocked, GarminUploader
class FakeGarmin:
def __init__(self, *args, prompt_mfa=None, import_result=None, login_error=None, import_error=None, **kwargs):
self.prompt_mfa = prompt_mfa
self.import_result = import_result or {"activityId": 42}
self.login_error = login_error
self.import_error = import_error
self.login_path = None
def login(self, tokenstore=None):
self.login_path = tokenstore
if self.login_error:
raise self.login_error
def import_activity(self, activity_path: str):
if self.import_error:
raise self.import_error
return self.import_result
```
Add these concrete tests below the fake client:
```python
def test_successful_import_returns_activity_id(tmp_path: Path) -> None:
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_result={"activityId": 42}),
)
result = uploader.import_fit(tmp_path / "ride.fit")
assert result.status == "imported"
assert result.garmin_activity_id == "42"
def test_duplicate_is_terminal_success(tmp_path: Path) -> None:
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=lambda *a, **kw: FakeGarmin(*a, **kw, import_error=RuntimeError("409 duplicate")),
)
result = uploader.import_fit(tmp_path / "ride.fit")
assert result.duplicate is True
assert result.status == "duplicate"
def test_mfa_without_code_is_blocked(tmp_path: Path) -> None:
class MfaGarmin(FakeGarmin):
def login(self, tokenstore=None):
self.prompt_mfa()
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=MfaGarmin,
)
with pytest.raises(GarminUploadBlocked):
uploader.import_fit(tmp_path / "ride.fit")
def test_mfa_code_is_returned_only_to_prompt(tmp_path: Path) -> None:
seen = []
class MfaGarmin(FakeGarmin):
def login(self, tokenstore=None):
seen.append(self.prompt_mfa())
uploader = GarminUploader(
email="g@example.com",
password="pw",
tokenstore=tmp_path / "garmin",
client_factory=MfaGarmin,
)
uploader.import_fit(tmp_path / "ride.fit", mfa_code="123456")
assert seen == ["123456"]
```
- [ ] **Step 3: Run and verify failure**
Run: `pytest tests/garmin/test_uploader.py -v`
Expected: import failure.
- [ ] **Step 4: Implement the adapter**
```python
# app/garmin/uploader.py
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Protocol
class GarminClientProtocol(Protocol):
def login(self, tokenstore: str | None = None) -> Any: ...
def import_activity(self, activity_path: str) -> Any: ...
@dataclass(frozen=True)
class UploadResult:
status: str
duplicate: bool
garmin_activity_id: str | None
raw_response: Any
class GarminUploadBlocked(RuntimeError):
pass
class GarminAuthError(RuntimeError):
pass
class GarminTransientError(RuntimeError):
pass
```
Implement the uploader fully:
```python
class GarminUploader:
def __init__(
self,
*,
email: str,
password: str,
tokenstore: Path,
client_factory: Callable[..., GarminClientProtocol] | None = None,
) -> None:
self.email = email
self.password = password
self.tokenstore = tokenstore
self.client_factory = client_factory
self._client: GarminClientProtocol | None = None
self._mfa_code: str | None = None
def import_fit(self, fit_path: Path, mfa_code: str | None = None) -> UploadResult:
self._mfa_code = mfa_code
try:
client = self._ensure_client()
try:
response = client.import_activity(str(fit_path))
except Exception as exc:
if _looks_duplicate_error(exc):
return UploadResult("duplicate", True, None, str(exc))
text = str(exc).lower()
if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")):
raise GarminTransientError("Garmin import failed transiently") from exc
raise
return UploadResult("imported", False, _extract_activity_id(response), response)
finally:
self._mfa_code = None
def _ensure_client(self) -> GarminClientProtocol:
if self._client is not None:
return self._client
self.tokenstore.mkdir(parents=True, exist_ok=True)
factory = self.client_factory or _default_garmin_factory
client = factory(self.email, self.password, prompt_mfa=self._prompt_mfa)
try:
client.login(str(self.tokenstore))
except GarminUploadBlocked:
raise
except Exception as exc:
text = str(exc).lower()
if "mfa" in text:
raise GarminUploadBlocked("Garmin MFA is required") from exc
if any(token in text for token in ("password", "credential", "unauthorized", "401")):
raise GarminAuthError("Garmin authentication failed") from exc
if any(token in text for token in ("timeout", "temporar", "connection", "502", "503", "504")):
raise GarminTransientError("Garmin login failed transiently") from exc
raise GarminAuthError("Garmin login failed") from exc
self._client = client
return client
def _prompt_mfa(self) -> str:
if self._mfa_code:
return self._mfa_code
raise GarminUploadBlocked("Garmin requested MFA")
def _default_garmin_factory(*args: Any, **kwargs: Any) -> GarminClientProtocol:
from garminconnect import Garmin
return Garmin(*args, **kwargs)
```
- [ ] **Step 5: Keep duplicate and response-ID extraction deterministic**
Use these helpers:
```python
def _looks_duplicate_error(exc: Exception) -> bool:
text = str(exc).lower()
return any(token in text for token in ("duplicate", "already exists", "409"))
def _extract_activity_id(response: Any) -> str | None:
if not isinstance(response, dict):
return None
candidates = [response.get("activityId"), response.get("activity_id"), response.get("id")]
detailed = response.get("detailedImportResult")
if isinstance(detailed, dict):
candidates.extend([detailed.get("uploadId"), detailed.get("activityId")])
for key in ("successes", "success", "importedActivities"):
items = response.get(key)
if isinstance(items, list) and items and isinstance(items[0], dict):
candidates.extend([items[0].get("activityId"), items[0].get("id")])
return next((str(value) for value in candidates if value is not None), None)
```
- [ ] **Step 6: Run Garmin tests**
Run: `pytest tests/garmin/test_uploader.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add pyproject.toml app/garmin/uploader.py tests/garmin/test_uploader.py
git commit -m "feat: import FIT activities into Garmin"
```

804
docs/superpowers/plans/2026-08-15-sync-scheduler-web.md Normal file
View File

@@ -0,0 +1,804 @@
# Sync Engine, Scheduler, and Operational UI Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Integrate the database, MyWhoosh client, FIT rewriter, Garmin importer, scheduler, MFA workflow, retry behavior, and operational admin pages into a resilient multi-user sync service.
**Architecture:** A `SyncManager` owns per-user `asyncio.Lock` instances and executes a durable activity state machine. External clients are injected via factories for tests. A lightweight FastAPI lifespan scheduler triggers syncs at the configured interval; different users run concurrently, while each user's pipeline is serialized.
**Tech Stack:** Python 3.12, asyncio, FastAPI lifespan, SQLAlchemy, HTMX, Jinja2, pytest/pytest-asyncio.
## Global Constraints
- Multiple users sync independently and may run concurrently.
- At most one sync may run for a given user at a time.
- Manual sync and scheduled sync use the same pipeline and lock.
- Durable activity stages are `discovered`, `downloaded`, `converted`, `imported`, `duplicate`, `failed` with `last_completed_stage` retained on failure.
- `imported` and `duplicate` are terminal.
- Transient network/server failures retry at most once in a run; later attempts occur on future scheduler ticks.
- Invalid MyWhoosh/Garmin credentials and Garmin MFA set `action_required`.
- Corrupt/unsupported FIT is non-retryable per activity.
- Failure of one user or one activity must never stop other users.
- Original and converted FIT files remain on disk in v1.
- MFA codes are never persisted or logged.
---
## File Structure
```text
app/sync/
__init__.py
states.py
manager.py
scheduler.py
app/web/
operations.py
templates/
dashboard.html
users/detail.html
system.html
fragments/user_card.html
fragments/sync_result.html
fragments/mfa_form.html
app/db/
repositories.py
tests/sync/
fakes.py
test_manager.py
test_concurrency.py
test_scheduler.py
tests/web/
test_operations.py
test_mfa.py
```
## Task 1: Add durable activity/sync-run repository operations
**Files:**
- Modify: `app/db/repositories.py`
- Create: `tests/db/test_sync_state.py`
**Interfaces:**
- Produces methods to advance activity stages, mark failures without losing `last_completed_stage`, list pending activities, and create/finalize sync runs.
- [ ] **Step 1: Write failing state-transition tests**
```python
# tests/db/test_sync_state.py
from app.db.models import ActivityStatus
def test_failure_retains_last_completed_stage(activity_repository, seeded_activity) -> None:
activity_repository.mark_downloaded(seeded_activity.id, "/data/activities/1/a/source.fit")
activity_repository.mark_failed(seeded_activity.id, "Garmin timeout", retryable=True)
activity = activity_repository.get(seeded_activity.id)
assert activity.status == ActivityStatus.FAILED
assert activity.last_completed_stage == ActivityStatus.DOWNLOADED
assert activity.retryable is True
def test_converted_activity_is_pending_until_terminal(activity_repository, seeded_activity) -> None:
activity_repository.mark_converted(seeded_activity.id, "/data/activities/1/a/converted.fit")
ids = [item.id for item in activity_repository.list_pending_for_user(seeded_activity.user_id)]
assert seeded_activity.id in ids
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/db/test_sync_state.py -v`
Expected: missing repository methods.
- [ ] **Step 3: Implement explicit transition methods**
Add methods with these exact effects:
```python
def mark_downloaded(self, activity_id: int, path: str) -> Activity:
activity = self._require(activity_id)
activity.source_fit_path = path
activity.status = ActivityStatus.DOWNLOADED
activity.last_completed_stage = ActivityStatus.DOWNLOADED
activity.last_error = None
activity.retryable = True
self.session.commit()
return activity
def mark_converted(self, activity_id: int, path: str) -> Activity:
activity = self._require(activity_id)
activity.converted_fit_path = path
activity.status = ActivityStatus.CONVERTED
activity.last_completed_stage = ActivityStatus.CONVERTED
activity.last_error = None
activity.retryable = True
self.session.commit()
return activity
def mark_imported(self, activity_id: int, garmin_activity_id: str | None) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.IMPORTED
activity.last_completed_stage = ActivityStatus.IMPORTED
activity.garmin_activity_id = garmin_activity_id
activity.last_error = None
activity.retryable = False
self.session.commit()
return activity
def mark_duplicate(self, activity_id: int) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.DUPLICATE
activity.last_completed_stage = ActivityStatus.DUPLICATE
activity.last_error = None
activity.retryable = False
self.session.commit()
return activity
def mark_failed(self, activity_id: int, error: str, *, retryable: bool) -> Activity:
activity = self._require(activity_id)
activity.status = ActivityStatus.FAILED
activity.last_error = error[:2000]
activity.retryable = retryable
self.session.commit()
return activity
```
`list_pending_for_user()` must exclude terminal states and include failed rows only when `retryable=True`.
- [ ] **Step 4: Add `SyncRunRepository` create/finalize methods**
`start(user_id)` creates `RUNNING`; `finish(...)` sets counts, `finished_at`, status, and optional summary error. Do not store exception tracebacks in SQLite.
- [ ] **Step 5: Run DB state tests**
Run: `pytest tests/db/test_sync_state.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/db/repositories.py tests/db/test_sync_state.py
git commit -m "feat: add durable sync state transitions"
```
## Task 2: Implement the single-user sync state machine
**Files:**
- Create: `app/sync/states.py`
- Create: `app/sync/manager.py`
- Create: `tests/sync/fakes.py`
- Create: `tests/sync/test_manager.py`
**Interfaces:**
- Produces `SyncManager.sync_user(user_id: int, mfa_code: str | None = None) -> SyncOutcome`.
- Constructor receives `session_factory`, `credential_cipher`, `settings`, `mywhoosh_factory`, `garmin_factory`, and `fit_converter`.
- `mywhoosh_factory(token_store: MyWhooshTokenStore) -> MyWhooshClient`.
- `garmin_factory(email: str, password: str, tokenstore: Path) -> GarminUploader`.
- `fit_converter(source_path: Path, output_path: Path) -> FitConversionResult`.
- [ ] **Step 1: Define result models and fake integration factories**
```python
# app/sync/states.py
from dataclasses import dataclass
@dataclass(frozen=True)
class SyncOutcome:
user_id: int
status: str
discovered: int
imported: int
skipped: int
failed: int
message: str | None = None
```
Implement concrete fakes:
```python
# tests/sync/fakes.py
from app.garmin.uploader import UploadResult
class FakeMyWhooshClient:
def __init__(self, activities, fit_bytes: bytes) -> None:
self.activities = activities
self.fit_bytes = fit_bytes
self.list_calls = 0
self.download_calls = 0
async def list_activities(self, email: str, password: str):
self.list_calls += 1
return list(self.activities)
async def download_fit(self, activity_file_id: str, email: str, password: str) -> bytes:
self.download_calls += 1
return self.fit_bytes
class FakeGarminUploader:
def __init__(self, result: UploadResult | None = None, error: Exception | None = None) -> None:
self.result = result or UploadResult("imported", False, "g-1", {"activityId": "g-1"})
self.error = error
self.calls = 0
def import_fit(self, fit_path, mfa_code=None):
self.calls += 1
if self.error is not None:
raise self.error
return self.result
```
- [ ] **Step 2: Write the happy-path test**
```python
@pytest.mark.asyncio
async def test_new_activity_downloads_converts_and_imports(manager, seeded_user, tmp_path) -> None:
outcome = await manager.sync_user(seeded_user.id)
assert outcome.discovered == 1
assert outcome.imported == 1
assert outcome.failed == 0
activity = load_only_activity(seeded_user.id)
assert activity.status == ActivityStatus.IMPORTED
assert Path(activity.source_fit_path).exists()
assert Path(activity.converted_fit_path).exists()
```
- [ ] **Step 3: Write resume tests before implementation**
```python
@pytest.mark.asyncio
@pytest.mark.parametrize(
("status", "last_stage", "expected_downloads", "expected_conversions", "expected_imports"),
[
(ActivityStatus.DOWNLOADED, ActivityStatus.DOWNLOADED, 0, 1, 1),
(ActivityStatus.CONVERTED, ActivityStatus.CONVERTED, 0, 0, 1),
(ActivityStatus.IMPORTED, ActivityStatus.IMPORTED, 0, 0, 0),
(ActivityStatus.FAILED, ActivityStatus.CONVERTED, 0, 0, 1),
],
)
async def test_resume_from_durable_stage(
manager_factory, seeded_activity_factory, status, last_stage,
expected_downloads, expected_conversions, expected_imports,
) -> None:
activity = seeded_activity_factory(status=status, last_completed_stage=last_stage, retryable=True)
manager, mywhoosh, converter, garmin = manager_factory(activity)
await manager.sync_user(activity.user_id)
assert mywhoosh.download_calls == expected_downloads
assert converter.calls == expected_conversions
assert garmin.calls == expected_imports
```
- [ ] **Step 4: Run and verify failure**
Run: `pytest tests/sync/test_manager.py -v`
Expected: missing manager.
- [ ] **Step 5: Implement per-activity filesystem layout and state machine**
Use paths:
```python
activity_dir = settings.activities_dir / str(user.id) / activity.mywhoosh_activity_id
source_path = activity_dir / "source.fit"
converted_path = activity_dir / "edge-1030-plus.fit"
```
Create per-user integration instances from decrypted credentials and isolated token paths:
```python
mw_email = self.credential_cipher.decrypt(user.mywhoosh_email_enc)
mw_password = self.credential_cipher.decrypt(user.mywhoosh_password_enc)
garmin_email = self.credential_cipher.decrypt(user.garmin_email_enc)
garmin_password = self.credential_cipher.decrypt(user.garmin_password_enc)
token_dir = self.settings.tokens_dir / str(user.id)
mywhoosh = self.mywhoosh_factory(MyWhooshTokenStore(token_dir / "mywhoosh.json"))
garmin = self.garmin_factory(garmin_email, garmin_password, token_dir / "garmin")
remote_activities = await mywhoosh.list_activities(mw_email, mw_password)
```
For each remote activity, call `get_or_create_discovered(...)`, then resume from `activity.last_completed_stage` when `activity.status == FAILED`; otherwise use `activity.status`.
Core sequence:
```python
if stage == ActivityStatus.DISCOVERED:
fit_bytes = await mywhoosh.download_fit(remote.activity_file_id, mw_email, mw_password)
activity_dir.mkdir(parents=True, exist_ok=True)
source_path.write_bytes(fit_bytes)
repo.mark_downloaded(activity.id, str(source_path))
if stage in {ActivityStatus.DOWNLOADED}:
fit_converter(source_path, converted_path)
repo.mark_converted(activity.id, str(converted_path))
if stage in {ActivityStatus.CONVERTED}:
upload = await asyncio.to_thread(garmin.import_fit, converted_path, mfa_code)
if upload.duplicate:
repo.mark_duplicate(activity.id)
else:
repo.mark_imported(activity.id, upload.garmin_activity_id)
```
After each repository transition, update the local `stage` variable from the returned record so resume behavior is deterministic.
- [ ] **Step 6: Implement exception mapping**
Map exceptions with explicit user connection-state updates:
```python
except MyWhooshTransientError as exc:
user.health_state = HealthState.DEGRADED
user.mywhoosh_state = "error"
repo.mark_failed(activity.id, str(exc), retryable=True)
except MyWhooshAuthError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.mywhoosh_state = "auth_required"
user.action_reason = "mywhoosh_auth_required"
stop_user_run = True
except MyWhooshIntegrationError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.mywhoosh_state = "integration_error"
user.action_reason = "mywhoosh_integration_changed"
stop_user_run = True
except GarminUploadBlocked:
user.health_state = HealthState.ACTION_REQUIRED
user.garmin_state = "mfa_required"
user.action_reason = "garmin_mfa_required"
stop_user_run = True
except GarminAuthError as exc:
user.health_state = HealthState.ACTION_REQUIRED
user.garmin_state = "auth_required"
user.action_reason = "garmin_auth_required"
stop_user_run = True
except GarminTransientError as exc:
user.health_state = HealthState.DEGRADED
user.garmin_state = "error"
repo.mark_failed(activity.id, str(exc), retryable=True)
except FitFormatError as exc:
repo.mark_failed(activity.id, str(exc), retryable=False)
```
On successful MyWhoosh listing set `mywhoosh_state="connected"`; on successful Garmin import set `garmin_state="connected"`. Persist the user after each connection-state change. Unexpected exceptions mark the run/user `degraded` and log only exception class plus sanitized message.
- [ ] **Step 7: Run manager tests**
Run: `pytest tests/sync/test_manager.py -v`
Expected: PASS.
- [ ] **Step 8: Commit**
```bash
git add app/sync/states.py app/sync/manager.py tests/sync
git commit -m "feat: add resumable per-user sync pipeline"
```
## Task 3: Add per-user locks and cross-user isolation
**Files:**
- Modify: `app/sync/manager.py`
- Create: `tests/sync/test_concurrency.py`
**Interfaces:**
- Produces `SyncAlreadyRunning` and ensures only one active `sync_user()` call per user.
- [ ] **Step 1: Write concurrency tests**
```python
@pytest.mark.asyncio
async def test_same_user_cannot_run_twice(manager, seeded_user) -> None:
first_started = asyncio.Event()
release_first = asyncio.Event()
manager.test_hooks = SyncTestHooks(first_started=first_started, release=release_first)
first = asyncio.create_task(manager.sync_user(seeded_user.id))
await first_started.wait()
with pytest.raises(SyncAlreadyRunning):
await manager.sync_user(seeded_user.id)
release_first.set()
await first
@pytest.mark.asyncio
async def test_different_users_can_run_concurrently(manager, user_a, user_b) -> None:
results = await asyncio.gather(manager.sync_user(user_a.id), manager.sync_user(user_b.id))
assert {result.user_id for result in results} == {user_a.id, user_b.id}
```
Do not leave production-only `test_hooks`; instead inject a fake MyWhoosh client whose `list_activities()` blocks on test events.
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/sync/test_concurrency.py -v`
Expected: same-user duplicate execution is not yet blocked.
- [ ] **Step 3: Implement lock registry**
```python
class SyncAlreadyRunning(RuntimeError):
pass
class SyncManager:
def __init__(...):
self._locks: dict[int, asyncio.Lock] = {}
self._locks_guard = asyncio.Lock()
async def _lock_for(self, user_id: int) -> asyncio.Lock:
async with self._locks_guard:
return self._locks.setdefault(user_id, asyncio.Lock())
async def sync_user(self, user_id: int, mfa_code: str | None = None) -> SyncOutcome:
lock = await self._lock_for(user_id)
if lock.locked():
raise SyncAlreadyRunning(f"sync already running for user {user_id}")
async with lock:
return await self._sync_user_locked(user_id, mfa_code)
```
- [ ] **Step 4: Add a `sync_all_enabled()` isolation method**
```python
async def sync_all_enabled(self) -> list[SyncOutcome | Exception]:
user_ids = self._load_enabled_user_ids()
return await asyncio.gather(
*(self.sync_user(user_id) for user_id in user_ids),
return_exceptions=True,
)
```
A failure for one user must appear as one list element and must not cancel sibling jobs.
- [ ] **Step 5: Run concurrency tests**
Run: `pytest tests/sync/test_concurrency.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/sync/manager.py tests/sync/test_concurrency.py
git commit -m "feat: isolate concurrent user syncs"
```
## Task 4: Add the periodic scheduler through FastAPI lifespan
**Files:**
- Create: `app/sync/scheduler.py`
- Modify: `app/main.py`
- Create: `tests/sync/test_scheduler.py`
**Interfaces:**
- Produces `SyncScheduler.start()`, `stop()`, `run_once()`, `last_tick`, `next_tick`.
- Scheduler interval is `Settings.sync_interval_minutes`.
- [ ] **Step 1: Write scheduler test with a short injected interval**
```python
@pytest.mark.asyncio
async def test_scheduler_calls_sync_all_and_survives_failure() -> None:
fake = FakeSyncManager(results=[RuntimeError("one user failed")])
scheduler = SyncScheduler(fake, interval_seconds=0.01)
await scheduler.start()
await asyncio.sleep(0.035)
await scheduler.stop()
assert fake.calls >= 2
assert scheduler.last_tick is not None
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/sync/test_scheduler.py -v`
Expected: missing scheduler.
- [ ] **Step 3: Implement scheduler loop**
```python
class SyncScheduler:
def __init__(self, manager, *, interval_seconds: float) -> None:
self.manager = manager
self.interval_seconds = interval_seconds
self._task: asyncio.Task | None = None
self._stop = asyncio.Event()
self.last_tick = None
self.next_tick = None
async def run_once(self) -> None:
self.last_tick = datetime.now(timezone.utc)
await self.manager.sync_all_enabled()
self.next_tick = datetime.now(timezone.utc) + timedelta(seconds=self.interval_seconds)
async def _run(self) -> None:
while not self._stop.is_set():
await self.run_once()
try:
await asyncio.wait_for(self._stop.wait(), timeout=self.interval_seconds)
except TimeoutError:
pass
```
`stop()` sets the event and awaits the task. Never allow one `sync_all_enabled()` exception to kill the loop; log it and continue.
- [ ] **Step 4: Wire into FastAPI lifespan**
Build the concrete `SyncManager` once during app startup, store it on `app.state.sync_manager`, create `SyncScheduler(... interval_minutes * 60)`, start it, and stop it during lifespan shutdown.
- [ ] **Step 5: Run scheduler tests**
Run: `pytest tests/sync/test_scheduler.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/sync/scheduler.py app/main.py tests/sync/test_scheduler.py
git commit -m "feat: schedule periodic user synchronization"
```
## Task 5: Add dashboard/manual sync/system operational routes
**Files:**
- Create: `app/web/operations.py`
- Modify: `app/web/routes.py`
- Modify: `app/web/templates/dashboard.html`
- Create: `app/web/templates/system.html`
- Create: `app/web/templates/fragments/sync_result.html`
- Create: `tests/web/test_operations.py`
**Interfaces:**
- Routes: `POST /users/{id}/sync`, `POST /sync-all`, `GET /system`.
- Manual actions use the same `SyncManager` instance and lock as the scheduler.
- [ ] **Step 1: Write manual-sync tests**
```python
def test_manual_sync_calls_shared_manager(authenticated_client, fake_sync_manager) -> None:
response = authenticated_client.post(
"/users/1/sync",
data={"csrf_token": authenticated_client.csrf_token},
)
assert response.status_code == 200
assert fake_sync_manager.user_calls == [1]
def test_manual_sync_reports_already_running(authenticated_client, fake_sync_manager) -> None:
fake_sync_manager.raise_already_running = True
response = authenticated_client.post(
"/users/1/sync",
data={"csrf_token": authenticated_client.csrf_token},
)
assert response.status_code == 409
assert "already running" in response.text.lower()
```
- [ ] **Step 2: Run and verify failure**
Run: `pytest tests/web/test_operations.py -v`
Expected: routes missing.
- [ ] **Step 3: Implement routes with admin and CSRF checks**
Each state-changing route must execute in this order:
```python
require_admin(request)
validate_csrf(request, csrf_token)
```
Then call `await request.app.state.sync_manager.sync_user(user_id)` or `sync_all_enabled()`.
- [ ] **Step 4: Expand dashboard data**
Add a repository projection that contains only safe display fields:
```python
@dataclass(frozen=True)
class UserDashboardRow:
id: int
name: str
enabled: bool
health_state: str
action_reason: str | None
last_sync_at: datetime | None
last_activity_name: str | None
last_activity_status: str | None
def dashboard_rows(self) -> list[UserDashboardRow]:
users = self.list_all()
rows = []
for user in users:
last_run = self.session.scalar(
select(SyncRun).where(SyncRun.user_id == user.id).order_by(SyncRun.started_at.desc()).limit(1)
)
last_activity = self.session.scalar(
select(Activity).where(Activity.user_id == user.id).order_by(Activity.created_at.desc()).limit(1)
)
rows.append(UserDashboardRow(
id=user.id,
name=user.name,
enabled=user.enabled,
health_state=user.health_state.value,
action_reason=user.action_reason,
last_sync_at=last_run.finished_at if last_run else None,
last_activity_name=last_activity.activity_name if last_activity else None,
last_activity_status=last_activity.status.value if last_activity else None,
))
return rows
```
Pass only these rows to `dashboard.html`. Render the MFA action only when `row.action_reason == "garmin_mfa_required"`. No decrypted credential is part of this projection.
- [ ] **Step 5: Implement read-only system page**
Expose application version, configured interval, scheduler `last_tick` and `next_tick`, user count, and activity count. The only action is a CSRF-protected `sync all now` POST.
- [ ] **Step 6: Run tests**
Run: `pytest tests/web/test_operations.py -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add app/web tests/web/test_operations.py
git commit -m "feat: add operational sync controls"
```
## Task 6: Add Garmin MFA lifecycle and failed-activity retry
**Files:**
- Modify: `app/web/routes.py`
- Modify: `app/web/templates/users/detail.html`
- Create: `app/web/templates/fragments/mfa_form.html`
- Create: `tests/web/test_mfa.py`
- Modify: `app/sync/manager.py`
**Interfaces:**
- Route: `POST /users/{id}/garmin-mfa` with one-time `code`.
- Route: `POST /activities/{id}/retry`.
- MFA code exists only in request memory and the immediate `sync_user(user_id, mfa_code=code)` call.
- [ ] **Step 1: Write MFA lifecycle test**
```python
def test_mfa_code_is_used_once_and_not_persisted(authenticated_client, fake_sync_manager, db_session) -> None:
response = authenticated_client.post(
"/users/1/garmin-mfa",
data={"csrf_token": authenticated_client.csrf_token, "code": "123456"},
)
assert response.status_code == 200
assert fake_sync_manager.mfa_calls == [(1, "123456")]
persisted_text = " ".join(str(row) for row in db_session.execute(text("select * from sync_runs")).all())
assert "123456" not in persisted_text
```
- [ ] **Step 2: Write retry test for non-terminal failed activity**
Assert the route changes a retryable failed activity back to `status=last_completed_stage`, clears `last_error`, then calls the user's normal sync. Reject retry for `retryable=False` with HTTP 409.
- [ ] **Step 3: Implement MFA route**
Validate code as a non-empty short string, never log it, and call:
```python
outcome = await request.app.state.sync_manager.sync_user(user_id, mfa_code=code.strip())
```
After a successful Garmin login/import, clear `action_reason` and restore health to `healthy` or `degraded` according to the resulting sync outcome.
- [ ] **Step 4: Implement failed-activity reset operation**
Repository method:
```python
def reset_retryable_failure(self, activity_id: int) -> Activity:
activity = self._require(activity_id)
if activity.status != ActivityStatus.FAILED or not activity.retryable:
raise ValueError("activity is not retryable")
activity.status = activity.last_completed_stage
activity.last_error = None
self.session.commit()
return activity
```
- [ ] **Step 5: Run MFA/retry tests**
Run: `pytest tests/web/test_mfa.py tests/web/test_operations.py -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add app/web app/sync/manager.py app/db/repositories.py tests/web/test_mfa.py
git commit -m "feat: handle Garmin MFA and activity retries"
```
## Task 7: End-to-end regression and Docker acceptance
**Files:**
- Modify: `docker-compose.example.yml` only if integration exposes a missing runtime configuration
- Create: `tests/test_acceptance.py`
**Interfaces:**
- No new interface; verifies the v1 acceptance criteria with fake external services.
- [ ] **Step 1: Add an application-level acceptance test with two users**
Build the app with temporary SQLite/data directories and injected fake MyWhoosh/Garmin factories. Seed two enabled users, give each one distinct remote activity IDs, run `sync_all_enabled()`, and assert:
```python
assert all(result.status == "success" for result in results)
assert count_terminal_activities(user_a.id) == 1
assert count_terminal_activities(user_b.id) == 1
assert user_a_source_path.parent != user_b_source_path.parent
assert user_a_garmin_factory.tokenstore != user_b_garmin_factory.tokenstore
```
- [ ] **Step 2: Add isolation acceptance test**
Configure User B to raise `GarminUploadBlocked`; assert User A still imports and User B ends `action_required` with no impact on User A.
- [ ] **Step 3: Run the full suite**
Run: `pytest -v`
Expected: PASS.
- [ ] **Step 4: Build Docker image again**
Run: `docker build -t mywhoosh-garmin-sync:test .`
Expected: successful build with the complete dependency set.
- [ ] **Step 5: Start local container and exercise smoke paths**
Start with a temporary bind-mounted `/data`, then verify:
```bash
curl -fsS http://127.0.0.1:18080/healthz
curl -I http://127.0.0.1:18080/
```
Expected: health JSON and dashboard redirect to `/login` when unauthenticated.
- [ ] **Step 6: Verify secrets are absent from captured test logs**
Run:
```bash
pytest -v 2>&1 | tee /tmp/mywhoosh-garmin-test.log
! grep -F "mw-secret" /tmp/mywhoosh-garmin-test.log
! grep -F "garmin-secret" /tmp/mywhoosh-garmin-test.log
! grep -F "123456" /tmp/mywhoosh-garmin-test.log
```
Expected: all three negated `grep` commands succeed.
- [ ] **Step 7: Commit**
```bash
git add tests/test_acceptance.py docker-compose.example.yml
git commit -m "test: cover multi-user sync acceptance"
```