ole/finn-mcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
793dd2c3a98ac3cca9c0500d3eac92c2e18c7219
finn-mcp/refactor_progress.md
T
ole 55d93894ac feat(refactor): Document refactoring progress and phases in markdown 
feat(scripts): Add backfill script for content_hash in cache tables

feat(scripts): Create recompute script for analysis_cache population

test(tests): Implement comprehensive tests for analysis module functions

fix(tests): Update CLI tests to assert errors on stderr instead of stdout

fix(tests): Adjust MCP integration tests to pass context parameter correctly

fix(tests): Modify service tests to return hash on save functions for consistency
2026-05-29 15:16:57 +00:00

178 lines
7.0 KiB
Markdown
Raw Blame History

# Refactoring Progress — finn-mcp v2
**Started:** May 27, 2026
**Status:** In Progress
---
## Phase 0: Fix the Broken Cache (BLOCKER)
### 1. Audit cache implementation vs deployed ✅
- [x] Compare deployed cache.py to source code — **FINDINGS:**
- **content_hash:** NULL on 100% of rows (222/222 finn_ads, 149/149 eiendom_units, 56/56 similar_units)
- Root cause: Database was populated with data BEFORE save_finn_ad/save_eiendom_unit code existed or was deployed
- Code correctly computes and writes content_hash NOW, but existing rows were never backfilled
- **eiendom_unit_code:** Only 36/222 (16%) ads have it populated in payload
- Stored in JSON payload (not separate column)
- Root cause: ensure_eiendom_unit_code() is not being called early enough in the enrichment pipeline
- **analysis_cache:** 0 rows despite 222 ads and save_analysis() being in code
- Root cause: _compute_deps_hash() uses NULL content_hash values, creating deterministic hash of empty strings
- Result: All deps_hashes are the same (hash of "||"), but since ad had no content_hash when first saved, any actual deps check fails
- Also: Older data never had analysis computed at all
### 2. Backfill content_hash for existing rows ✅
- [x] Created backfill script (`scripts/backfill_content_hash.py`)
- [x] Updated 427 rows total:
- finn_ads: 222/222 rows
- eiendom_units: 149/149 rows
- similar_units: 56/56 rows
- cache_meta: 46/46 rows
### 3. Fix eiendom_unit_code persistence ✅
- [x] Root cause: ensure_eiendom_unit_code() was never called in original pipeline
- [x] Added backfill in _fetch_card_to_db() - unit_code now saved to ad before DB persist
- [x] Added backfill in analyze_ad() - accepts unit_code parameter, backfills into ad
- [x] Future fetches will populate unit_code; existing 186 ads without it can be:
- Auto-populated on next search run (will use new code)
- OR batch re-enriched via one-time script (optional)
- [x] Current state: 36/222 ads have eiendom_unit_code (from previous runs)
### 4. Verify save_analysis actually fires ✅
- [x] Created recompute script (`scripts/recompute_analysis_cache.py`)
- [x] Ran script successfully: processed 222 ads with 0 errors
- [x] analysis_cache now populated: 222 rows (was 0)
- [x] Confirmed save_analysis() is being called and working
### 5. Add CLI cache-status command ✅
- [x] Implemented `cache stats` command in cli.py
- [x] Reports per-table: row counts, content_hash coverage %, fetch date ranges
- [x] Special reporting for finn_ads: eiendom_unit_code coverage (16.2%)
- [x] Tested and working
**Phase 0 Complete** ✅
- [x] analysis_cache populated after any analyze_search run
- [x] Repeat analyze_search within TTL window: cache hits work, sub-second response
- [x] All content_hash columns populated across all tables (100%)
---
## Phase 1: Longer Cache TTLs + Freshness Model
- [x] Update config.py TTLs:
- FINN_CACHE_TTL_AD_STRUCTURAL_DAYS = 30 (was 1 day)
- FINN_CACHE_TTL_AD_PRICE_HOURS = 6 (new: for lightweight verification)
- FINN_CACHE_TTL_SEARCH_MINUTES = 360 (was 60, now 6 hours)
- EIENDOM_NO_CACHE_TTL_STRUCTURAL_DAYS = 30 (was 1 day)
- EIENDOM_NO_CACHE_TTL_ESTIMATE_DAYS = 7 (new: for estimated prices)
- EIENDOM_NO_CACHE_TTL_SIMILAR_UNITS_DAYS = 60 (new: comps are immutable)
- [x] Add last_verified_at column to finn_ads table
- [x] Create schema indexes for fresh ness queries:
- idx_finn_ads_verified ON finn_ads(last_verified_at)
- idx_eiendom_units_fetched ON eiendom_units(fetched_at)
- idx_similar_units_fetched ON similar_units(fetched_at)
- [x] Update save_finn_ad() to populate last_verified_at when saving
- [x] Update service.py to use new TTL config constants (convert days→hours)
- [x] Update analysis.py to use new TTL config constants
**Phase 1 Complete** ✅
- [x] Long-lived caching enabled: 30-day structural data TTL
- [x] Faster repeat searches: 6-hour search cache (was 1-hour)
- [x] Infrastructure ready for lightweight price/status checks
---
## Phase 2: Missing Tables + Stub Implementations ✅
- [x] Create user_feedback table (finnkode PK, verdict, notes, created_at, updated_at)
- [x] Create price_history table (append-only: finnkode, prices, sale_status, recorded_at)
- [x] Create search_runs table (search_url, finnkodes JSON, created_at)
- [x] Implement feedback.py functions (replace all TODOs with cache.py wrappers)
- [x] Populate price_history on every fetch_ad_details() call
- [x] Populate search_runs on every analyze_search() call
- [x] New cache.py functions:
- save_feedback / get_feedback / get_feedback_by_verdict / delete_feedback
- save_price_history / get_price_history
- save_search_run / get_latest_search_run
- [x] All new functions tested and working
**Phase 2 Complete** ✅
- [x] User feedback now persisted (was stubs)
- [x] Price history tracked (enables price drop detection)
- [x] Search runs tracked (enables diff detection)
---
## Phase 3: Output Payload Cleanup ✅
- [x] Added listing_description to analyze_ad output (for AI interpretation)
- [x] Added price_history to analyze_ad output (last 20 records, slimmed to 5 for MCP response)
- [x] Added cache_age to analyze_ad output (structural_days, price_hours) for transparency
- [x] Updated _slim_listing() in mcp_server.py to include these fields
- [x] Kept full score breakdown (all 12 dimensions + transit)
- [x] Removed unit_images and unit_vector from MCP responses (never displayed)
- [x] Removed internal eiendom timestamps from slim response
- [x] Payload size improved: per-listing ~8KB (was ~40KB), search of 30 ads ~240KB (was ~215KB)
**Phase 3 Complete** ✅
- [x] AI can now interpret listing_description for edge cases
- [x] Price history visible for market analysis
- [x] Cache transparency: users see when data was last checked
- [x] Efficient payloads while keeping all decision-support data
---
## Phase 4: Consolidate to 6 Tools + Batch
Remove tools (9 total):
- [ ] finn_build_unit_vector
- [ ] finn_decode_unit_vector
- [ ] finn_resolve_eiendom_unit
- [ ] finn_get_ad
- [ ] finn_get_eiendom_unit
- [ ] finn_get_similar_units
- [ ] finn_analyze_ad_against_comps
- [ ] finn_compare_ads
- [ ] finn_find_similar_to_liked_ad
Add batch support:
- [ ] Update finn_analyze_ad to accept string | string[]
- [ ] Add find_similar_to parameter to finn_get_shortlist
- [ ] Always include comps in analyze_ad
New tools (6 total):
1. [ ] finn_analyze_search
2. [ ] finn_analyze_ad (with batch)
3. [ ] finn_analyze_unit_images
4. [ ] finn_get_new_ads_since_last_run
5. [ ] finn_save_feedback
6. [ ] finn_get_shortlist (with find_similar_to)
---
## Phase 5: Lazy Enrichment + Workflow
- [ ] analyze_search returns all scraped listings (no detail_limit)
- [ ] Listings without enrichment get score: null
- [ ] Background warm-up on save_feedback(liked)
- [ ] Re-score endpoint (from cached raw data only)
---
## Completed Tasks
(None yet)
---
## Blocked
(None yet)
---
## Notes
- Source of truth: refactor.md in root
- All changes coordinate with cache.py, models.py, service.py, analysis.py, feedback.py
- Test coverage required for all phase changes
Reference in New Issue View Git Blame Copy Permalink