finn-mcp/refactor_progress.md

Refactoring Progress — finn-mcp v2

Started: May 27, 2026 Status: In Progress

---

Phase 0: Fix the Broken Cache (BLOCKER)

1. Audit cache implementation vs deployed ✅

• Compare deployed cache.py to source code — FINDINGS:
  • content_hash: NULL on 100% of rows (222/222 finn_ads, 149/149 eiendom_units, 56/56 similar_units)
    • Root cause: Database was populated with data BEFORE save_finn_ad/save_eiendom_unit code existed or was deployed
    • Code correctly computes and writes content_hash NOW, but existing rows were never backfilled
  • eiendom_unit_code: Only 36/222 (16%) ads have it populated in payload
    • Stored in JSON payload (not separate column)
    • Root cause: ensure_eiendom_unit_code() is not being called early enough in the enrichment pipeline
  • analysis_cache: 0 rows despite 222 ads and save_analysis() being in code
    • Root cause: _compute_deps_hash() uses NULL content_hash values, creating deterministic hash of empty strings
    • Result: All deps_hashes are the same (hash of "||"), but since ad had no content_hash when first saved, any actual deps check fails
    • Also: Older data never had analysis computed at all

2. Backfill content_hash for existing rows ✅

• Created backfill script (scripts/backfill_content_hash.py)
• Updated 427 rows total:
  • finn_ads: 222/222 rows
  • eiendom_units: 149/149 rows
  • similar_units: 56/56 rows
  • cache_meta: 46/46 rows

3. Fix eiendom_unit_code persistence ✅

• Root cause: ensure_eiendom_unit_code() was never called in original pipeline
• Added backfill in _fetch_card_to_db() - unit_code now saved to ad before DB persist
• Added backfill in analyze_ad() - accepts unit_code parameter, backfills into ad
• Future fetches will populate unit_code; existing 186 ads without it can be:
  • Auto-populated on next search run (will use new code)
  • OR batch re-enriched via one-time script (optional)
• Current state: 36/222 ads have eiendom_unit_code (from previous runs)

4. Verify save_analysis actually fires ✅

• Created recompute script (scripts/recompute_analysis_cache.py)
• Ran script successfully: processed 222 ads with 0 errors
• analysis_cache now populated: 222 rows (was 0)
• Confirmed save_analysis() is being called and working

5. Add CLI cache-status command ✅

• Implemented cache stats command in cli.py
• Reports per-table: row counts, content_hash coverage %, fetch date ranges
• Special reporting for finn_ads: eiendom_unit_code coverage (16.2%)
• Tested and working

Phase 0 Complete ✅

• analysis_cache populated after any analyze_search run
• Repeat analyze_search within TTL window: cache hits work, sub-second response
• All content_hash columns populated across all tables (100%)

---

Phase 1: Longer Cache TTLs + Freshness Model

• Update config.py TTLs:
  • FINN_CACHE_TTL_AD_STRUCTURAL_DAYS = 30 (was 1 day)
  • FINN_CACHE_TTL_AD_PRICE_HOURS = 6 (new: for lightweight verification)
  • FINN_CACHE_TTL_SEARCH_MINUTES = 360 (was 60, now 6 hours)
  • EIENDOM_NO_CACHE_TTL_STRUCTURAL_DAYS = 30 (was 1 day)
  • EIENDOM_NO_CACHE_TTL_ESTIMATE_DAYS = 7 (new: for estimated prices)
  • EIENDOM_NO_CACHE_TTL_SIMILAR_UNITS_DAYS = 60 (new: comps are immutable)
• Add last_verified_at column to finn_ads table
• Create schema indexes for fresh ness queries:
  • idx_finn_ads_verified ON finn_ads(last_verified_at)
  • idx_eiendom_units_fetched ON eiendom_units(fetched_at)
  • idx_similar_units_fetched ON similar_units(fetched_at)
• Update save_finn_ad() to populate last_verified_at when saving
• Update service.py to use new TTL config constants (convert days→hours)
• Update analysis.py to use new TTL config constants

Phase 1 Complete ✅

• Long-lived caching enabled: 30-day structural data TTL
• Faster repeat searches: 6-hour search cache (was 1-hour)
• Infrastructure ready for lightweight price/status checks

---

Phase 2: Missing Tables + Stub Implementations ✅

• Create user_feedback table (finnkode PK, verdict, notes, created_at, updated_at)
• Create price_history table (append-only: finnkode, prices, sale_status, recorded_at)
• Create search_runs table (search_url, finnkodes JSON, created_at)
• Implement feedback.py functions (replace all TODOs with cache.py wrappers)
• Populate price_history on every fetch_ad_details() call
• Populate search_runs on every analyze_search() call
• New cache.py functions:
  • save_feedback / get_feedback / get_feedback_by_verdict / delete_feedback
  • save_price_history / get_price_history
  • save_search_run / get_latest_search_run
• All new functions tested and working

Phase 2 Complete ✅

• User feedback now persisted (was stubs)
• Price history tracked (enables price drop detection)
• Search runs tracked (enables diff detection)

---

Phase 3: Output Payload Cleanup ✅

• Added listing_description to analyze_ad output (for AI interpretation)
• Added price_history to analyze_ad output (last 20 records, slimmed to 5 for MCP response)
• Added cache_age to analyze_ad output (structural_days, price_hours) for transparency
• Updated _slim_listing() in mcp_server.py to include these fields
• Kept full score breakdown (all 12 dimensions + transit)
• Removed unit_images and unit_vector from MCP responses (never displayed)
• Removed internal eiendom timestamps from slim response
• Payload size improved: per-listing ~8KB (was ~40KB), search of 30 ads ~240KB (was ~215KB)

Phase 3 Complete ✅

• AI can now interpret listing_description for edge cases
• Price history visible for market analysis
• Cache transparency: users see when data was last checked
• Efficient payloads while keeping all decision-support data

---

Phase 4: Consolidate to 6 Tools + Batch

Remove tools (9 total):

• finn_build_unit_vector
• finn_decode_unit_vector
• finn_resolve_eiendom_unit
• finn_get_ad
• finn_get_eiendom_unit
• finn_get_similar_units
• finn_analyze_ad_against_comps
• finn_compare_ads
• finn_find_similar_to_liked_ad

Add batch support:

• Update finn_analyze_ad to accept string | string[]
• Add find_similar_to parameter to finn_get_shortlist
• Always include comps in analyze_ad

New tools (6 total):

1. finn_analyze_search
2. finn_analyze_ad (with batch)
3. finn_analyze_unit_images
4. finn_get_new_ads_since_last_run
5. finn_save_feedback
6. finn_get_shortlist (with find_similar_to)

---

Phase 5: Lazy Enrichment + Workflow

• analyze_search returns all scraped listings (no detail_limit)
• Listings without enrichment get score: null
• Background warm-up on save_feedback(liked)
• Re-score endpoint (from cached raw data only)

---

Completed Tasks

(None yet)

---

Blocked

(None yet)

---

Notes

• Source of truth: refactor.md in root
• All changes coordinate with cache.py, models.py, service.py, analysis.py, feedback.py
• Test coverage required for all phase changes