ID-6 Add Automatic Goal Recommendation / Crawler 6.x-dev Port - #25149

ID-6 — Innovation Team
Available on demo3. (5.x-dev version)

Description

This is the 6.x-dev port of #24803.

Due to the long runtime of this Project and PR, the base branch has moved from 5.x-dev to 6.x-dev, making the reverse port necessary.
The changes should be identical up to before the last two commits, after that some necessary fixes were introduced, given the 6.x-dev changes.
Other possible changes or improvements given 6.x-dev have been deferred and will be solved in future PRs.

Changelog from #24803:

Adds automatic goal recommendations to the Goals management screen. Users can scan their website and create relevant goals with one click instead of configuring every goal from scratch.

Recommendations work without AI. AI-assisted recommendations are available when the bundled AIProviders plugin is active and configured.

How it works

• Site analysis: Matomo performs a bounded, same-origin crawl starting from the site’s homepage. It detects relevant pages, forms, downloads, contact links, outbound links, and other conversion signals.
• Safe crawling: Requests use Matomo’s SSRF-safe HTTP handling, including redirect validation. The crawler follows the effective URL after redirects. For this, Piwik\Http’s extended info now also returns the new effectiveUrl entry (documented in the developer changelog).
• Rule-based recommendations: A deterministic recommender converts the detected signals into goal suggestions.
• Optional AI ranking: Users can explicitly enable AI for richer, business-value-ranked suggestions. The AI toggle is off by default. If AI is unavailable or the request fails, Matomo falls back to deterministic recommendations and informs the user.
• Manual suggestions: Recommendations that cannot be created automatically can prefill the regular goal form.
• Persistence and deduplication: Results are stored per site and restored on page reload without repeating the crawl or AI request. Stable recommendation identifiers prevent duplicates and support individual dismissal. Suggestions already covered by an existing goal are filtered out.
• Concurrency protection: Per-user and per-site locks prevent overlapping recommendation scans.

Configuration

Managed environments can limit AI usage and the number of pages crawled:

[Goals]
; Maximum number of same-origin pages crawled per scan. Default: 50
recommendation_max_crawl_pages = 50

; Maximum number of AI-assisted scans per site and day.
; Default: 0 (unlimited). Example: allow 3 AI-assisted scans per day.
recommendation_ai_daily_scan_limit = 3

Managed environments can also replace the privacy note shown next to the AI toggle via the Goals.customizeRecommendationPrivacyNote event.

Checklist

• [✔] I have understood, reviewed, and tested all AI outputs before use
• [✔] All AI instructions respect security, IP, and privacy rules

Review

• Functional review done
• Potential edge cases thought about (behaviour of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• Security review done
• Wording review done
• Code review done
• Tests were added if useful/possible
• Reviewed for breaking changes
• Developer changelog updated if needed
• New documentation added or existing documentation updated

Replaces #24619 (closed automatically when the head branch was renamed from ID-35-goal-recommendation to ID-6-goal-recommendation).