# Internal Link Recommender (internal-link-recommender)

Paste an article and point to your sitemap: this one-page PHP app ranks your site’s pages against the article and suggests internal links with anchor text. Outcome: faster on-page SEO with high-quality, deduplicated link suggestions you can export.

## Features
- Inputs
  - Article (textarea, up to 20k chars)
  - Sitemap URL (supports sitemap.xml and sitemap index)
  - Min score (0–1), Max suggestions (1–100)
  - Anchor style: natural, partial, exact
  - Exclude paths (comma-separated patterns)
- Crawling and content
  - Parses sitemap/index and respects robots.txt
  - Fetches titles + snippets, caches page data for 24h
  - Limits to 200 URLs per run; 4 concurrent fetches with timeouts
  - Canonical dedupe (prefers rel=canonical where present)
- Relevance + anchors
  - AI mode (OpenAI) ranks relevance and generates anchors
  - Local fallback mode without AI (keyword similarity + deterministic anchors)
- Results
  - Suggestions list: url, title, anchor, score (0–1)
  - Stats: scanned_urls, candidates, suggestions, elapsed_ms
  - History of recent runs
  - Export as CSV and JSON
- Safety and fairness
  - Per-IP rate limit: 10 requests/min
  - Freemium quotas: 5 free runs/day, 50 pro runs/day
- Security
  - Content Security Policy (CSP) headers
  - CSRF tokens on all POSTs
  - Prepared statements (PDO) and output escaping
- UI
  - Light theme, blue accent, split layout (inputs left, results right)
- Optional
  - Nightly cron to refresh cached page data

## Requirements
- PHP 8.2+
  - Extensions: PDO MySQL, cURL, DOM, SimpleXML, JSON, mbstring, OpenSSL
- MySQL 8.x
- OpenAI API key optional (only required for AI mode)

## Quick start
1) Database
- Create a database and user:
```sql
CREATE DATABASE ilr CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
CREATE USER 'ilr_user'@'%' IDENTIFIED BY 'strong-password';
GRANT ALL PRIVILEGES ON ilr.* TO 'ilr_user'@'%';
FLUSH PRIVILEGES;
```
- Tables used: runs, rate_limits, pages, suggestions. The app can auto-create tables on first request if the DB user has CREATE TABLE privileges.

2) Configure environment
- Either export environment variables in your server or create a .env file alongside index.php.
- Minimal example:
```bash
APP_ENV=production
APP_URL=https://ilr.example.com
APP_SECRET=$(openssl rand -base64 32)

APP_DB_HOST=127.0.0.1
APP_DB_PORT=3306
APP_DB_NAME=ilr
APP_DB_USER=ilr_user
APP_DB_PASS=strong-password

# Optional tuning
APP_RATE_LIMIT_PER_MINUTE=10
APP_FREE_QUOTA=5
APP_PRO_QUOTA=50
APP_CACHE_TTL_SECONDS=86400
APP_MAX_SITEMAP_URLS=200
APP_CONCURRENCY=4

# AI (optional)
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o-mini
OPENAI_TEMPERATURE=0.2
OPENAI_TIMEOUT=10
# OPENAI_BASE_URL=https://api.openai.com/v1  # or your provider
```

3) Deploy
- Place index.php on a PHP 8.2+ web server with MySQL access.
- For local testing:
```bash
php -S 127.0.0.1:8000
```
- Visit http://127.0.0.1:8000 and run your first test.

4) Cron (optional)
- Nightly cache refresh to keep titles/snippets warm:
```bash
# Every night at 03:00, refresh cache (example query param and secret)
0 3 * * * curl -fsS "https://ilr.example.com/?cron=refresh-cache&key=YOUR_CRON_SECRET" >/dev/null
```
- Set APP_CRON_SECRET and ensure your server blocks this endpoint without the correct key.

## Configuration / environment variables
| Name | Required | Default | Example | Purpose |
|---|---|---:|---|---|
| APP_ENV | no | production | production | Environment mode (production/development) |
| APP_URL | no | — | https://ilr.example.com | Used in links and exports |
| APP_SECRET | yes | — | base64… | Secret for CSRF tokens/HMAC |
| APP_RATE_LIMIT_PER_MINUTE | no | 10 | 10 | Global per-IP throttle |
| APP_FREE_QUOTA | no | 5 | 5 | Daily free run quota |
| APP_PRO_QUOTA | no | 50 | 50 | Daily pro run quota |
| APP_CACHE_TTL_SECONDS | no | 86400 | 86400 | Page-data cache TTL (24h) |
| APP_MAX_SITEMAP_URLS | no | 200 | 200 | Max URLs scanned per run |
| APP_CONCURRENCY | no | 4 | 4 | Concurrent fetches for page data |
| APP_USER_IP_HEADER | no | — | X-Forwarded-For | Trust proxy header for rate limits |
| APP_CRON_SECRET | no | — | random-long-string | Protects cron endpoint |
| APP_DB_HOST | yes | 127.0.0.1 | 127.0.0.1 | MySQL host |
| APP_DB_PORT | yes | 3306 | 3306 | MySQL port |
| APP_DB_NAME | yes | — | ilr | MySQL database |
| APP_DB_USER | yes | — | ilr_user | MySQL user |
| APP_DB_PASS | yes | — | strong-password | MySQL password |
| OPENAI_API_KEY | no | — | sk-… | Enables AI mode if set |
| OPENAI_MODEL | no | gpt-4o-mini | gpt-4o-mini | Model for ranking/anchors |
| OPENAI_TEMPERATURE | no | 0.2 | 0.2 | Sampling temperature |
| OPENAI_TIMEOUT | no | 10 | 10 | HTTP timeout (seconds) |
| OPENAI_BASE_URL | no | https://api.openai.com/v1 | Custom base for compatible providers |

## How it works (high level)
- Fetch sitemap or sitemap index
  - Resolve and load up to 200 URLs, respecting robots.txt, with 4 concurrent fetches and sane timeouts
  - Extract title and a short snippet (meta description or first content text); cache for 24h
  - Canonicalize URLs and deduplicate by rel=canonical
- Score and select
  - AI mode: send compact page summaries to the model to score against the article; generate anchors according to the selected style
  - Fallback mode: compute keyword similarity (e.g., tf-idf/cosine) and generate deterministic anchors from titles or URL slugs
- Filter and output
  - Apply min-score, suggestion limit, exclude-path rules
  - Return suggestions with stats; persist run in history

## Security measures
- Content Security Policy (configure at the web server or via headers)
  - Recommended: default-src 'self'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self' https://api.openai.com
  - Adjust connect-src if using a custom OpenAI-compatible endpoint
- CSRF protection
  - Per-session token (HMAC with APP_SECRET), required on all form POSTs and validated server-side
- Prepared statements and escaping
  - All DB reads/writes use PDO prepared statements; all HTML output escaped; JSON output via safe encoding
- Rate limiting and quotas
  - 10 requests/min per IP (configurable), with free/pro daily quotas; supports trusted proxy header via APP_USER_IP_HEADER
- Input validation
  - Strict limits on text lengths, numeric bounds, allowed URL schemes/hosts; normalization and canonical handling
- Robots and fetch safety
  - Respects robots.txt disallows, verifies TLS, and uses reasonable timeouts/retries

## Exports
- CSV
  - Columns: url, title, anchor, score
  - Generated per run; downloadable via the UI (or query: ?export=csv&run_id=…)
- JSON
  - Keys: suggestions (array of objects), stats (object with scanned_urls, candidates, suggestions, elapsed_ms)
  - Download via the UI (or query: ?export=json&run_id=…)

## Optional AI behavior and fallback mode
- AI mode (OPENAI_API_KEY set)
  - Purpose: rank sitemap pages vs the pasted article and generate anchor text
  - Model configured by OPENAI_MODEL; temperature defaults to 0.2
  - Anchor style
    - natural: phrase-like anchors blended to read naturally
    - partial: includes a key term from the target page
    - exact: matches the target page’s primary keyword/title where feasible
- Fallback mode (no OPENAI_API_KEY)
  - Ranks candidates via keyword overlap/tf-idf similarity between the article and page snippets
  - Anchors derived from page titles or URL slugs, trimmed to a sensible length, respecting the selected style as best effort

## UI notes
- Split layout: inputs on the left, results and stats on the right
- Light theme with blue accent
- History view to revisit recent runs and re-export

## Acceptance checklist
- [ ] Server runs PHP 8.2+ with required extensions
- [ ] MySQL 8 database created and reachable from the app host
- [ ] APP_SECRET set to a strong random value
- [ ] Database credentials configured (APP_DB_*)
- [ ] Rate limit and quotas configured to your needs
- [ ] OPENAI_* set (or intentionally omitted to use fallback mode)
- [ ] CSP header enabled and adjusted to your deployment
- [ ] First run creates tables (or tables pre-created) without errors
- [ ] Sitemap fetch respects robots.txt; caps at 200 URLs; concurrent fetches set to 4
- [ ] Suggestions appear with scores; min-score, limit, exclude paths honored
- [ ] CSV and JSON exports download and contain expected fields
- [ ] History shows recent runs; exports work for past runs
- [ ] Optional cron set with APP_CRON_SECRET and runs successfully

## License
MIT (see LICENSE file)