Promptfoo + Ollama AIRed Team Lab — Setup & Results (Step-by-Step)
Scope: Local LLM red teaming on Kali Linux (VMware), using Node.js (via NVM), Promptfoo CLI, and Ollama with llama2:7b-chat. This document explains what each screenshot shows, why the step matters, and what to verify before moving on.
Step 1: Update Kali packages and install prerequisites
Screenshot reference (Step 1)
What you are doing
⦁ Update package index and apply pending upgrades to keep the VM stable before installing developer tooling.
⦁ Install core prerequisites needed later (curl, ca-certificates, git, build-essential).
Commands used
sudo apt update
sudo apt -y full-upgrade
sudo apt install -y curl ca-certificates git build-essential
What to verify
⦁ No broken packages: the apt run finishes without errors (warnings are ok).
⦁ curl and git are installed: curl --version and git --version work.
Notes
⦁ If you see interactive package prompts (e.g., services restart), selecting the default safe option is usually fine for a lab VM.
⦁ Running upgrades first avoids dependency conflicts later when installing Node and CLI tooling.
Common issues & fixes
⦁ Issue: Repository signature / NO_PUBKEY errors during apt update
⦁ Fix: Refresh Kali archive keyring and re-run: sudo apt install -y kali-archive-keyring && sudo apt update.
Step 2: Install NVM and Node.js (Node 20) on Kali
Screenshot reference (Step 2)
What you are doing
⦁ Install NVM (Node Version Manager) to manage Node versions cleanly.
⦁ Install Node.js 20 and set it as the default version for the shell.
Commands used
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
source ~/.bashrc
nvm –version
nvm install 20
nvm use 20
What to verify
⦁ node -v prints v20.x and npm -v prints a version number.
⦁ Re-open a new terminal and confirm Node is still available (NVM loaded).
Notes
⦁ Using Node 20 is recommended for modern CLI tooling and better compatibility with Promptfoo.
⦁ If NVM is installed but commands do not work in a new terminal, ensure your shell is bash and that ~/.bashrc contains the NVM init lines.
Common issues & fixes
⦁ Issue: nvm: command not found after installation
⦁ Fix: Run source ~/.bashrc (or start a new terminal). If you use zsh, add NVM init lines to ~/.zshrc.
⦁ Issue: Slow downloads inside a VM
⦁ Fix: This is normal. VM disk + shared network makes installs slower; avoid running multiple heavy downloads in parallel.
Step 3: Update npm and confirm Promptfoo CLI runs
Screenshot reference (Step 3)
What you are doing
⦁ Confirm your Node and npm versions.
⦁ Update npm to the latest stable version.
⦁ Run Promptfoo via npx and confirm it prints a version.
Commands used
node -v
npm -v
npm install -g npm@latest
npm -v
npx promptfoo@latest –version
What to verify
⦁ npx promptfoo@latest --version prints a version (example shown: 0.120.23).
⦁ npm ‘deprecated’ warnings are acceptable; they do not mean install failed.
Notes
⦁ The first npx promptfoo@latest run is slower because npx downloads and caches dependencies under ~/.npm/_npx/.
⦁ For faster repeated usage you can optionally install globally: npm install -g promptfoo.
Common issues & fixes
⦁ Issue: npx appears to ‘hang’ on first run
⦁ Fix: Wait a bit; it is downloading dependencies. Verify activity with ps aux | grep node or ps aux | grep npm.
Step 4: Install Ollama and verify the service is running
Screenshot reference (Step 4)
What you are doing
⦁ Install Ollama (local LLM runtime) using the official install script.
⦁ Confirm Ollama version and systemd service status.
⦁ Note: CPU-only mode is expected if no GPU is detected.
Commands used
curl -fsSL https://ollama.com/install.sh | sh
ollama –version
systemctl status ollama –no-pager
What to verify
⦁ ollama --version prints a version (example shown: 0.15.6).
⦁ systemctl status ollama shows: Active: active (running).
⦁ Ollama API is reachable: curl -s http://localhost:11434/api/tags | head returns JSON.
Notes
⦁ The installer creates/starts ollama.service and binds the local API on 127.0.0.1:11434.
⦁ CPU-only mode is slower but fine for demos; model downloads and inference will take longer.
Common issues & fixes
⦁ Issue: Ollama service not running after install
⦁ Fix: Start it: sudo systemctl enable --now ollama then re-check status.
⦁ Issue: Download progress looks stuck
⦁ Fix: Large archives can pause on slow networks; check with ps aux | grep ollama and wait.
Step 5: Pull the target model (llama2:7b-chat) and verify it is available
Screenshot reference (Step 5)
What you are doing
⦁ Download the model weights to your local machine using ollama pull.
⦁ Confirm the model appears in ollama list before moving to Promptfoo.
Commands used
ollama pull llama2:7b-chat
ollama list
What to verify
⦁ ollama list shows llama2:7b-chat with a non-zero size (several GB).
⦁ Optional quick test: ollama run llama2:7b-chat "Say: ready" returns a response.
Notes
⦁ Model pulls are the biggest downloads in this lab. Keep enough free disk space (10–15 GB recommended per 7B model).
⦁ If you later add mistral:7b and llama3:8b, expect additional multi-GB downloads.
Common issues & fixes
⦁ Issue: Model pull fails due to low disk space
⦁ Fix: Free space: sudo apt clean, remove unused models (ollama rm <model>), or expand VM disk and resize partition.
⦁ Issue: Model pull is slow
⦁ Fix: Normal on shared networks/VM. Avoid parallel downloads.
Step 6: Create Promptfoo project folder and red-team config file
Screenshot reference (Step 6)
What you are doing
⦁ Create a dedicated working directory for the demo run.
⦁ Create promptfooconfig.yaml (or promptfooconfig.yml) with Ollama provider + prompt + redteam plugins.
Commands used
mkdir -p ~/ai-redteam-promptfoo
cd ~/ai-redteam-promptfoo
nano promptfooconfig.yaml
What to verify
⦁ File exists: ls -la promptfooconfig.yaml.
⦁ Config includes provider: ollama:chat:llama2:7b-chat and redteam plugins (example: prompt-extraction, system-prompt-override).
Notes
⦁ The ‘tests’ section is used by promptfoo eval. For promptfoo redteam run, Promptfoo generates its own adversarial tests based on plugins.
⦁ To make injection targeting explicit, include an input variable in your prompt like User message: {{query}}.
Common issues & fixes
⦁ Issue: YAML indentation errors
⦁ Fix: Use 2 spaces for indentation and avoid tabs. Run npx promptfoo@latest eval to validate parsing.
Step 7: Run the automated red-team scan and interpret the CLI output
Screenshot reference (Step 7)
What you are doing
⦁ Start the red-team scan; Promptfoo will generate attack cases and run them against the model.
⦁ Review the generated test summary and confirm results complete without errors.
Commands used
npx promptfoo@latest redteam run
What to verify
⦁ You see ‘Test Generation Summary’ with plugin counts (example: 16 total tests).
⦁ Scan completes with a result line similar to: Results: X passed, Y failed, Z errors.
⦁ A report is available via promptfoo redteam report (local web UI or HTML report).
Notes
⦁ A 0% ‘attack success rate’ means attacks did not cause a policy failure under the current plugins and target prompt — the scan still ran correctly.
⦁ If scans take a long time in a CPU-only VM, reduce concurrency and/or number of tests: set maxConcurrency: 1 and reduce numTests.
Common issues & fixes
⦁ Issue: Run takes too long on VM
⦁ Fix: Reduce load: maxConcurrency: 1 and numTests: 6–10. Also test one plugin at a time.
⦁ Issue: Email verification prompt appears
⦁ Fix: Promptfoo may request email verification for red-team scans. Complete it once, then future runs proceed normally.
Step 8: View results in the Promptfoo UI and review pass/fail evidence
Screenshot reference (Step 8)
What you are doing
⦁ Open the Promptfoo results UI (served locally) and drill into the run.
⦁ Review per-plugin results and individual prompt/output evidence.
Commands used
npx promptfoo@latest redteam report
npx promptfoo@latest view
xdg-open .promptfoo/redteam/report.html
What to verify
⦁ The UI shows the run name, target provider, number of tests, and ‘Attack Success Rate’.
⦁ You can open an individual test to see the prompt, model output, and evaluation status.
⦁ Export option works if you want CSV/JSON for documentation.
Notes
⦁ Attack Success Rate is the percentage of tests that produced a vulnerability finding (fail). If it shows 0%, it means no failures were detected for that run.
⦁ For a stronger demo, compare multiple models (llama2 vs mistral vs llama3) and/or add plugins like indirect prompt injection where appropriate.
Common issues & fixes
⦁ Issue: Browser does not open automatically
⦁ Fix: Copy the localhost URL printed by Promptfoo or run xdg-open on the HTML report path.
⦁ Issue: UI loads but shows no rows
⦁ Fix: Ensure you are looking at the correct run/eval ID. Re-run redteam report to refresh the report index.
Next steps for better leraning:
⦁ Run the same scan against multiple models (mistral:7b, llama3:8b) and compare attack success rates.
⦁ Add explicit injection variable {{query}} in your prompt to make the attack surface clearer.
⦁ For faster runs on CPU, set maxConcurrency: 1 and start with numTests: 6.
