The Hidden Risks of Your Self-Hosted AI Stack (And How I Locked Mine Down)

Your self-hosted AI stack is a juicy target, and 2026 has already handed us two reminders: a poisoned LiteLLM release that quietly harvested credentials, and tens of thousands of Ollama servers sitting wide open on the internet. The good news? You can audit your own setup and lock it down in about 20 minutes. Here's how I did mine.

The night the LiteLLM supply-chain news broke, I did the thing every self-hoster does. I told myself "I'm probably fine," and then immediately went to check anyway.

I run Open WebUI, LiteLLM, and Ollama in my home lab. It's a setup I love precisely because it's mine — no API bills, no rate limits, no sending my prompts to someone else's server. But "it's mine" and "it's safe" are two very different statements, and that night I found out my pinned LiteLLM version was a lot closer to the danger zone than I was comfortable with.

So let's talk about the two security stories that should be on every home labber's radar right now — and then the short, boring checklist that actually fixes them.

The wake-up call: LiteLLM got poisoned

On March 24, 2026, two releases of LiteLLM — versions 1.82.7 and 1.82.8 — were published to PyPI with malicious code baked in. They were live for roughly 40 minutes before PyPI quarantined them.

Forty minutes doesn't sound like much. But LiteLLM pulls something like 95 million downloads a month, and automated CI/CD pipelines don't sleep. Plenty of machines grabbed the bad version in that window.

Here's the part that keeps me up at night: this wasn't someone guessing a weak password. The attackers (who called themselves "TeamPCP") got in by first compromising Trivy — an open-source security scanner that LiteLLM used in its own CI/CD pipeline. The tool meant to catch bad code became the way in to publish it. That's the supply chain biting from a direction most of us never think to watch.

And the payload was nasty. Once installed, it ran in three stages:

• A credential harvester that swept up more than 50 categories of secrets — environment variables, SSH keys, cloud credentials, Kubernetes data, Docker configs, database passwords, even CI/CD secrets.
• A Kubernetes lateral-movement toolkit built to spread from one compromised box across an entire cluster.
• A persistent backdoor for ongoing remote code execution.

The sneaky bit: in 1.82.7 the code hid inside proxy_server.py and only fired when something imported the LiteLLM proxy. In 1.82.8 they added a .pth file to your Python site-packages — and that runs on every Python interpreter startup, no import required. You didn't even have to use LiteLLM for it to execute.

If your LiteLLM box has your OpenAI key, your Anthropic key, your cloud creds, or your SSH keys sitting in environment variables — and let's be honest, mine did — then a 40-minute window was plenty.

The official LiteLLM security advisory has the full timeline, and the Datadog Security Labs writeup traces the whole campaign if you want the gory details.

The other problem: your Ollama is probably listening

The LiteLLM incident was a sharp, dramatic event. This next one is the slow, quiet kind — the kind that's been true the whole time and most of us just never checked.

Ollama has no authentication. None, by default. There's no login, no API key, no token. Out of the box it binds to localhost (127.0.0.1), which is fine. The problem starts when you want to reach it from another machine — your desktop, your phone, a second server — and the quickest fix you'll find on every forum is to set OLLAMA_HOST=0.0.0.0.

That one change tells Ollama to listen on every network interface. If your router forwards a port, or your VPS has a public IP, you've just put an unauthenticated AI server on the open internet.

How common is this? Cisco Talos went looking with Shodan and found over 1,100 exposed Ollama servers, with roughly 20% actively serving models with zero authentication. A broader sweep by SentinelLABS and Censys put the number at 175,000 exposed Ollama hosts across 130 countries, with more than 12,000 fully open — anyone could list the models, run inference on someone else's GPU, and in some cases pull down proprietary fine-tuned weights.

Think about what that means for a home labber. Best case, a stranger is burning your electricity and GPU running their own prompts. Worse case, they're poking at every other service on that same machine, now that they have a foothold. Cisco's own takeaway was blunt: this is "widespread neglect of fundamental security practices." You can read their Shodan case study for the breakdown.

Audit your own stack in 15 minutes

Enough doom. Let's actually look at your setup. Grab a terminal on whatever box runs your AI tools.

1. Check which LiteLLM you're running.

# If you installed it with pip
pip show litellm
pip freeze | grep litellm

If you somehow landed on 1.82.7 or 1.82.8, treat every credential that ever touched that machine as compromised — rotate them, all of them. If you're on a clean version, good, but keep reading.

2. See what's actually listening. This is the big one. Find anything bound to 0.0.0.0:

# Show listening ports and what's behind them
ss -tulpn

# Or, if you prefer lsof
sudo lsof -i -P -n | grep LISTEN

Scan for Ollama's port (11434) and your Open WebUI port. If you see 0.0.0.0 in front of them rather than 127.0.0.1, they're reachable from your whole network — and possibly beyond.

3. Check your Docker port mappings. Containers love to publish ports, and it's easy to forget which.

docker ps

Look at the PORTS column. A mapping like 0.0.0.0:11434->11434/tcp means that container is exposed on every interface. 127.0.0.1:11434->11434/tcp means it's local-only — that's what you want for anything without auth.

4. Find where your keys live. Grep your compose files and .env files for API keys, and ask yourself an honest question: is the same key reused across services? Because if one box leaks, you don't want that to mean all of them did.

The fixes (and what I changed)

None of this is exotic. It's the security hygiene we all nod along to and then skip. Here's what actually moved the needle for me:

Pin your versions, and don't blind-upgrade. I now pin LiteLLM (and the rest of my AI stack) to specific, known-good versions instead of pulling latest. When a new release drops, I give it a few days and a glance at the release notes before I touch it. A poisoned package can't hurt you if you're not racing to install it the minute it appears.

Keep Ollama on localhost. I set it back to local-only and stopped exposing the port:

OLLAMA_HOST=127.0.0.1

"But I need to reach it from my other devices." Right — so did I. The answer isn't 0.0.0.0, it's reaching it through something that handles identity for you.

Put everything behind a reverse proxy or Tailscale. Anything without its own auth — Ollama, and in a lot of setups Open WebUI too — should sit behind a reverse proxy that requires a login, or only be reachable over a private network. I use Tailscale for a lot of my services, so my AI services are reachable from my laptop and phone but invisible to the public internet. No ports forwarded, no exposure. If you want something purpose-built for AI traffic specifically, gateways like Tailscale's Aperture centralize keys and add audit logs on top.

Rotate anything that was ever exposed. The boring one, and the one people skip. If a key sat in an environment variable on a box that ran a bad package — or on a server that was listening on 0.0.0.0 — it's burned. Rotate it and move on. It takes ten minutes and it's cheaper than finding out the hard way.

My before-and-after wasn't dramatic to look at: same services, same hardware, same workflow. The difference is that Ollama is back on localhost, my whole AI stack lives behind Tailscale, my versions are pinned, and the keys that were sitting in the blast radius have been rotated. Twenty minutes, start to finish.

Final thoughts

The whole appeal of self-hosting AI is control. You run the models, you keep the data, you skip the middleman. But control cuts both ways — when there's no vendor babysitting your security, the babysitting is on you.

Neither of these stories is a reason to stop self-hosting. They're a reason to do it like you mean it. If you take one habit away from this, make it this: pin what you install, and don't expose what doesn't have a password. Do that, and you've sidestepped the vast majority of how these things actually go wrong.

I almost didn't check my own setup that night. I'm very glad I did.