What Windows AI agents are and why local LLMs help
Windows AI agents are background assistants that can open apps, move files, and act on your desktop using large language models, so securing them means limiting what they can touch, where they can send data, and how long they keep access to your files.
On modern Windows 11 systems, agent features like Agent Workspace and Copilot Actions run inside their own contained desktop sessions under a standard Windows account, without admin rights and with logging and access controls your IT team already understands. That is the good news for Windows AI agents security: Microsoft split the runtime from your main account, ships it off by default, and signs agents so bad ones can be revoked. The bad news is that when Copilot Actions wants to work with your documents, it requests broad read and write access to six known folders: Documents, Downloads, Desktop, Music, Pictures, and Videos, which is far more than many people would give a human helper. If you care about privacy or sensitive work, this is where local LLM setup and scoped permissions become worth the effort.
Using Intelligent Terminal with a local LLM instead of Copilot
Intelligent Terminal is an experimental fork of Windows Terminal that adds an AI agent pane to your shell while staying installed as a separate app. By default it expects to talk to GitHub Copilot and can even offer to install it through WinGet, but the underlying design is built on the Agent Client Protocol (ACP), which means it does not care which model or provider is on the other end as long as it can receive shell context and send back responses. In other words, if you want a local model running on your own GPU instead of a cloud agent, Intelligent Terminal treats that as a first-class option and can work offline as long as the model is reachable.
One practical way to set up a local LLM is through Pi, an open-source coding agent that can work with many providers, including local models via llama.cpp or Ollama. In one real-world setup, the user installed the pi-acp extension, pointed Intelligent Terminal’s agent dropdown at it, and set Qwen 3.6 27B running on a home server as the default model in a configuration file. According to that user, “I got it running with Qwen 3.6 27B, and it took about five minutes.” The end result was a Microsoft terminal talking through an ACP bridge to an open-source agent backed by a model from Alibaba, working side by side with the normal terminal window. That kind of local LLM setup means shell context stays on your own hardware rather than going to a cloud model.
- Install Intelligent Terminal as a separate app so your main Windows Terminal remains unchanged and agent-free if you prefer.
- Add or configure a compatible AI agent that speaks ACP, such as Pi with the pi-acp extension, so Intelligent Terminal can talk to it through standard input and output.
- Point the agent configuration at a local LLM provider like llama.cpp or Ollama and set a default model such as Qwen 3.6 27B running on your own server or GPU.
- Confirm that Intelligent Terminal’s agent dropdown selects your local agent instead of Copilot, then run a few commands to verify that error messages and shell context are sent to the local model.
- Keep this agent scoped to terminal tasks and avoid connecting it directly to broad file locations or cloud tools, so its access stays close to the shell environment you expect.
The main advantage of this approach is that Intelligent Terminal lets you decide where the model runs and who sees your shell data, instead of hardwiring you into Copilot. You still get the AI pane, error detection, and session tracking, but the brain behind it can live entirely on your network. Until built-in agents earn more trust, some power users are choosing to keep Copilot off and rely on local LLMs they know are not sending data to external servers.
Why default file access from Windows AI agents is risky
Once you move beyond the terminal and into full desktop automation, the problem shifts from where the AI model runs to what it can touch on your file system. Windows 11’s Agent Workspace can open apps, click, type, scroll, move files between folders, and extract tables out of PDFs while you keep working, which is exactly what many people want from AI automation. It runs in its own standard account, so it is isolated in the sense that the agent session is separate from your user and not running as admin. However, isolated and temporary are not the same, and a contained account can still hold onto broad privileges if you hand them over once.
The critical detail is that when Copilot Actions needs to touch your files, it requests read and write access to six known folders: Documents, Downloads, Desktop, Music, Pictures, and Videos. These are the locations where many people keep tax PDFs, work-in-progress documents, SSH configurations, and screenshots that might include API keys and other secrets. “I wouldn’t give another person access to those folders, so why should I give an agent the entirety of my digital footprint?” the tester asked. With that much access, a compromised or misused agent could be told to search for banking details and send them to an external server, and broad permission to your Documents folder would make that scenario far easier than many people realize. That is why Windows AI agents security is less about intent and more about the permission surface you expose.
Scoped permissions: limit what AI agents can touch
One practical answer is to move from standing access to scoped permissions: give an AI agent access to a narrow slice of files for one task, then take it away when the task ends. The same user who experiments with MCP connectors in a home lab applies this discipline everywhere: every tool has an explicit permission boundary it cannot cross, and nothing keeps permanent rights to critical systems like Proxmox clusters or DNS settings. Instead, they spin up new environments through tools such as Terraform and Ansible, connect those virtual machines to a local LLM server, and give each VM a copy of only the files it needs for one job. Once the task completes, the environment can be torn down or disconnected, which prevents agents from lingering with access long after their work is finished.
Applied to Windows AI agents security, the principle is clear: scope access to the task at hand, isolate the runtime, keep private data on-device, and log everything. That means resisting the temptation to give Copilot Actions permanent read and write rights to all your known folders, and instead creating smaller working directories or separate accounts for AI-driven file work. You can still enjoy agentic file tasks without giving up long-standing control of your digital life: one user put it bluntly, they have not given up on agentic file work, but they refuse to grant those agents lasting access to their file system. Until broader trust is earned and more granular controls arrive, many users will prefer to keep built-in agents off and run local LLMs that stay on their own servers and stay within clearly marked boundaries.
Takeaway: is the extra setup worth it?
Setting up Intelligent Terminal with a local LLM and adopting scoped permissions for Windows 11 AI features is less about paranoia and more about matching access to your comfort level. For many users, it is enough to know that Intelligent Terminal will speak to any ACP-compatible agent, whether that is Copilot or a local Qwen model, and that a basic local LLM setup can be running in minutes. For others, the idea of an agent with broad standing access to Documents, Downloads, Desktop, Music, Pictures, and Videos is a deal-breaker, no matter how carefully Microsoft isolates Agent Workspace. If you fall in the second camp, the extra work of local models and scoped permissions is worth it, because your assistants become guests in your system instead of permanent roommates. Watch your file permissions, keep AI sessions contained, and you can enjoy helpful automation without handing over your entire digital footprint.
