Wearing the Agent: Engineering a Family-and-Friends Personal Agent, from Group Chats to Glasses

What changes when an agent built to help one person must serve a whole group? The familiar architecture still applies: a model supplies reasoning, an orchestration harness manages its work, and tools let it act. That combination can already do useful things. But most assistants—including coding assistants deployed inside enterprises—still organize their work around one user.

A shared assistant changes that assumption. It may serve friends, family members, or colleagues whose interests and permissions overlap without being identical. Put that assistant into glasses, leave it available throughout the day, and the question changes again: its audience is no longer necessarily everyone within earshot. The engineering problem expands from completing a user’s task to operating appropriately among several people.

Sai Krishna Rallabandi reports deploying Judith among friends and family for eight months. One conference example makes the shared-agent problem concrete: he asks a group chat how to get to the venue, and Judith responds by direct message to preserve privacy. The destination of the answer is part of the answer’s correctness. In another exchange, he and his wife are organizing an event; Judith works across calendars to find a slot available to everyone.

The same distinction appears outside chat. While Rallabandi is driving, the glasses agent offers proactive advice related to insufficient sleep and decision-making. Recognizing the situation is only part of the behavior he values. The agent speaks through his glasses rather than the car’s audio system, keeping the advice private instead of broadcasting it to the vehicle.

Longer-lived interactions add another requirement. A friends’ discussion keeps evolving, so Judith curates relevant context and filters out material no longer needed. A learning agent used by his daughter supports capitals, numbers, and retention while keeping the parents informed about her progress. These examples span private delivery, coordination, selective memory, and different responsibilities toward different participants.

These are deployment examples from Rallabandi’s own system. He also points to controlled experiments with group agents elsewhere, without developing their results. The recurring design issue is that a group introduces relationships and disclosure boundaries that a single-user conversation can leave implicit.

The harness now has three connected responsibilities: guard what enters and what actions follow, organize memory, and route information to the appropriate recipient. Routing means choosing who receives information, not selecting a cheaper model.

Judith’s security layer, Jataayu, starts from a distinction between a language model and an acting system. A model produces text that can be inspected. An agent also reads resources, invokes tools, and changes things. Each capability enlarges the security boundary, and a group introduces more paths by which content reaches it.

An always-available agent may consume webpages, screenshots uploaded to a group, GitHub issues, and incoming email—including promotional messages. A trusted participant can upload an untrusted screenshot. Membership in the group therefore does not confer authority on every instruction embedded in the group’s content. Blocking everything suspicious at ingestion would also block much of the material the agent needs to do its job.

Runtime Skill Audit motivates looking beyond static inspection, while When Safe Skills Collide motivates examining combinations of capabilities. Rallabandi illustrates the problem with an OCR skill and a reporting skill. Both can look acceptable in isolation. An attacker then controls content that OCR reads; when the reporting stage acts on the extracted material, it sends personally identifiable information to a third party along with the intended report. The dangerous behavior emerges across the workflow.

Rallabandi cites an approximately 90% attack figure, but does not identify its denominator or measured property. It should not be read as a general attack-prevalence estimate: Runtime Skill Audit’s headline accuracy result concerns classification of skills, a different quantity. The useful design implication here is the runtime failure mode, not an unspecified probability attached to it.

The action boundary is the enforcement point. Instead of trying to prevent the agent from ever encountering hostile material, inspect what it is about to do: access shell variables, read configuration secrets, or export information. Deterministic checks can make this decision without another generative-model call.

Rallabandi proposes a simple three-class baseline using traditional NLP or regular expressions. He explicitly describes benign actions that can proceed and actions that need approval, without naming every class. The central ordering constraint is more important than the labels: a sensitive operation must remain pending until its check has passed. A small TypeScript example makes that ordering explicit for configuration access:

type ConfigRead = {
  kind: "read-config";
  key: string;
};

type Decision = "allow" | "approval-required";

function inspect(action: ConfigRead): Decision {
  return /(?:secret|token|password)/i.test(action.key)
    ? "approval-required"
    : "allow";
}

async function guardedRead(
  action: ConfigRead,
  approve: (action: ConfigRead) => Promise<boolean>,
  readConfig: (key: string) => Promise<string>
): Promise<string> {
  const decision = inspect(action);
  if (decision === "approval-required" && !(await approve(action))) {
    throw new Error("Configuration read not approved");
  }
  return readConfig(action.key);
}

For a proposed read of API_TOKEN, the regular expression sends the request through approval before readConfig runs. This illustrates the gate’s placement; a name-matching rule only covers the patterns it recognizes.

Judith also has a component aimed at training a small language model through supervised fine-tuning with LoRA. Its job is to distinguish hostile content in the data channel from legitimate instructions. Consider a request to summarize a webpage: the page is evidence to summarize, not a source of authority over the assistant. The desired behavior is to suppress embedded attack instructions while preserving enough benign content to complete the summary.

Rallabandi reports evaluating the approach on InjecAgent. The displayed InjecAgent slide reports detection rates of 50% and 66%. His narration identifies the lower result as the naive baseline and describes improvement, but does not give the model, evaluation split, or detailed scoring setup. These are reported results for this presentation, not interchangeable with the original benchmark paper’s results.

The accompanying failure case is simple: intersperse dots through text, as in I.L.I.K, and static matching may miss a pattern it would recognize in ordinary spelling. A learned model might recognize the underlying instruction despite that surface change. Rallabandi presents that as a reason to pursue learned detection, not as a guarantee that it catches every obfuscation.

The learned filter adds a layer; it does not remove the need to check actions. The architecture still needs enforcement where the agent performs a task, with a detector that can be improved iteratively as new attack patterns appear.

Once information enters the system, the next question is what should persist. Embeddings or a FAISS index can provide a basic retrieval mechanism, but they do not decide what the group’s memory ought to contain. Memory curation shapes what the agent becomes over time. Storage, revision, and forgetting affect its future behavior as well as the amount of context it must process.

Take a group planning an outing. One person proposes a place and a day; another adds information; later messages modify the plan. Storing the entire exchange and repeatedly compacting it preserves a conversation-shaped record. Extracting atomic facts instead produces pieces that can be retrieved and revised independently: the proposed place, the planned day, and subsequent updates.

That changes what evaluation must measure. An autorater should assess the memory operation rather than merely whether a summary sounds plausible:

• Extraction: Did the system retain high-value facts that could matter later?
• Relevance: Are those facts useful to this conversation?
• Relationships: Does it preserve how facts relate hierarchically?
• Time: Does it recognize that some facts become important while others become obsolete?
• Retrieval: Does the current query receive the right facts without retrieving everything?

A graph is one suggested way to organize relationships for selective retrieval. It is a means of finding the right context, not a substitute for evaluating whether the extracted facts are useful.

As memory grows, a small model or relevance scorer can help decide what to retain. The important part is continual scoring: a fact’s value changes as a group’s plans and priorities change. Rallabandi introduces this idea under the spoken reference title “Learning What Not to Forget.” The conversational-memory mechanism described here is a scorer over stored context; it should not be confused with the similarly titled Memory Aware Synapses work on neural-network parameter importance.

Continually adapting relevance scores enable knowledge-based compaction: retain context because it remains useful, rather than simply compressing whatever has accumulated. The intended benefit is less context and lower token use. Serving imposes another constraint, however. Both local and cloud model execution use KV caches, and changes to the material supplied to a model can disrupt reuse. Memory ingestion therefore needs to account for caching as well as semantic relevance. The talk recommends cache-aware ingestion without specifying an invalidation algorithm.

A shared assistant is more than one model used by several accounts. It participates in a social setting where disclosure depends on context. A grocery list may be suitable for broad sharing; salary or health information may be appropriate for one recipient and inappropriate for another. The fact itself can remain unchanged while the room—and therefore its permissible audience—changes.

Rallabandi proposes a common memory layer with a separate LoRA adapter for each user, aiming to learn permission-sensitive behavior rather than encode it solely in application logic. This is a proposal for learned behavior, not a demonstrated access-control guarantee. He cites a work verbally as “The User as a N-gram,” but that incomplete reference does not establish that per-user LoRA adapters reliably enforce permissions.

Recipient selection is only one part of participation. Group agents can also speak too often when nobody has addressed them. Rallabandi describes an unfinished approach using a classifier—potentially RoBERTa—to determine the agent’s role and whether it is allowed to speak. That makes silence an explicit behavior to select, rather than an accidental absence of a generated answer.

The resulting harness must carry these decisions through the whole interaction: contextual security checks govern what the agent may act on, curated memory determines what remains available, and recipient routing controls where information goes. For a shared agent, completing the task includes delivering the result to the responsible people—and recognizing when it should say nothing.