AI Security and Safety

This page provides a practical starting point for Genesis Mission RFA teams that use AI models, retrieval-augmented generation (RAG), coding assistants, or tool-using agents.

Companion pages:

• Data Management

• AI and Agents

Start here: eight questions before connecting data or tools

1. What data will the system receive? Identify the owner, sensitivity, agreements, restrictions, and approved uses.

2. Is the exact service approved for that data? Confirm the model endpoint, tenant or workspace, retention, deletion, training use, and access terms.

3. What can the system see? Include prompts, uploaded files, retrieved documents, embeddings, memory, logs, tool outputs, local files, and environment information.

4. What can the system write? List read, write, delete, code execution, network access, API calls, job submission, external communication, and physical actions.

5. What could go wrong scientifically? Consider incorrect facts, citations, code, units, parameters, assumptions, uncertainty, or experimental recommendations.

6. Which actions require a person to approve them? Name the reviewer and define the approval point before testing.

7. How will you stop and recover the workflow? Set time, cost, token, retry, job, and action limits. Provide cancellation and rollback.

8. Who will you contact if something goes wrong? Use your institution's required reporting path and the Genesis Mission contact listed below.

Do not proceed until the team can answer these questions.

Start at the lowest capability level that can accomplish the task. Adding tools or autonomy adds risk.

Minimum rules for every team

1. Confirm the data and service boundaries

Do not enter classified, controlled unclassified, export-controlled, proprietary, privacy-sensitive, security-sensitive, unpublished partner, or otherwise restricted information unless the data owner and the responsible institution or program have approved the exact service and workflow.

Confirm what the service retains, who can access it, whether submitted content may be used for training or service improvement, where it is processed, and how it is deleted. Use synthetic, public, or minimized data for early tests.

2. Name accountable people

Record the scientific owner, technical owner, data owner, and person responsible for approving consequential actions. An AI system cannot own a decision or accept risk.

3. Use the least privilege

Begin read-only. Give the system only the files, tools, data, network destinations, and operations needed for the current task. Assume a coding agent will read or write to any directory or file it can find.

4. Keep authorization outside the model

A prompt such as “never delete files” is not an access control. The application, operating system, scheduler, database, API gateway, or policy service must deny actions that are not allowed.

Require human approval before:

• Writing to shared or authoritative data

• Deleting or overwriting information

• Installing software or changing an environment

• Submitting large or expensive computing jobs

• Sending email, posting, publishing, or contacting an external service

• Changing permissions or credentials

• Controlling equipment or changing an operating setpoint

• Using an output for a safety-relevant or scientifically consequential decision

Do not approve a vague future action.

5. Protect credentials

Never place secrets (passwords, keys, tokens, certificates, or connection strings) in prompts, notebooks, screenshots, agent instructions, committed configuration files, test fixtures, or logs.

Use an institution-approved secret service for shared or deployed systems. For local development, use approved secret management and ensure generated code or the agent sandbox cannot read the parent process environment, home directory, credential files, or cloud metadata service.

Use separate, short-lived, scope-limited credentials where supported. Know how to revoke them.

6. Verify scientific outputs

Treat model responses, citations, code, equations, parameters, interpretations, and conclusions as unverified.

For consequential work:

• Compare against a non-AI baseline or trusted reference case.

• Check citations for correctness

• Test generated code .

• Check units, ranges, conservation laws, invariants, boundary conditions, and domain constraints.

• Record uncertainty and cases where the system should abstain.

• Require review by a qualified team member.

Agreement between two models does not establish correctness. Models can share the same errors.

7. Preserve useful evidence without leaking sensitive data

Record enough information to reproduce and investigate the workflow:

• Model provider, endpoint, model ID, and date

• Framework, package, and tool versions

• Prompts or versioned instructions

• Tool definitions and permission policy

• Data source, version, and provenance

• Actions, approvals, denials, errors, and timestamps

• Resource use and cost

• Tests, outputs, and human review decisions

Do not log secrets or unrestricted copies of sensitive prompts and data. Apply access controls, redaction, and retention rules to logs.

8. Set limits and a (process) kill path

Set maximum iterations, tool calls, recursion depth, retries, tokens, cost, runtime, submitted jobs, files changed, and external actions. Stop when a limit is reached rather than asking the model whether it should continue.

9. Test before shared or operational use

At minimum, test that the system:

• Refuses or blocks access to another project or user's data

• Cannot call an unauthorized tool

• Cannot turn a read operation into a write

• Requires valid approval for high-impact actions

• Ignores instructions embedded in retrieved content or tool output

• Stops at resource, cost, time, and retry limits

• Does not expose secrets in output or logs

• Produces acceptable results on representative scientific tasks

• Fails safely when a model, tool, network service, or data source is unavailable

Repeat tests after material changes to the model, prompts, tools, retrieval corpus, memory, permissions, or framework.

What to do when something goes wrong

Examples include:

• A secret appears in a prompt, output, repository, screenshot, or log

• The system accesses another user or project's data

• An agent takes or attempts an unapproved action

• Retrieved content or tool output changes the agent's goal

• A job, loop, API call, or cost runs beyond its limit

• A model or tool sends data to an unexpected destination

• An incorrect AI result influences a consequential decision

• Equipment behaves unexpectedly after an AI recommendation or action

Response steps:

1. Stop or disable the workflow. Use the external kill path; do not rely on the model or agent to stop itself.

2. Prevent additional actions. Isolate the service, tool, credential, job, or equipment connection as appropriate.

3. Revoke exposed credentials. Rotate keys and tokens using the approved process.

4. Preserve evidence. Keep relevant prompts, logs, tool calls, files, model and framework versions, approvals, and timestamps. Do not destroy evidence while cleaning up.

5. Report the event. Contact your institution's required security or safety function and the Genesis Mission contact below (i.e. submit an issue through the ticketing system).

6. Do not resume until the cause, impact, corrective action, and required approvals are reviewed.

Do not include secrets or unnecessarily sensitive data in a help ticket or chat message.

Security and evaluation tools

Tools support review; they do not certify a system or replace data approval, access control, scientific testing, or human judgment.

Promptfoo

Promptfoo is an open-source evaluation and red-team framework for model, RAG, agent, and API targets. Use only approved test data and endpoints. Review generated tests before execution, especially when the target can call tools or external services.

- Public documentation: https://www.promptfoo.dev/docs/red-team/

What not to rely on

Do not rely on any of the following as the only protection:

• A system prompt that tells the model to behave

• XML tags, delimiters, or keyword filters

• A model's confidence statement

• Agreement between multiple models

• A red-team scan with no human review

• Logs that no one monitors

• A human approval button that does not show the exact action and parameters

Reference resources

• NIST AI Risk Management Framework

• NIST AI RMF: Generative AI Profile

• OWASP Top 10 for Large Language Model Applications

• OWASP AI Agent Security Cheat Sheet

• OWASP RAG Security Cheat Sheet

• MITRE ATLAS

• CISA and partner Guidelines for Secure AI System Development

• NSA and partner guidance for deploying AI systems securely