web
You’re offline. This is a read only version of the page.
close
Skip to main content
Community site session details

Community site session details

Session Id :
Power Platform Community / Blogs / Community Help & Announcements / Concierge Agent for Community

Concierge Agent for Community

Views (0)
Bret Fraser Profile Picture Bret Fraser Community Manager
Responsible AI FAQ – Concierge Agent for Community
 
This FAQ provides answers to frequently asked questions about the AI technology used in the Concierge Agent for Community, available in the Dynamics 365 and Power Platform Communities.  
 
What is the Concierge Agent for Community in the Dynamics 365 and Power Platform Community?
The Concierge Agent for Community is an AI-powered Agent built using Microsoft Copilot Studio. It helps users quickly find relevant discussions, articles, and learning resources across the Dynamics 365 and Power Platform communities. 
 
The agent interprets natural language questions and retrieves grounded answers based only on approved Microsoft-owned sources — specifically the Community site content and Microsoft Learn. It provides a self-service experience that improves content discoverability and reduces the time users spend searching across multiple pages. 
 
What are the capabilities of the Concierge Agent for Community? 
The Concierge Agent for Community provides the following capabilities: 
  • Grounded information retrieval: Generates answers only from Dynamics 365 and Power Platform Community and Microsoft Learn content. 
  • Natural language understanding: Interprets user queries and reformulates them to retrieve the most relevant content. 
  • Scoped focus: Responds exclusively to Dynamics 365 and Power Platform topics; off-topic queries trigger a fixed fallback message. 
  • Transparent citations: Provides direct links to the original Community or Learn sources so users can verify information. 
  • Built-in safety filters: Uses Copilot Studio’s Responsible AI filters to block harmful or disallowed content, including hate, violence, sexual, or self-harm topics. 
  • Continuous evaluation: Tested regularly through Power Automate + AI Builder pipelines to validate grounding accuracy and relevancy. 
How is the Concierge Agent for Community evaluated? What metrics are used to measure performance? 
The Concierge Agent for Community underwent comprehensive Responsible AI and functional testing before being released to Public Preview.
Testing methodology includes: 
  • Automated batch evaluation: Power Automate runs curated test queries and records AI responses. 
  • AI Builder evaluation prompt: Calculates relevancy, grounding overlap, and retrieval quality with defined thresholds for pass/fail. 
  • Quantitative Metrics
    • Relevancy Score: Degree to which the response answers the user’s question 
    • Grounding overlap: Confidence that the response aligns with verified Community or Learn sources 
    • Retrieval in Top-k: Boolean estimate that the answer reflects top relevant information
  • Continuous monitoring: Weekly regression testing, canary traffic sampling, and manual reviews maintain performance and RAI compliance.
What are the limitations of the Concierge Agent for Community? How can users minimize the impact of these limitations? 
The Concierge Agent for Community is currently in Public Preview and has the following known limitations: 
  • Scope limitation: Answers only questions related to Dynamics 365 and Power Platform. Off-topic questions trigger a fallback message instead of open-domain generation. 
  • English language support: Currently supports English-language content only. 
  • No human escalation: Does not hand off to human agents; users are guided to official product support pages if needed. 
  • Potential partial grounding: In rare cases, responses may summarize broadly related content if exact matches are unavailable, though all outputs are drawn from Microsoft-owned domains. 
  • AI variability: As with any AI system, minor variations in phrasing may occur for similar queries. 
Users are encouraged to review the provided source links to verify the information. 
 
What data is collected or used by the Concierge Agent for Community? 
The Concierge Agent for Community is designed with privacy and Responsible AI principles in mind: 
  • Only the text input from the user is processed to retrieve relevant Community and Learn content. 
  • No personally identifiable information (PII) such as names, emails, or IP addresses is stored or transmitted. 
  • Inputs and outputs are processed transiently within the secure Microsoft Azure environment and are not used for model training. 
  • All audit and telemetry logs are redacted of any identifiable information and managed in accordance with Microsoft’s security and compliance standards. 
What mitigations are in place to ensure Responsible AI use? 
Multiple safety and governance controls are implemented: 
  • RAI filters: Copilot Studio’s built-in filters block unsafe or harmful content categories. 
  • Strict grounding enforcement: Responses are limited to approved Community and Learn sources; unsupported queries trigger a fallback message. 
  • Prompt guardrails: Reduce speculative or unverified generation. 
  • Automated evaluation: Power Automate and AI Builder testing pipeline tracks grounding and relevancy. 
How are users made aware that they are interacting with AI, and that responses may be inaccurate? 
The Concierge Agent for Community clearly identifies itself as an AI-powered assistant and displays a disclaimer in the chat interface: 
 
“This chatbot uses AI to help you find information from the Dynamics 365 and Power Platform Community and Microsoft Learn. Responses are generated automatically and may not always be accurate. Please verify details using the linked sources.” 
 
This ensures users understand they are interacting with an automated system and encourages verification of provided information. 
 
For information about best practices when using AI to help answer forum questions, see  Writing effective responses in the community.
 
 

Comments

*This post is locked for comments