- [Home](/content/site-root.html) Home - [Wiki](/content/wiki/index.html) Wiki - [Review](/content/review/index.html) Review - [Docs](https://docs.devin.ai/) Docs PR #12668 - ## 1ToolReference: Struct → Enum 3 files+122−34 common.rssgl-router/src/protocols chat.rssgl-router/src/protocols chat\_completion.rssgl-router/tests/spec Read explanation The `tool_choice` parameter can reference different tool types. The old struct couldn't represent this: rust ```rust // BEFOREstruct ToolReference { tool_type: String, // "function" name: String,}// AFTERenum ToolReference { Function { name: String }, Mcp { server_label: String, name: Option }, FileSearch, WebSearchPreview, // ...} ``` Consequence → Chat API validation Chat Completion API only supports function tools. Added validation to reject other types with a clear error message chat.rs:483-496 Consequence → Test updates Tests updated to use new enum syntax. - ## 2Responses API Validation 1 file+78 responses.rssgl-router/src/protocols Read explanation Added cross-field validation for `tool_choice` \+ `tools` in `ResponsesRequest`: rust ```rust fn validate_responses_cross_parameters(req) { // If tool_choice references a function, // verify it exists in tools array if let Function { name } = tool_choice { if !tools.contains(name) { return Err("function not found") } } // Validate AllowedTools mode is // "auto" or "required" if let AllowedTools { mode, .. } = tool_choice { if mode != "auto" && mode != "required" { return Err("invalid mode") } }} ``` - ## 3Harmony Router: tool\_choice Constraint Generation 4 files+64−9 preparation.rssgl-router/src/routers/grpc/harmony/stages request\_building.rssgl-router/src/routers/grpc/harmony/stages responses.rs(1/2)sgl-router/src/routers/grpc/harmony sglang\_scheduler.rssgl-router/src/grpc\_client Read explanation The Harmony router uses "structural tags" to constrain model output. Added tool constraint generation for Responses API in the preparation stage: preparation.rs:131-148 rust ```rust // 1. Extract function + MCP toolslet tools = extract_tools_from_response_tools( request.tools, true // include MCP);// 2. Filter by tool_choice if neededif let Some(filtered) = filter_tools_by_tool_choice(&tools, &tool_choice) { tools = filtered;}// 3. Generate Harmony structural tagslet constraints = generate_harmony_structural_tag( &tools, &tool_choice)?; ``` The constraints are passed through request building request\_building.rs:87 to the gRPC client, which now accepts `tool_call_constraint` for Responses API requests sglang\_scheduler.rs:316 Infinite loop prevention: After receiving tool results, switch to `"auto"` so the model can choose to stop responses.rs:1183-1186 - ## 4Regular Router: tool\_choice Pass-through 2 files+101−95 conversions.rssgl-router/src/routers/grpc/regular/responses tool\_loop.rssgl-router/src/routers/grpc/regular/responses Read explanation The Regular router converts Responses API requests to Chat API format. Now extracts function tools and passes `tool_choice` through: conversions.rs:154-165 rust ```rust // Extract ONLY function tools here// (MCP tools merged later by tool loop)let function_tools = extract_tools_from_response_tools( req.tools, false // exclude MCP);// Pass through tool_choice unchangedOk(ChatCompletionRequest { tools: Some(function_tools), tool_choice: req.tool_choice.clone(), // ...}) ``` Tool loop: merging MCP tools When MCP tools are present, the tool loop merges them with function tools before calling the chat pipeline. The iteration logic prevents infinite loops: tool\_loop.rs:70-92 rust ```rust fn prepare_chat_tools_and_choice( chat_request: &mut ChatCompletionRequest, mcp_tools: &[Tool], iteration: usize,) { // Merge function + MCP tools let mut all = chat_request.tools.unwrap_or_default(); all.extend(mcp_tools.iter().cloned()); chat_request.tools = Some(all); // Iteration 0: use user's tool_choice // Iteration 1+: force "auto" to avoid loops chat_request.tool_choice = if iteration == 0 { chat_request.tool_choice.clone() .or(Some(ToolChoice::Value(Auto))) } else { Some(ToolChoice::Value(Auto)) };} ``` - ## 5Shared Utilities 2 files+88−25 utils.rssgl-router/src/routers/grpc/common/responses utils.rssgl-router/src/routers/grpc Read explanation Extracted common logic to reduce duplication between routers: `extract_tools_from_response_tools()` \- Extracts function tools (and optionally MCP tools) from `ResponseTools`: - Harmony: `include_mcp: true` (MCP schemas pre-populated) - Regular: `include_mcp: false` (MCP merged later in tool loop) utils.rs:97-125 `filter_tools_by_tool_choice()` \- Generic tool filtering extracted from the Chat-specific function. Now used by both Chat and Responses APIs: utils.rs:276-305 rust ```rust // Generic: returns filtered tools or Nonefn filter_tools_by_tool_choice( tools: &[Tool], tool_choice: &Option,) -> Option>// Chat-specific: returns Cowfn filter_chat_request_by_tool_choice( body: &ChatCompletionRequest,) -> Cow<'_, ChatCompletionRequest> ``` - ## 6Miscellaneous 3 files+4−21 handlers.rssgl-router/src/routers/grpc/regular/responses preparation.rssgl-router/src/routers/grpc/regular/stages/chat responses.rs(2/2)sgl-router/src/routers/grpc/harmony Read explanation - Import changes and doc comment cleanup - `ToolChoice::serialize_to_string()` helper for response serialization common.rs:186-196 - Function rename: `filter_tools_for_request` → `filter_chat_request_by_tool_choice` Merged [sgl-project/sglang #12668](https://github.com/sgl-project/sglang/pull/12668) # \[router\]\[grpc\] Implement tool\_choice support for Responses API [CACatherineSue](https://github.com/CatherineSue) [main](https://github.com/sgl-project/sglang/tree/main) [chang/responses-fix](https://github.com/sgl-project/sglang/tree/chang/responses-fix) 14 files +457−184 DescriptionDiscussion2Commits3 Devin's AI analysis This PR adds `tool_choice` support to the `/v1/responses` endpoint, allowing users to control tool calling behavior: ``` tool_choice: "none" → prevent tool callstool_choice: "auto" → let model decidetool_choice: "required" → force tool calltool_choice: {function} → force specific tooltool_choice: {allowed} → restrict to subset ``` Key changes: 1. `ToolReference` becomes a tagged enum to properly represent different tool types (functions, MCP, hosted tools). This cascades to Chat API validation and tests. 2. Harmony router generates structural tag constraints from `tool_choice` in the preparation stage preparation.rs:131-148 3. Regular router passes `tool_choice` through to the chat pipeline. For MCP requests, tools are merged before constraint generation tool\_loop.rs:70-92 4. Infinite loop prevention: When `tool_choice: "required"`, the model would loop forever. After the first iteration, both routers switch to `"auto"`responses.rs:1183-1186 ## Motivation The Responses API (`/v1/responses`) did not support the `tool_choice` parameter, which is a critical feature for controlling tool calling behavior according to OpenAI's Responses API specification. This limitation prevented users from: - Forcing the model to call specific tools (`tool_choice: {type: "function", function: {name: "..."}}`) - Requiring at least one tool call (`tool_choice: "required"`) - Restricting tool calls to a subset of available tools (`tool_choice: {type: "allowed_tools", ...}`) - Preventing tool calls entirely (`tool_choice: "none"`) This PR implements full `tool_choice` support for the Responses API in both Harmony and Regular routers, following the same pattern as the existing Chat Completion API implementation. ## Modifications ### 1\. Harmony Router - Responses API tool\_choice Support File: `src/routers/grpc/harmony/stages/preparation.rs` - Added `tool_choice` constraint generation for Responses API requests - Extracts both Function and MCP tools from `ResponseTools` (schemas populated before pipeline) - Implements AllowedTools filtering to restrict tools based on user's tool\_choice - Generates Harmony structural tags using `triggered_tags` format for tool constraints - Supports all tool\_choice modes: none, auto, required, specific function, allowed\_tools ### 2\. Regular Router - Responses API tool\_choice Support Files: - `src/routers/grpc/regular/responses/conversions.rs` - `src/routers/grpc/regular/responses/tool_loop.rs` Flow: - `conversions.rs`: Extracts function tools and passes through `tool_choice` unchanged - Without MCP: Function tools → chat pipeline → `tool_choice` constraints applied - With MCP: Tool loop merges function + MCP tools → chat pipeline → constraints applied to ALL tools - Loop iteration logic: Iteration 0 uses user's `tool_choice`, iteration 1+ uses "auto" to prevent infinite loops ### 3\. ToolReference Protocol Enhancement File: `src/protocols/common.rs` Converted `ToolReference` from a simple struct to a tagged enum to properly support different tool types: rust ```rust // Before (struct)pub struct ToolReference { pub tool_type: String, pub name: String,}// After (enum)pub enum ToolReference { Function { name: String }, Mcp { server_label: String, name: Option }, FileSearch, WebSearchPreview, ComputerUsePreview, CodeInterpreter, ImageGeneration,} ``` Benefits: - Type-safe representation of different tool types - Supports MCP and hosted tools (not just functions) - Helper methods: `identifier()`, `function_name()` ### 4\. Chat API Validation File: `src/protocols/chat.rs` Added validation that Chat Completion API ONLY accepts `Function` type ToolReference in tool\_choice. Rejects MCP and hosted tools with clear error messages, enforcing the API contract. ### 5\. Code Deduplication File: `src/routers/grpc/common/responses/utils.rs` Created shared utility function `extract_tools_from_response_tools()` to eliminate duplication: - Used by both Harmony preparation stage and Regular conversions - `include_mcp` parameter controls whether to extract MCP tools - Comprehensive documentation explains usage differences between routers ### 6\. Helper Function for Chat Requests File: `src/routers/grpc/regular/responses/tool_loop.rs` Created `prepare_chat_tools_and_choice()` helper to: - Merge function tools from request with MCP tools - Set `tool_choice` based on iteration (user's choice on iteration 0, "auto" on iteration 1+) - Reduces duplication in tool loop ### 7\. Bug Fixes - Fixed unnecessary `pub use`: Changed `pub use crate::tokenizer::StopSequenceDecoder` to regular `use` in `utils.rs` - Updated test cases: Fixed 6 test functions in `tests/spec/chat_completion.rs` to use new ToolReference enum syntax ### 8\. Documentation Added comprehensive comments explaining: - Tool extraction flow in `conversions.rs` (why MCP path "wastes" initial extraction) - Differences between Harmony and Regular router tool handling - Tool loop iteration behavior with `tool_choice` ## Accuracy Tests ## Benchmarking and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed). Filter GE gemini-code-assist177d > \[!WARNING\] > You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! 641 lines left # 1ToolReference: Struct → Enum 0 / 3 common.rssgl-router/src/protocols +78−4 Mark as viewed 5 linesAll 161 lines5 lines 5 linesAll 161 lines5 lines 162 `/// Tool choice for both Chat Completion and Responses APIs` 162 `/// Tool choice for both Chat Completion and Responses APIs` 163 `#[derive(Debug, Clone, Deserialize, Serialize)]` 163 `#[derive(Debug, Clone, Deserialize, Serialize)]` 164 `#[serde(untagged)]` 164 `#[serde(untagged)]` 165 `pub enum ToolChoice {` 165 `pub enum ToolChoice {` 5 linesAll 12 lines5 lines 5 linesAll 12 lines5 lines 178 `}` 178 `}` 179 179 180 `impl Default for ToolChoice {` 180 `impl Default for ToolChoice {` 181 ` fn default() -> Self {` 181 ` fn default() -> Self {` 182 ` Self::Value(ToolChoiceValue::Auto)` 182 ` Self::Value(ToolChoiceValue::Auto)` 183 ` }` 183 ` }` 184 `}` 184 `}` 185 185 186 `impl ToolChoice {` 187 ` /// Serialize tool_choice to string for ResponsesResponse` 188 ` ///` 189 ` /// Returns the JSON-serialized tool_choice or "auto" as default` 190 ` pub fn serialize_to_string(tool_choice: &Option) -> String {` 191 ` tool_choice` 192 ` .as_ref()` 193 ` .map(|tc| serde_json::to_string(tc).unwrap_or_else(|_| "auto".to_string()))` 194 ` .unwrap_or_else(|| "auto".to_string())` 195 ` }` 196 `}` 197 All 6 lines All 6 lines 192 `/// Tool reference for ToolChoice::AllowedTools` 204 `/// Tool reference for ToolChoice::AllowedTools` 205 `///` 206 `/// Represents a reference to a specific tool in the allowed_tools array.` 207 `/// Different tool types have different required fields.` 193 `#[derive(Debug, Clone, Deserialize, Serialize)]` 208 `#[derive(Debug, Clone, Deserialize, Serialize)]` 194 `pub struct ToolReference {` 209 `#[serde(tag = "type")]` 195 ` #[serde(rename = "type")]` 210 `#[serde(rename_all = "snake_case")]` 196 ` pub tool_type: String, // "function"` 211 `pub enum ToolReference {` 197 ` pub name: String,` 212 ` /// Reference to a function tool` 213 ` #[serde(rename = "function")]` 214 ` Function { name: String },` 215 216 ` /// Reference to an MCP tool` 217 ` #[serde(rename = "mcp")]` 218 ` Mcp {` 219 ` server_label: String,` 220 ` #[serde(skip_serializing_if = "Option::is_none")]` 221 ` name: Option,` 222 ` },` 223 224 ` /// File search hosted tool` 225 ` #[serde(rename = "file_search")]` 226 ` FileSearch,` 227 228 ` /// Web search preview hosted tool` 229 ` #[serde(rename = "web_search_preview")]` 230 ` WebSearchPreview,` 231 232 ` /// Computer use preview hosted tool` 233 ` #[serde(rename = "computer_use_preview")]` 234 ` ComputerUsePreview,` 235 236 ` /// Code interpreter hosted tool` 237 ` #[serde(rename = "code_interpreter")]` 238 ` CodeInterpreter,` 239 240 ` /// Image generation hosted tool` 241 ` #[serde(rename = "image_generation")]` 242 ` ImageGeneration,` 243 `}` 244 245 `impl ToolReference {` 246 ` /// Get a unique identifier for this tool reference` 247 ` pub fn identifier(&self) -> String {` 248 ` match self {` 249 ` ToolReference::Function { name } => format!("function:{}", name),` 250 ` ToolReference::Mcp { server_label, name } => {` 251 ` if let Some(n) = name {` 252 ` format!("mcp:{}:{}", server_label, n)` 253 ` } else {` 254 ` format!("mcp:{}", server_label)` 255 ` }` 256 ` }` 257 ` ToolReference::FileSearch => "file_search".to_string(),` 258 ` ToolReference::WebSearchPreview => "web_search_preview".to_string(),` 259 ` ToolReference::ComputerUsePreview => "computer_use_preview".to_string(),` 260 ` ToolReference::CodeInterpreter => "code_interpreter".to_string(),` 261 ` ToolReference::ImageGeneration => "image_generation".to_string(),` 262 ` }` 263 ` }` 264 265 ` /// Get the tool name if this is a function tool` 266 ` pub fn function_name(&self) -> Option<&str> {` 267 ` match self {` 268 ` ToolReference::Function { name } => Some(name.as_str()),` 269 ` _ => None,` 270 ` }` 271 ` }` 198 `}` 272 `}` 5 linesAll 148 lines5 lines 5 linesAll 148 lines5 lines chat.rssgl-router/src/protocols +36−14 Mark as viewed 5 linesAll 337 lines5 lines 5 linesAll 337 lines5 lines 338 `fn validate_chat_cross_parameters(` 338 `fn validate_chat_cross_parameters(` 339 ` req: &ChatCompletionRequest,` 339 ` req: &ChatCompletionRequest,` 340 `) -> Result<(), validator::ValidationError> {` 340 `) -> Result<(), validator::ValidationError> {` 5 linesAll 68 lines5 lines 5 linesAll 68 lines5 lines 409 ` if let Some(ref tool_choice) = req.tool_choice {` 409 ` if let Some(ref tool_choice) = req.tool_choice {` 5 linesAll 12 lines5 lines 5 linesAll 12 lines5 lines 422 ` if has_tools {` 422 ` if has_tools {` 423 ` let tools = req.tools.as_ref().unwrap();` 423 ` let tools = req.tools.as_ref().unwrap();` 424 424 425 ` match tool_choice {` 425 ` match tool_choice {` 5 linesAll 19 lines5 lines 5 linesAll 19 lines5 lines 445 ` ToolChoice::AllowedTools {` 445 ` ToolChoice::AllowedTools {` 446 ` mode,` 446 ` mode,` 447 ` tools: allowed_tools,` 447 ` tools: allowed_tools,` 448 ` ..` 448 ` ..` 449 ` } => {` 449 ` } => {` 5 linesAll 10 lines5 lines 5 linesAll 10 lines5 lines 460 ` // Validate that all referenced tool names exist in tools` 460 ` // Validate that all ToolReferences are Function type (Chat API only supports function tools)` 461 ` for tool_ref in allowed_tools {` 461 ` for tool_ref in allowed_tools {` 462 ` let tool_exists = tools.iter().any(|tool| {` 462 ` match tool_ref {` 463 ` tool.tool_type == tool_ref.tool_type` 463 ` ToolReference::Function { name } => {` 464 ` && tool.function.name == tool_ref.name` 464 ` // Validate that the function exists in tools array` 465 ` });` 465 ` let tool_exists = tools.iter().any(|tool| {` 466 466 ` tool.tool_type == "function" && tool.function.name == *name` 467 ` if !tool_exists {` 467 ` });` 468 ` let mut e =` 468 469 ` validator::ValidationError::new("tool_choice_tool_not_found");` 469 ` if !tool_exists {` 470 ` e.message = Some(format!(` 470 ` let mut e = validator::ValidationError::new(` 471 ` "Invalid value for 'tool_choice.tools': tool '{}' not found in 'tools'.",` 471 ` "tool_choice_tool_not_found",` 472 ` tool_ref.name` 472 ` );` 473 ` ).into());` 473 ` e.message = Some(` 474 ` return Err(e);` 474 ` format!(` 475 ` "Invalid value for 'tool_choice.tools': tool '{}' not found in 'tools'.",` 476 ` name` 477 ` )` 478 ` .into(),` 479 ` );` 480 ` return Err(e);` 481 ` }` 482 ` }` 483 ` _ => {` 484 ` // Chat Completion API only supports function tools in tool_choice` 485 ` let mut e = validator::ValidationError::new(` 486 ` "tool_choice_invalid_tool_type",` 487 ` );` 488 ` e.message = Some(` 489 ` format!(` 490 ` "Invalid value for 'tool_choice.tools': Chat Completion API only supports function tools, got '{}'.",` 491 ` tool_ref.identifier()` 492 ` )` 493 ` .into(),` 494 ` );` 495 ` return Err(e);` 496 ` }` 475 ` }` 497 ` }` 476 ` }` 498 ` }` 477 ` }` 499 ` }` 478 ` _ => {}` 500 ` _ => {}` 479 ` }` 501 ` }` 480 ` }` 502 ` }` 481 ` }` 503 ` }` 482 504 483 ` Ok(())` 505 ` Ok(())` 484 `}` 506 `}` 5 linesAll 197 lines5 lines 5 linesAll 197 lines5 lines chat\_completion.rssgl-router/tests/spec +8−16 Mark as viewed 5 linesAll 332 lines5 lines 5 linesAll 332 lines5 lines 333 `#[test]` 333 `#[test]` 334 `fn test_tool_choice_allowed_tools_invalid_mode() {` 334 `fn test_tool_choice_allowed_tools_invalid_mode() {` 335 ` let req = ChatCompletionRequest {` 335 ` let req = ChatCompletionRequest {` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 350 ` tool_choice: Some(ToolChoice::AllowedTools {` 350 ` tool_choice: Some(ToolChoice::AllowedTools {` 351 ` mode: "invalid_mode".to_string(),` 351 ` mode: "invalid_mode".to_string(),` 352 ` tools: vec![ToolReference {`\ \ 352\ \ ` tools: vec![ToolReference::Function {`\ \ 353\ \ ` tool_type: "function".to_string(),`\ \ 354\ \ ` name: "get_weather".to_string(),`\ \ 353\ \ ` name: "get_weather".to_string(),`\ \ 355\ \ ` }],`\ \ 354\ \ ` }],` 356 ` tool_type: "function".to_string(),` 355 ` tool_type: "function".to_string(),` 357 ` }),` 356 ` }),` 358 ` ..Default::default()` 357 ` ..Default::default()` 359 ` };` 358 ` };` All 9 lines All 9 lines 369 `}` 368 `}` 370 369 371 `#[test]` 370 `#[test]` 372 `fn test_tool_choice_allowed_tools_valid_mode_auto() {` 371 `fn test_tool_choice_allowed_tools_valid_mode_auto() {` 373 ` let req = ChatCompletionRequest {` 372 ` let req = ChatCompletionRequest {` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 388 ` tool_choice: Some(ToolChoice::AllowedTools {` 387 ` tool_choice: Some(ToolChoice::AllowedTools {` 389 ` mode: "auto".to_string(),` 388 ` mode: "auto".to_string(),` 390 ` tools: vec![ToolReference {`\ \ 389\ \ ` tools: vec![ToolReference::Function {`\ \ 391\ \ ` tool_type: "function".to_string(),`\ \ 392\ \ ` name: "get_weather".to_string(),`\ \ 390\ \ ` name: "get_weather".to_string(),`\ \ 393\ \ ` }],`\ \ 391\ \ ` }],` 394 ` tool_type: "function".to_string(),` 392 ` tool_type: "function".to_string(),` 395 ` }),` 393 ` }),` 396 ` ..Default::default()` 394 ` ..Default::default()` 397 ` };` 395 ` };` 398 396 399 ` let result = req.validate();` 397 ` let result = req.validate();` 400 ` assert!(result.is_ok(), "Should accept 'auto' mode");` 398 ` assert!(result.is_ok(), "Should accept 'auto' mode");` 401 `}` 399 `}` 402 400 403 `#[test]` 401 `#[test]` 404 `fn test_tool_choice_allowed_tools_valid_mode_required() {` 402 `fn test_tool_choice_allowed_tools_valid_mode_required() {` 405 ` let req = ChatCompletionRequest {` 403 ` let req = ChatCompletionRequest {` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 420 ` tool_choice: Some(ToolChoice::AllowedTools {` 418 ` tool_choice: Some(ToolChoice::AllowedTools {` 421 ` mode: "required".to_string(),` 419 ` mode: "required".to_string(),` 422 ` tools: vec![ToolReference {`\ \ 420\ \ ` tools: vec![ToolReference::Function {`\ \ 423\ \ ` tool_type: "function".to_string(),`\ \ 424\ \ ` name: "get_weather".to_string(),`\ \ 421\ \ ` name: "get_weather".to_string(),`\ \ 425\ \ ` }],`\ \ 422\ \ ` }],` 426 ` tool_type: "function".to_string(),` 423 ` tool_type: "function".to_string(),` 427 ` }),` 424 ` }),` 428 ` ..Default::default()` 425 ` ..Default::default()` 429 ` };` 426 ` };` 430 427 431 ` let result = req.validate();` 428 ` let result = req.validate();` 432 ` assert!(result.is_ok(), "Should accept 'required' mode");` 429 ` assert!(result.is_ok(), "Should accept 'required' mode");` 433 `}` 430 `}` 434 431 435 `#[test]` 432 `#[test]` 436 `fn test_tool_choice_allowed_tools_tool_not_found() {` 433 `fn test_tool_choice_allowed_tools_tool_not_found() {` 437 ` let req = ChatCompletionRequest {` 434 ` let req = ChatCompletionRequest {` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 452 ` tool_choice: Some(ToolChoice::AllowedTools {` 449 ` tool_choice: Some(ToolChoice::AllowedTools {` 453 ` mode: "auto".to_string(),` 450 ` mode: "auto".to_string(),` 454 ` tools: vec![ToolReference {`\ \ 451\ \ ` tools: vec![ToolReference::Function {`\ \ 455\ \ ` tool_type: "function".to_string(),`\ \ 456\ \ ` name: "nonexistent_tool".to_string(),`\ \ 452\ \ ` name: "nonexistent_tool".to_string(),`\ \ 457\ \ ` }],`\ \ 453\ \ ` }],` 458 ` tool_type: "function".to_string(),` 454 ` tool_type: "function".to_string(),` 459 ` }),` 455 ` }),` 460 ` ..Default::default()` 456 ` ..Default::default()` 461 ` };` 457 ` };` All 9 lines All 9 lines 471 `}` 467 `}` 472 468 473 `#[test]` 469 `#[test]` 474 `fn test_tool_choice_allowed_tools_multiple_tools_valid() {` 470 `fn test_tool_choice_allowed_tools_multiple_tools_valid() {` 475 ` let req = ChatCompletionRequest {` 471 ` let req = ChatCompletionRequest {` 5 linesAll 25 lines5 lines 5 linesAll 25 lines5 lines 501 ` tool_choice: Some(ToolChoice::AllowedTools {` 497 ` tool_choice: Some(ToolChoice::AllowedTools {` 502 ` mode: "auto".to_string(),` 498 ` mode: "auto".to_string(),` 503 ` tools: vec![`\ \ 499\ \ ` tools: vec![`\ \ 504\ \ ` ToolReference {`\ \ 500\ \ ` ToolReference::Function {`\ \ 505\ \ ` tool_type: "function".to_string(),`\ \ 506\ \ ` name: "get_weather".to_string(),`\ \ 501\ \ ` name: "get_weather".to_string(),`\ \ 507\ \ ` },`\ \ 502\ \ ` },`\ \ 508\ \ ` ToolReference {`\ \ 503\ \ ` ToolReference::Function {`\ \ 509\ \ ` tool_type: "function".to_string(),`\ \ 510\ \ ` name: "get_time".to_string(),`\ \ 504\ \ ` name: "get_time".to_string(),`\ \ 511\ \ ` },`\ \ 505\ \ ` },`\ \ 512\ \ ` ],`\ \ 506\ \ ` ],` 513 ` tool_type: "function".to_string(),` 507 ` tool_type: "function".to_string(),` 514 ` }),` 508 ` }),` 515 ` ..Default::default()` 509 ` ..Default::default()` 516 ` };` 510 ` };` 517 511 518 ` let result = req.validate();` 512 ` let result = req.validate();` 519 ` assert!(result.is_ok(), "Should accept all valid tool references");` 513 ` assert!(result.is_ok(), "Should accept all valid tool references");` 520 `}` 514 `}` 521 515 522 `#[test]` 516 `#[test]` 523 `fn test_tool_choice_allowed_tools_one_invalid_among_valid() {` 517 `fn test_tool_choice_allowed_tools_one_invalid_among_valid() {` 524 ` let req = ChatCompletionRequest {` 518 ` let req = ChatCompletionRequest {` 5 linesAll 25 lines5 lines 5 linesAll 25 lines5 lines 550 ` tool_choice: Some(ToolChoice::AllowedTools {` 544 ` tool_choice: Some(ToolChoice::AllowedTools {` 551 ` mode: "auto".to_string(),` 545 ` mode: "auto".to_string(),` 552 ` tools: vec![`\ \ 546\ \ ` tools: vec![`\ \ 553\ \ ` ToolReference {`\ \ 547\ \ ` ToolReference::Function {`\ \ 554\ \ ` tool_type: "function".to_string(),`\ \ 555\ \ ` name: "get_weather".to_string(),`\ \ 548\ \ ` name: "get_weather".to_string(),`\ \ 556\ \ ` },`\ \ 549\ \ ` },`\ \ 557\ \ ` ToolReference {`\ \ 550\ \ ` ToolReference::Function {`\ \ 558\ \ ` tool_type: "function".to_string(),`\ \ 559\ \ ` name: "nonexistent_tool".to_string(),`\ \ 551\ \ ` name: "nonexistent_tool".to_string(),`\ \ 560\ \ ` },`\ \ 552\ \ ` },`\ \ 561\ \ ` ],`\ \ 553\ \ ` ],` 562 ` tool_type: "function".to_string(),` 554 ` tool_type: "function".to_string(),` 563 ` }),` 555 ` }),` 564 ` ..Default::default()` 556 ` ..Default::default()` 565 ` };` 557 ` };` 5 linesAll 12 lines5 lines 5 linesAll 12 lines5 lines 578 `}` 570 `}` # 2Responses API Validation 0 / 1 responses.rssgl-router/src/protocols +78 Mark as viewed 5 linesAll 448 lines5 lines 5 linesAll 448 lines5 lines 449 `#[derive(Debug, Clone, Deserialize, Serialize, Validate)]` 449 `#[derive(Debug, Clone, Deserialize, Serialize, Validate)]` 450 `#[validate(schema(function = "validate_responses_cross_parameters"))]` 450 `pub struct ResponsesRequest {` 451 `pub struct ResponsesRequest {` 5 linesAll 98 lines5 lines 5 linesAll 98 lines5 lines 549 550 5 linesAll 10 lines5 lines 5 linesAll 10 lines5 lines 560 ` pub stop: Option,` 561 ` pub stop: Option,` 561 562 5 linesAll 162 lines5 lines 5 linesAll 162 lines5 lines 1 Copy + paste725–800 725 `/// Schema-level validation for cross-field dependencies` 726 `fn validate_responses_cross_parameters(` 727 ` request: &ResponsesRequest,` 728 `) -> Result<(), validator::ValidationError> {` 729 ` use super::common::{ToolChoice, ToolReference};` 730 731 ` // Only validate if both tools and tool_choice are present` 732 ` if let (Some(tools), Some(tool_choice)) = (&request.tools, &request.tool_choice) {` 733 ` // Extract function tool names from ResponseTools` 734 ` let function_tool_names: Vec<&str> = tools` 735 ` .iter()` 736 ` .filter_map(|t| match t.r#type {` 737 ` ResponseToolType::Function => t.function.as_ref().map(|f| f.name.as_str()),` 738 ` _ => None,` 739 ` })` 740 ` .collect();` 741 742 ` match tool_choice {` 743 ` ToolChoice::Function { function, .. } => {` 744 ` // Validate the specific function exists` 745 ` if !function_tool_names.contains(&function.name.as_str()) {` 746 ` let mut e = validator::ValidationError::new("tool_choice_function_not_found");` 747 ` e.message = Some(` 748 ` format!(` 749 ` "Invalid value for 'tool_choice': function '{}' not found in 'tools'.",` 750 ` function.name` 751 ` )` 752 ` .into(),` 753 ` );` 754 ` return Err(e);` 755 ` }` 756 ` }` 757 ` ToolChoice::AllowedTools {` 758 ` mode,` 759 ` tools: allowed_tools,` 760 ` ..` 761 ` } => {` 762 ` // Validate mode is "auto" or "required"` 763 ` if mode != "auto" && mode != "required" {` 764 ` let mut e = validator::ValidationError::new("tool_choice_invalid_mode");` 765 ` e.message = Some(` 766 ` format!(` 767 ` "Invalid value for 'tool_choice.mode': must be 'auto' or 'required', got '{}'.",` 768 ` mode` 769 ` )` 770 ` .into(),` 771 ` );` 772 ` return Err(e);` 773 ` }` 774 775 ` // Validate that all function tool references exist` InformationalR776-793 776 ` for tool_ref in allowed_tools {` 777 ` if let ToolReference::Function { name } = tool_ref {` 778 ` if !function_tool_names.contains(&name.as_str()) {` 779 ` let mut e =` 780 ` validator::ValidationError::new("tool_choice_tool_not_found");` 781 ` e.message = Some(` 782 ` format!(` 783 ` "Invalid value for 'tool_choice.tools': tool '{}' not found in 'tools'.",` 784 ` name` 785 ` )` 786 ` .into(),` 787 ` );` 788 ` return Err(e);` 789 ` }` 790 ` }` 791 ` // Note: MCP and hosted tools don't need existence validation here` 792 ` // as they are resolved dynamically at runtime` 793 ` }` 794 ` }` 795 ` _ => {}` 796 ` }` 797 ` }` 798 799 ` Ok(())` 800 `}` 801 5 linesAll 236 lines5 lines 5 linesAll 236 lines5 lines # 3Harmony Router: tool\_choice Constraint Generation 0 / 4 preparation.rssgl-router/src/routers/grpc/harmony/stages +24−5 Mark as viewed All 7 lines All 7 lines 8 `use crate::{` 8 `use crate::{` All 5 lines All 5 lines 14 ` routers::grpc::{` 14 ` routers::grpc::{` 15 ` common::stages::PipelineStage,` 15 ` common::{responses::utils::extract_tools_from_response_tools, stages::PipelineStage},` 16 ` context::{PreparationOutput, RequestContext, RequestType},` 16 ` context::{PreparationOutput, RequestContext, RequestType},` 17 ` error, utils,` 17 ` error, utils,` 18 ` },` 18 ` },` 19 `};` 19 `};` All 5 lines All 5 lines 25 `pub struct HarmonyPreparationStage {` 25 `pub struct HarmonyPreparationStage {` 26 ` builder: HarmonyBuilder,` 26 ` builder: HarmonyBuilder,` 27 `}` 27 `}` 5 linesAll 44 lines5 lines 5 linesAll 44 lines5 lines 72 `impl HarmonyPreparationStage {` 72 `impl HarmonyPreparationStage {` 73 ` /// Prepare a chat completion request using Harmony encoding` 73 ` /// Prepare a chat completion request using Harmony encoding` 74 ` async fn prepare_chat(` 74 ` async fn prepare_chat(` 75 ` &self,` 75 ` &self,` 76 ` ctx: &mut RequestContext,` 76 ` ctx: &mut RequestContext,` 77 ` request: &ChatCompletionRequest,` 77 ` request: &ChatCompletionRequest,` 78 ` ) -> Result, Response> {` 78 ` ) -> Result, Response> {` All 6 lines All 6 lines 85 85 86 ` // Step 1: Filter tools if needed` 86 ` // Step 1: Filter tools if needed` 87 ` let body_ref = utils::filter_tools_for_request(request);` 87 ` let body_ref = utils::filter_chat_request_by_tool_choice(request);` 88 88 5 linesAll 28 lines5 lines 5 linesAll 28 lines5 lines 117 ` });` 117 ` });` 118 118 119 ` Ok(None)` 119 ` Ok(None)` 120 ` }` 120 ` }` 121 121 122 ` /// Prepare a responses API request using Harmony encoding` 122 ` /// Prepare a responses API request using Harmony encoding` 123 ` ///` 123 ` ///` 124 ` /// For responses API, we build from conversation history using the same Harmony` 124 ` /// For responses API, we build from conversation history using the same Harmony` 125 ` /// encoding that the builder provides. This handles the MCP loop integration.` 125 ` /// encoding that the builder provides. This handles the MCP loop integration.` 126 ` pub async fn prepare_responses(` 126 ` pub async fn prepare_responses(` 127 ` &self,` 127 ` &self,` 128 ` ctx: &mut RequestContext,` 128 ` ctx: &mut RequestContext,` 129 ` request: &ResponsesRequest,` 129 ` request: &ResponsesRequest,` 130 ` ) -> Result, Response> {` 130 ` ) -> Result, Response> {` InformationalR131-140 131 ` // Build via Harmony from responses API request` 131 ` // Step 1: Extract function and MCP tools with schemas from ResponseTools` 132 ` let mut function_tools = extract_tools_from_response_tools(request.tools.as_deref(), true);` 133 134 ` // Step 2: Filter tools based on tool_choice (AllowedTools or Function)` 135 ` // Note: Tool existence is already validated in ResponsesRequest::validate()` 136 ` if let Some(filtered) =` 137 ` utils::filter_tools_by_tool_choice(&function_tools, &request.tool_choice)` 138 ` {` 139 ` function_tools = filtered;` 140 ` }` 141 142 ` // Step 3: Generate Harmony structural tags from filtered tools` 143 ` let tool_constraints = if !function_tools.is_empty() {` 144 ` Self::generate_harmony_structural_tag(&function_tools, &request.tool_choice)` 145 ` .map_err(|e| *e)?` 146 ` } else {` 147 ` None` 148 ` };` 149 150 ` // Step 3: Build via Harmony from responses API request` 132 ` let build_output = self` 151 ` let build_output = self` 133 ` .builder` 152 ` .builder` 134 ` .build_from_responses(request)` 153 ` .build_from_responses(request)` 135 ` .map_err(|e| error::bad_request(format!("Harmony build failed: {}", e)))?;` 154 ` .map_err(|e| error::bad_request(format!("Harmony build failed: {}", e)))?;` 136 155 137 ` // Store results in preparation output` 156 ` // Step 4: Store results with tool_constraints` 138 ` ctx.state.preparation = Some(PreparationOutput {` 157 ` ctx.state.preparation = Some(PreparationOutput {` 139 ` original_text: None,` 158 ` original_text: None,` 140 ` token_ids: build_output.input_ids,` 159 ` token_ids: build_output.input_ids,` 141 ` processed_messages: None,` 160 ` processed_messages: None,` 142 ` tool_constraints: None,` 161 ` tool_constraints,` 143 ` filtered_request: None,` 162 ` filtered_request: None,` 144 ` harmony_mode: true,` 163 ` harmony_mode: true,` 145 ` selection_text: Some(build_output.selection_text),` 164 ` selection_text: Some(build_output.selection_text),` 146 ` harmony_messages: Some(build_output.harmony_messages),` 165 ` harmony_messages: Some(build_output.harmony_messages),` 147 ` harmony_stop_ids: Some(build_output.stop_token_ids),` 166 ` harmony_stop_ids: Some(build_output.stop_token_ids),` 148 ` });` 167 ` });` 149 168 150 ` Ok(None)` 169 ` Ok(None)` 151 ` }` 170 ` }` 5 linesAll 93 lines5 lines 5 linesAll 93 lines5 lines 245 `}` 264 `}` request\_building.rssgl-router/src/routers/grpc/harmony/stages +1 Mark as viewed 5 linesAll 17 lines5 lines 5 linesAll 17 lines5 lines 18 `pub struct HarmonyRequestBuildingStage {` 18 `pub struct HarmonyRequestBuildingStage {` 19 ` inject_pd_metadata: bool,` 19 ` inject_pd_metadata: bool,` 20 `}` 20 `}` All 9 lines All 9 lines 30 `impl PipelineStage for HarmonyRequestBuildingStage {` 30 `impl PipelineStage for HarmonyRequestBuildingStage {` 31 ` async fn execute(&self, ctx: &mut RequestContext) -> Result, Response> {` 31 ` async fn execute(&self, ctx: &mut RequestContext) -> Result, Response> {` 5 linesAll 32 lines5 lines 5 linesAll 32 lines5 lines 64 ` let mut proto_request = match &ctx.input.request_type {` 64 ` let mut proto_request = match &ctx.input.request_type {` 5 linesAll 15 lines5 lines 5 linesAll 15 lines5 lines 80 ` RequestType::Responses(request) => builder_client` 80 ` RequestType::Responses(request) => builder_client` 81 ` .build_generate_request_from_responses(` 81 ` .build_generate_request_from_responses(` 82 ` request_id,` 82 ` request_id,` 83 ` request.as_ref(),` 83 ` request.as_ref(),` 84 ` placeholder_processed_text,` 84 ` placeholder_processed_text,` 85 ` prep.token_ids.clone(),` 85 ` prep.token_ids.clone(),` 86 ` prep.harmony_stop_ids.clone(),` 86 ` prep.harmony_stop_ids.clone(),` 87 ` prep.tool_constraints.clone(),` 87 ` )` 88 ` )` 88 ` .map_err(|e| error::bad_request(format!("Invalid request parameters: {}", e)))?,` 89 ` .map_err(|e| error::bad_request(format!("Invalid request parameters: {}", e)))?,` 89 ` _ => unreachable!(),` 90 ` _ => unreachable!(),` 90 ` };` 91 ` };` 5 linesAll 23 lines5 lines 5 linesAll 23 lines5 lines 114 ` }` 115 ` }` All 4 lines All 4 lines 119 `}` 120 `}` responses.rssgl-router/src/routers/grpc/harmony(1 of 2) +5 Mark as viewed 5 linesAll 1097 lines5 lines 5 linesAll 1097 lines5 lines 1098 `fn build_next_request_with_tools(` 1089 `fn build_next_request_with_tools(` 1099 ` mut request: ResponsesRequest,` 1090 ` mut request: ResponsesRequest,` 1100 ` tool_calls: Vec,` 1091 ` tool_calls: Vec,` 1101 ` tool_results: Vec,` 1092 ` tool_results: Vec,` 1102 ` analysis: Option,` 1093 ` analysis: Option,` 1103 ` partial_text: String,` 1094 ` partial_text: String,` 1104 `) -> Result> {` 1095 `) -> Result> {` 5 linesAll 83 lines5 lines 5 linesAll 83 lines5 lines 1188 1179 1189 ` // Update request with new items` 1180 ` // Update request with new items` 1190 ` request.input = ResponseInput::Items(items);` 1181 ` request.input = ResponseInput::Items(items);` 1191 1182 InformationalR1183-1187 1183 ` // Switch tool_choice to "auto" for subsequent iterations` 1184 ` // This prevents infinite loops when original tool_choice was "required" or specific function` 1185 ` // After receiving tool results, the model should be free to decide whether to call more tools or finish` 1186 ` request.tool_choice = Some(ToolChoice::Value(ToolChoiceValue::Auto));` 1187 1192 ` Ok(request)` 1188 ` Ok(request)` 1193 `}` 1189 `}` 5 linesAll 214 lines5 lines 5 linesAll 214 lines5 lines sglang\_scheduler.rssgl-router/src/grpc\_client +34−4 Mark as viewed 5 linesAll 124 lines5 lines 5 linesAll 124 lines5 lines 125 `impl SglangSchedulerClient {` 125 `impl SglangSchedulerClient {` 5 linesAll 179 lines5 lines 5 linesAll 179 lines5 lines 305 ` /// Build a GenerateRequest from ResponsesRequest (OpenAI Responses API)` 305 ` /// Build a GenerateRequest from ResponsesRequest (OpenAI Responses API)` 306 ` ///` 307 ` /// NOTE: This is used by the Harmony router only. The Regular router uses` 308 ` /// responses_to_chat() conversion and goes through the chat pipeline.` 306 ` pub fn build_generate_request_from_responses(` 309 ` pub fn build_generate_request_from_responses(` 307 ` &self,` 310 ` &self,` 308 ` request_id: String,` 311 ` request_id: String,` 309 ` body: &ResponsesRequest,` 312 ` body: &ResponsesRequest,` 310 ` processed_text: String,` 313 ` processed_text: String,` 311 ` token_ids: Vec,` 314 ` token_ids: Vec,` 312 ` harmony_stop_ids: Option>,` 315 ` harmony_stop_ids: Option>,` 316 ` tool_call_constraint: Option<(String, String)>,` 313 ` ) -> Result {` 317 ` ) -> Result {` 314 ` // Build sampling params from ResponsesRequest` 318 ` // Build sampling params from ResponsesRequest` 315 ` let mut sampling_params = self.build_grpc_sampling_params_from_responses(body)?;` 319 ` let mut sampling_params =` 320 ` self.build_grpc_sampling_params_from_responses(body, tool_call_constraint)?;` 5 linesAll 22 lines5 lines 5 linesAll 22 lines5 lines 338 ` Ok(grpc_request)` 343 ` Ok(grpc_request)` 339 ` }` 344 ` }` 5 linesAll 101 lines5 lines 5 linesAll 101 lines5 lines 441 ` fn build_grpc_sampling_params_from_responses(` 446 ` fn build_grpc_sampling_params_from_responses(` 442 ` &self,` 447 ` &self,` 443 ` request: &ResponsesRequest,` 448 ` request: &ResponsesRequest,` 449 ` tool_call_constraint: Option<(String, String)>,` 444 ` ) -> Result {` 450 ` ) -> Result {` 445 ` // ResponsesRequest doesn't have stop sequences in the same way` 451 ` // ResponsesRequest doesn't have stop sequences in the same way` 446 ` // Tools are handled externally by MCP loop, not via constraints` 452 ` // For Harmony router: Tools are handled via structural_tag constraints` 447 453 448 ` let max_new_tokens = request.max_output_tokens.map(|v| v as i32);` 454 ` let max_new_tokens = request.max_output_tokens.map(|v| v as i32);` 449 455 450 ` Ok(proto::SamplingParams {` 456 ` Ok(proto::SamplingParams {` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 465 ` n: 1, // Responses API doesn't support n>1` 471 ` n: 1, // Responses API doesn't support n>1` 466 ` constraint: None, // No constraints - tools handled by MCP` 472 ` constraint: self.build_constraint_for_responses(tool_call_constraint)?,` 467 ` ..Default::default()` 473 ` ..Default::default()` 468 ` })` 474 ` })` 469 ` }` 475 ` }` 470 476 477 ` /// Build constraint for Responses API (simpler than Chat API's build_constraint)` 478 ` ///` 479 ` /// Responses API doesn't support response_format, ebnf, or regex constraints,` 480 ` /// so this only handles tool_call_constraint.` 481 ` fn build_constraint_for_responses(` 482 ` &self,` 483 ` tool_call_constraint: Option<(String, String)>,` 484 ` ) -> Result, String> {` 485 ` if let Some((constraint_type, constraint_value)) = tool_call_constraint {` 1 Copy + paste486–494 486 ` let tool_constraint = match constraint_type.as_str() {` 487 ` "structural_tag" => {` 488 ` proto::sampling_params::Constraint::StructuralTag(constraint_value)` 489 ` }` 490 ` "json_schema" => proto::sampling_params::Constraint::JsonSchema(constraint_value),` 491 ` "ebnf" => proto::sampling_params::Constraint::EbnfGrammar(constraint_value),` 492 ` "regex" => proto::sampling_params::Constraint::Regex(constraint_value),` 493 ` _ => return Err(format!("Unknown constraint type: {}", constraint_type)),` 494 ` };` 495 ` Ok(Some(tool_constraint))` 496 ` } else {` 497 ` Ok(None)` 498 ` }` 499 ` }` 500 5 linesAll 101 lines5 lines 5 linesAll 101 lines5 lines 572 `}` 602 `}` 5 linesAll 167 lines5 lines 5 linesAll 167 lines5 lines # 4Regular Router: tool\_choice Pass-through 0 / 2 conversions.rssgl-router/src/routers/grpc/regular/responses +56−71 Mark as viewed All 9 lines All 9 lines 10 `use crate::protocols::{` 10 `use crate::{` 11 ` chat::{ChatCompletionRequest, ChatCompletionResponse, ChatMessage, UserMessageContent},` 11 ` protocols::{` 12 ` common::{FunctionCallResponse, StreamOptions, ToolCall, UsageInfo},` 12 ` chat::{ChatCompletionRequest, ChatCompletionResponse, ChatMessage, UserMessageContent},` 13 ` responses::{` 13 ` common::{FunctionCallResponse, StreamOptions, ToolCall, ToolChoice, UsageInfo},` 14 ` ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,` 14 ` responses::{` 15 ` ResponseReasoningContent::ReasoningText, ResponseStatus, ResponsesRequest,` 15 ` ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,` 16 ` ResponsesResponse, ResponsesUsage, StringOrContentParts,` 16 ` ResponseReasoningContent::ReasoningText, ResponseStatus, ResponsesRequest,` 17 ` ResponsesResponse, ResponsesUsage, StringOrContentParts,` 18 ` },` 17 ` },` 19 ` },` 20 ` routers::grpc::common::responses::utils::extract_tools_from_response_tools,` 18 `};` 21 `};` 19 22 20 `/// Convert a ResponsesRequest to ChatCompletionRequest for processing through the chat pipeline` 23 `/// Convert a ResponsesRequest to ChatCompletionRequest for processing through the chat pipeline` 21 `///` 24 `///` 22 `/// # Conversion Logic` 25 `/// # Conversion Logic` 23 ``/// - `input` (text/items) → `messages` (chat messages)`` 26 ``/// - `input` (text/items) → `messages` (chat messages)`` 24 ``/// - `instructions` → system message (prepended)`` 27 ``/// - `instructions` → system message (prepended)`` 25 ``/// - `max_output_tokens` → `max_completion_tokens``` 28 ``/// - `max_output_tokens` → `max_completion_tokens``` 26 `/// - Tool-related fields are passed through` 29 ``/// - `tools` → function tools extracted from ResponseTools`` 30 ``/// - `tool_choice` → passed through from request`` 27 `/// - Response-specific fields (previous_response_id, conversation) are handled by router` 31 `/// - Response-specific fields (previous_response_id, conversation) are handled by router` 28 `pub fn responses_to_chat(req: &ResponsesRequest) -> Result {` 32 `pub fn responses_to_chat(req: &ResponsesRequest) -> Result {` 5 linesAll 10 lines5 lines 5 linesAll 10 lines5 lines 39 ` // 2. Convert input to chat messages` 43 ` // 2. Convert input to chat messages` 40 ` match &req.input {` 44 ` match &req.input {` All 7 lines All 7 lines 48 ` ResponseInput::Items(items) => {` 52 ` ResponseInput::Items(items) => {` 49 ` // Structured items → convert each to appropriate chat message` 53 ` // Structured items → convert each to appropriate chat message` 50 ` for item in items {` 54 ` for item in items {` 51 ` match item {` 55 ` match item {` 52 ` ResponseInputOutputItem::SimpleInputMessage { content, role, .. } => {` 56 ` ResponseInputOutputItem::SimpleInputMessage { content, role, .. } => {` 5 linesAll 18 lines5 lines 5 linesAll 18 lines5 lines 1 Move71–99 71 ` match role.as_str() {` 75 ` messages.push(role_to_chat_message(role.as_str(), text));` 72 ` "user" => {` 73 ` messages.push(ChatMessage::User {` 74 ` content: UserMessageContent::Text(text),` 75 ` name: None,` 76 ` });` 77 ` }` 78 ` "assistant" => {` 79 ` messages.push(ChatMessage::Assistant {` 80 ` content: Some(text),` 81 ` name: None,` 82 ` tool_calls: None,` 83 ` reasoning_content: None,` 84 ` });` 85 ` }` 86 ` "system" => {` 87 ` messages.push(ChatMessage::System {` 88 ` content: text,` 89 ` name: None,` 90 ` });` 91 ` }` 92 ` _ => {` 93 ` // Unknown role, treat as user message` 94 ` messages.push(ChatMessage::User {` 95 ` content: UserMessageContent::Text(text),` 96 ` name: None,` 97 ` });` 98 ` }` 99 ` }` 100 ` }` 76 ` }` 101 ` ResponseInputOutputItem::Message { role, content, .. } => {` 77 ` ResponseInputOutputItem::Message { role, content, .. } => {` 102 ` // Extract text from content parts` 78 ` // Extract text from content parts` 103 ` let text = extract_text_from_content(content);` 79 ` let text = extract_text_from_content(content);` 104 80 105 ` match role.as_str() {` 81 ` messages.push(role_to_chat_message(role.as_str(), text));` 106 ` "user" => {` 107 ` messages.push(ChatMessage::User {` 108 ` content: UserMessageContent::Text(text),` 109 ` name: None,` 110 ` });` 111 ` }` 112 ` "assistant" => {` 113 ` messages.push(ChatMessage::Assistant {` 114 ` content: Some(text),` 115 ` name: None,` 116 ` tool_calls: None,` 117 ` reasoning_content: None,` 118 ` });` 119 ` }` 120 ` "system" => {` 121 ` messages.push(ChatMessage::System {` 122 ` content: text,` 123 ` name: None,` 124 ` });` 125 ` }` 126 ` _ => {` 127 ` // Unknown role, treat as user message` 128 ` messages.push(ChatMessage::User {` 129 ` content: UserMessageContent::Text(text),` 130 ` name: None,` 131 ` });` 132 ` }` 133 ` }` 134 ` }` 82 ` }` 5 linesAll 61 lines5 lines 5 linesAll 61 lines5 lines 196 ` }` 144 ` }` 197 ` }` 145 ` }` 198 ` }` 146 ` }` 199 ` }` 147 ` }` All 6 lines All 6 lines 206 ` // 3. Build ChatCompletionRequest` 154 ` // 3. Extract function tools from ResponseTools` 155 ` // Only function tools are extracted here (include_mcp: false).` 156 ` // MCP tools are merged later by the tool loop (see tool_loop.rs:prepare_chat_tools_and_choice)` 157 ` // before the chat pipeline, where tool_choice constraints are applied to ALL tools combined.` 158 ` let function_tools = extract_tools_from_response_tools(req.tools.as_deref(), false);` 159 ` let tools = if function_tools.is_empty() {` 160 ` None` 161 ` } else {` 162 ` Some(function_tools)` 163 ` };` 164 165 ` // 4. Build ChatCompletionRequest` 207 ` let is_streaming = req.stream.unwrap_or(false);` 166 ` let is_streaming = req.stream.unwrap_or(false);` 208 167 209 ` Ok(ChatCompletionRequest {` 168 ` Ok(ChatCompletionRequest {` 5 linesAll 19 lines5 lines 5 linesAll 19 lines5 lines 229 ` skip_special_tokens: true,` 188 ` skip_special_tokens: true,` 230 ` // Note: tools and tool_choice will be handled separately for MCP transformation` 189 ` tools,` 231 ` tools: None, // Will be set by caller if needed` 190 ` tool_choice: req.tool_choice.clone(),` 232 ` tool_choice: None, // Will be set by caller if needed` 233 ` ..Default::default()` 191 ` ..Default::default()` 234 ` })` 192 ` })` 235 `}` 193 `}` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 1 Move208–232 208 `/// Convert role and text to ChatMessage` 209 `fn role_to_chat_message(role: &str, text: String) -> ChatMessage {` 210 ` match role {` 211 ` "user" => ChatMessage::User {` 212 ` content: UserMessageContent::Text(text),` 213 ` name: None,` 214 ` },` 215 ` "assistant" => ChatMessage::Assistant {` 216 ` content: Some(text),` 217 ` name: None,` 218 ` tool_calls: None,` 219 ` reasoning_content: None,` 220 ` },` 221 ` "system" => ChatMessage::System {` 222 ` content: text,` 223 ` name: None,` 224 ` },` 225 ` _ => {` 226 ` // Unknown role, treat as user message` 227 ` ChatMessage::User {` 228 ` content: UserMessageContent::Text(text),` 229 ` name: None,` 230 ` }` 231 ` }` 232 ` }` 233 `}` 234 250 `/// Convert a ChatCompletionResponse to ResponsesResponse` 235 `/// Convert a ChatCompletionResponse to ResponsesResponse` All 7 lines All 7 lines 258 `pub fn chat_to_responses(` 243 `pub fn chat_to_responses(` 259 ` chat_resp: &ChatCompletionResponse,` 244 ` chat_resp: &ChatCompletionResponse,` 260 ` original_req: &ResponsesRequest,` 245 ` original_req: &ResponsesRequest,` 261 ` response_id_override: Option,` 246 ` response_id_override: Option,` 262 `) -> Result {` 247 `) -> Result {` 5 linesAll 76 lines5 lines 5 linesAll 76 lines5 lines 339 ` // Generate response` 324 ` // Generate response` 340 ` Ok(ResponsesResponse {` 325 ` Ok(ResponsesResponse {` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 355 ` temperature: original_req.temperature,` 340 ` temperature: original_req.temperature,` 356 ` text: None,` 341 ` text: None,` 357 ` tool_choice: "auto".to_string(), // TODO: Map from original request` 342 ` tool_choice: ToolChoice::serialize_to_string(&original_req.tool_choice),` 358 ` tools: original_req.tools.clone().unwrap_or_default(),` 343 ` tools: original_req.tools.clone().unwrap_or_default(),` All 6 lines All 6 lines 365 ` })` 350 ` })` 366 `}` 351 `}` 5 linesAll 63 lines5 lines 5 linesAll 63 lines5 lines tool\_loop.rssgl-router/src/routers/grpc/regular/responses +45−24 Mark as viewed All 8 lines All 8 lines 9 `use axum::{` 9 `use axum::{` 10 ` body::Body,` 10 ` body::Body,` 11 ` http::{header, StatusCode},` 11 ` http::{header, StatusCode},` 12 ` response::Response,` 12 ` response::Response,` 13 `};` 13 `};` 14 `use bytes::Bytes;` 14 `use bytes::Bytes;` 15 `use futures_util::StreamExt;` 15 `use futures_util::StreamExt;` 16 `use serde_json::json;` 16 `use serde_json::{json, Value};` 17 `use tokio::sync::mpsc;` 17 `use tokio::sync::mpsc;` All 5 lines All 5 lines 23 `use crate::{` 23 `use crate::{` 24 ` mcp::{self, McpManager},` 24 ` mcp::{self, McpManager},` 25 ` protocols::{` 25 ` protocols::{` 26 ` chat::{` 26 ` chat::{` 27 ` ChatChoice, ChatCompletionMessage, ChatCompletionResponse, ChatCompletionStreamResponse,` 27 ` ChatChoice, ChatCompletionMessage, ChatCompletionRequest, ChatCompletionResponse,` 28 ` ChatCompletionStreamResponse,` 28 ` },` 29 ` },` All 6 lines All 6 lines 35 ` },` 36 ` },` All 4 lines All 4 lines 40 `};` 41 `};` 5 linesAll 28 lines5 lines 5 linesAll 28 lines5 lines 70 `/// Merge function tools from request with MCP tools and set tool_choice based on iteration` 71 `fn prepare_chat_tools_and_choice(` 72 ` chat_request: &mut ChatCompletionRequest,` 73 ` mcp_chat_tools: &[Tool],` 74 ` iteration: usize,` 75 `) {` 76 ` // Merge function tools from request with MCP tools` 77 ` let mut all_tools = chat_request.tools.clone().unwrap_or_default();` 78 ` all_tools.extend(mcp_chat_tools.iter().cloned());` 79 ` chat_request.tools = Some(all_tools);` 80 81 ` // Set tool_choice based on iteration` 82 ` // - Iteration 0: Use user's tool_choice or default to auto` 83 ` // - Iteration 1+: Always use auto to avoid infinite loops` 84 ` chat_request.tool_choice = if iteration == 0 {` 85 ` chat_request` 86 ` .tool_choice` 87 ` .clone()` 88 ` .or(Some(ToolChoice::Value(ToolChoiceValue::Auto)))` 89 ` } else {` 90 ` Some(ToolChoice::Value(ToolChoiceValue::Auto))` 91 ` };` 92 `}` 93 5 linesAll 96 lines5 lines 5 linesAll 96 lines5 lines 165 `fn build_mcp_list_tools_item(mcp: &Arc, server_label: &str) -> ResponseOutputItem {` 190 `fn build_mcp_list_tools_item(mcp: &Arc, server_label: &str) -> ResponseOutputItem {` 166 ` let tools = mcp.list_tools();` 191 ` let tools = mcp.list_tools();` 167 ` let tools_info: Vec = tools` 192 ` let tools_info: Vec = tools` 168 ` .iter()` 193 ` .iter()` 169 ` .map(|t| {` 194 ` .map(|t| McpToolInfo {` 170 ` use serde_json::Value;` 195 ` name: t.name.to_string(),` 171 ` McpToolInfo {` 196 ` description: t.description.as_ref().map(|d| d.to_string()),` 172 ` name: t.name.to_string(),` 197 ` input_schema: Value::Object((*t.input_schema).clone()),` 173 ` description: t.description.as_ref().map(|d| d.to_string()),` 198 ` annotations: Some(json!({` 174 ` input_schema: Value::Object((*t.input_schema).clone()),` 199 ` "read_only": false` 175 ` annotations: Some(json!({` 200 ` })),` 176 ` "read_only": false` 177 ` })),` 178 ` }` 179 ` })` 201 ` })` 180 ` .collect();` 202 ` .collect();` All 6 lines All 6 lines 187 `}` 209 `}` 5 linesAll 29 lines5 lines 5 linesAll 29 lines5 lines 217 `pub(super) async fn execute_tool_loop(` 239 `pub(super) async fn execute_tool_loop(` All 6 lines All 6 lines 224 `) -> Result {` 246 `) -> Result {` 5 linesAll 23 lines5 lines 5 linesAll 23 lines5 lines 248 ` // Get MCP tools and convert to chat format (do this once before loop)` 270 ` // Get MCP tools and convert to chat format (do this once before loop)` 249 ` let mcp_tools = ctx.mcp_manager.list_tools();` 271 ` let mcp_tools = ctx.mcp_manager.list_tools();` 250 ` let chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);` 272 ` let mcp_chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);` 251 ` debug!("Converted {} MCP tools to chat format", chat_tools.len());` 273 ` debug!(` 274 ` "Converted {} MCP tools to chat format",` 275 ` mcp_chat_tools.len()` 276 ` );` 252 277 InformationalR278-301 253 ` loop {` 278 ` loop {` 254 ` // Convert to chat request` 279 ` // Convert to chat request` 255 ` let mut chat_request = conversions::responses_to_chat(¤t_request)` 280 ` let mut chat_request = conversions::responses_to_chat(¤t_request)` 256 ` .map_err(|e| error::bad_request(format!("Failed to convert request: {}", e)))?;` 281 ` .map_err(|e| error::bad_request(format!("Failed to convert request: {}", e)))?;` 257 282 258 ` // Add MCP tools to chat request so LLM knows about them` 283 ` // Prepare tools and tool_choice for this iteration` 259 ` chat_request.tools = Some(chat_tools.clone());` 284 ` prepare_chat_tools_and_choice(&mut chat_request, &mcp_chat_tools, state.iteration);` 260 ` chat_request.tool_choice = Some(ToolChoice::Value(ToolChoiceValue::Auto));` 5 linesAll 180 lines5 lines 5 linesAll 180 lines5 lines 441 ` }` 465 ` }` 442 `}` 466 `}` 5 linesAll 73 lines5 lines 5 linesAll 73 lines5 lines 516 `async fn execute_tool_loop_streaming_internal(` 540 `async fn execute_tool_loop_streaming_internal(` All 6 lines All 6 lines 523 `) -> Result<(), String> {` 547 `) -> Result<(), String> {` 5 linesAll 32 lines5 lines 5 linesAll 32 lines5 lines 556 ` // Get MCP tools and convert to chat format (do this once before loop)` 580 ` // Get MCP tools and convert to chat format (do this once before loop)` 557 ` let mcp_tools = ctx.mcp_manager.list_tools();` 581 ` let mcp_tools = ctx.mcp_manager.list_tools();` 558 ` let chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);` 582 ` let mcp_chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);` 559 ` debug!(` 583 ` debug!(` 560 ` "Streaming: Converted {} MCP tools to chat format",` 584 ` "Streaming: Converted {} MCP tools to chat format",` 561 ` chat_tools.len()` 585 ` mcp_chat_tools.len()` 562 ` );` 586 ` );` All 4 lines All 4 lines 1 Potential BugR591-662 567 ` loop {` 591 ` loop {` 5 linesAll 11 lines5 lines 5 linesAll 11 lines5 lines 579 ` if !mcp_list_tools_emitted {` 603 ` if !mcp_list_tools_emitted {` 580 ` let (output_index, item_id) =` 604 ` let (output_index, item_id) =` 581 ` emitter.allocate_output_index(OutputItemType::McpListTools);` 605 ` emitter.allocate_output_index(OutputItemType::McpListTools);` 582 606 583 ` // Build tools list for item structure` 607 ` // Build tools list for item structure` 584 ` let tool_items: Vec<_> = mcp_tools` 608 ` let tool_items: Vec<_> = mcp_tools` 585 ` .iter()` 609 ` .iter()` 586 ` .map(|t| {` 610 ` .map(|t| {` 587 ` use serde_json::Value;` 588 ` json!({` 611 ` json!({` 589 ` "name": t.name,` 612 ` "name": t.name,` 590 ` "description": t.description,` 613 ` "description": t.description,` 591 ` "input_schema": Value::Object((*t.input_schema).clone())` 614 ` "input_schema": Value::Object((*t.input_schema).clone())` 592 ` })` 615 ` })` 593 ` })` 616 ` })` 594 ` .collect();` 617 ` .collect();` 5 linesAll 37 lines5 lines 5 linesAll 37 lines5 lines 632 ` }` 655 ` }` 633 656 634 ` // Convert to chat request` 657 ` // Convert to chat request` 635 ` let mut chat_request = conversions::responses_to_chat(¤t_request)` 658 ` let mut chat_request = conversions::responses_to_chat(¤t_request)` 636 ` .map_err(|e| format!("Failed to convert request: {}", e))?;` 659 ` .map_err(|e| format!("Failed to convert request: {}", e))?;` 637 660 638 ` // Add MCP tools to chat request so LLM knows about them` 661 ` // Prepare tools and tool_choice for this iteration (same logic as non-streaming)` 639 ` chat_request.tools = Some(chat_tools.clone());` 662 ` prepare_chat_tools_and_choice(&mut chat_request, &mcp_chat_tools, state.iteration);` 640 ` chat_request.tool_choice = Some(ToolChoice::Value(ToolChoiceValue::Auto));` 5 linesAll 268 lines5 lines 5 linesAll 268 lines5 lines 909 ` }` 931 ` }` 910 932 911 ` Ok(())` 933 ` Ok(())` 912 `}` 934 `}` 913 935 914 `/// Convert MCP tools to Chat API tool format` 936 `/// Convert MCP tools to Chat API tool format` 915 `fn convert_mcp_tools_to_chat_tools(mcp_tools: &[mcp::Tool]) -> Vec {` 937 `fn convert_mcp_tools_to_chat_tools(mcp_tools: &[mcp::Tool]) -> Vec {` 916 ` use serde_json::Value;` 917 ` mcp_tools` 938 ` mcp_tools` 918 ` .iter()` 939 ` .iter()` 919 ` .map(|tool_info| Tool {` 940 ` .map(|tool_info| Tool {` All 8 lines All 8 lines 928 ` .collect()` 949 ` .collect()` 929 `}` 950 `}` 930 951 5 linesAll 141 lines5 lines 5 linesAll 141 lines5 lines # 5Shared Utilities 0 / 2 utils.rssgl-router/src/routers/grpc/common/responses +48−1 Mark as viewed 5 linesAll 10 lines5 lines 5 linesAll 10 lines5 lines 11 `use crate::{` 11 `use crate::{` 12 ` core::WorkerRegistry,` 12 ` core::WorkerRegistry,` 13 ` mcp::McpManager,` 13 ` mcp::McpManager,` 14 ` protocols::responses::{ResponseTool, ResponseToolType},` 14 ` protocols::{` 15 ` common::Tool,` 16 ` responses::{ResponseTool, ResponseToolType},` 17 ` },` 15 ` routers::{grpc::error, openai::mcp::ensure_request_mcp_client},` 18 ` routers::{grpc::error, openai::mcp::ensure_request_mcp_client},` 16 `};` 19 `};` 5 linesAll 62 lines5 lines 5 linesAll 62 lines5 lines 82 83 `/// Extract function tools (and optionally MCP tools) from ResponseTools` 84 `///` 85 `/// This utility consolidates the logic for extracting tools with schemas from ResponseTools.` 86 `/// It's used by both Harmony and Regular routers for different purposes:` 87 `///` 88 ``/// - **Harmony router**: Extracts both Function and MCP tools (with `include_mcp: true`)`` 89 `/// because MCP schemas are populated by convert_mcp_tools_to_response_tools() before the` 90 `/// pipeline runs. These tools are used to generate structural constraints in the` 91 `/// Harmony preparation stage.` 92 `///` 93 ``/// - **Regular router**: Extracts only Function tools (with `include_mcp: false`) during`` 94 `/// the initial conversion from ResponsesRequest to ChatCompletionRequest. MCP tools` 95 `/// are merged later by the tool loop before being sent to the chat pipeline, where` 96 `/// tool_choice constraints are generated for ALL tools (function + MCP combined).` 97 `pub fn extract_tools_from_response_tools(` 98 ` response_tools: Option<&[ResponseTool]>,` 99 ` include_mcp: bool,` 100 `) -> Vec {` 101 ` let Some(tools) = response_tools else {` 102 ` return Vec::new();` 103 ` };` 104 105 ` tools` 106 ` .iter()` 107 ` .filter_map(|rt| {` 108 ` match rt.r#type {` 109 ` // Function tools: Schema in request` 110 ` ResponseToolType::Function => rt.function.as_ref().map(|f| Tool {` 111 ` tool_type: "function".to_string(),` 112 ` function: f.clone(),` 113 ` }),` 114 ` // MCP tools: Schema populated by convert_mcp_tools_to_response_tools()` 115 ` // Only include if requested (Harmony case)` 116 ` ResponseToolType::Mcp if include_mcp => rt.function.as_ref().map(|f| Tool {` 117 ` tool_type: "function".to_string(),` 118 ` function: f.clone(),` 119 ` }),` 120 ` // Hosted tools: No schema available, skip` 121 ` _ => None,` 122 ` }` 123 ` })` 124 ` .collect()` 125 `}` utils.rssgl-router/src/routers/grpc +40−24 Mark as viewed 5 linesAll 10 lines5 lines 5 linesAll 10 lines5 lines 11 `use super::{error, ProcessedMessages};` 11 `use super::{error, ProcessedMessages};` 12 `pub use crate::tokenizer::StopSequenceDecoder;` 13 `use crate::{` 12 `use crate::{` 5 linesAll 14 lines5 lines 5 linesAll 14 lines5 lines 28 ` tokenizer::{` 27 ` tokenizer::{` 29 ` cache::CachedTokenizer,` 28 ` cache::CachedTokenizer,` 30 ` chat_template::{ChatTemplateContentFormat, ChatTemplateParams},` 29 ` chat_template::{ChatTemplateContentFormat, ChatTemplateParams},` 30 ` stop::StopSequenceDecoderBuilder,` 31 ` traits::Tokenizer,` 31 ` traits::Tokenizer,` 32 ` HuggingFaceTokenizer,` 32 ` HuggingFaceTokenizer, StopSequenceDecoder,` 33 ` },` 33 ` },` 34 ` tool_parser::{` 34 ` tool_parser::{` 35 ` ParserFactory as ToolParserFactory, PooledParser as ToolPooledParser, ToolParser,` 35 ` ParserFactory as ToolParserFactory, PooledParser as ToolPooledParser, ToolParser,` 36 ` },` 36 ` },` 37 `};` 37 `};` 5 linesAll 238 lines5 lines 5 linesAll 238 lines5 lines 276 `/// Filter tools based on tool_choice (shared by both routers)` 276 `/// Filter tools based on tool_choice (generic helper)` 277 `/// Returns a reference to the original body if no filtering needed,` 277 `///` 278 `/// otherwise returns a cloned and filtered body` 278 `/// Returns filtered tools if filtering is needed, otherwise returns None.` 279 `pub fn filter_tools_for_request(` 279 `/// Used by both Chat API and Responses API (Harmony) for constraint generation.` 280 ` body: &ChatCompletionRequest,` 280 `pub fn filter_tools_by_tool_choice(` 281 `) -> std::borrow::Cow<'_, ChatCompletionRequest> {` 281 ` tools: &[Tool],` 282 ` match &body.tool_choice {` 282 ` tool_choice: &Option,` 283 ` Some(ToolChoice::AllowedTools { tools: allowed, .. }) if body.tools.is_some() => {` 283 `) -> Option> {` 284 ` let mut filtered_body = body.clone();` 284 ` match tool_choice {` 1 InvestigateR285-293 285 ` let all_tools = filtered_body.tools.as_ref().unwrap();` 285 ` Some(ToolChoice::AllowedTools { tools: allowed, .. }) => {` 286 ` let allowed_names: std::collections::HashSet<&str> =` 286 ` let allowed_names: std::collections::HashSet<&str> =` 287 ` allowed.iter().map(|t| t.name.as_str()).collect();` 287 ` allowed.iter().filter_map(|t| t.function_name()).collect();` 288 ` let filtered_tools: Vec = all_tools` 288 ` let filtered: Vec = tools` 289 ` .iter()` 289 ` .iter()` 290 ` .filter(|t| allowed_names.contains(t.function.name.as_str()))` 290 ` .filter(|t| allowed_names.contains(t.function.name.as_str()))` 291 ` .cloned()` 291 ` .cloned()` 292 ` .collect();` 292 ` .collect();` 293 ` filtered_body.tools = Some(filtered_tools);` 293 ` Some(filtered)` 294 ` std::borrow::Cow::Owned(filtered_body)` 295 ` }` 294 ` }` 296 ` Some(ToolChoice::Function { function, .. }) if body.tools.is_some() => {` 295 ` Some(ToolChoice::Function { function, .. }) => {` 297 ` let mut filtered_body = body.clone();` 296 ` let filtered: Vec = tools` 298 ` let all_tools = filtered_body.tools.as_ref().unwrap();` 299 ` let filtered_tools: Vec = all_tools` 300 ` .iter()` 297 ` .iter()` 301 ` .filter(|t| t.function.name == function.name)` 298 ` .filter(|t| t.function.name == function.name)` 302 ` .cloned()` 299 ` .cloned()` 303 ` .collect();` 300 ` .collect();` 301 ` Some(filtered)` 302 ` }` 303 ` _ => None, // No filtering needed` 304 ` }` 305 `}` 306 307 `/// Filter ChatCompletionRequest by tool_choice` 308 `///` 309 `/// Returns a reference to the original request if no filtering needed,` 310 `/// otherwise returns a cloned request with filtered tools.` 311 `///` 312 `/// Note: Tool existence is validated earlier in ChatCompletionRequest::validate(),` 313 `/// so this function assumes tool_choice references valid tools.` 314 `pub fn filter_chat_request_by_tool_choice(` 315 ` body: &ChatCompletionRequest,` 316 `) -> std::borrow::Cow<'_, ChatCompletionRequest> {` 317 ` if let Some(tools) = &body.tools {` 318 ` if let Some(filtered_tools) = filter_tools_by_tool_choice(tools, &body.tool_choice) {` 319 ` let mut filtered_body = body.clone();` 304 ` filtered_body.tools = Some(filtered_tools);` 320 ` filtered_body.tools = Some(filtered_tools);` 305 ` std::borrow::Cow::Owned(filtered_body)` 321 ` return std::borrow::Cow::Owned(filtered_body);` 306 ` }` 322 ` }` 307 ` _ => std::borrow::Cow::Borrowed(body), // No filtering needed, use original` 308 ` }` 323 ` }` 324 325 ` // No filtering needed - return original request` 326 ` std::borrow::Cow::Borrowed(body)` 309 `}` 327 `}` 5 linesAll 124 lines5 lines 5 linesAll 124 lines5 lines 434 `pub fn create_stop_decoder(` 452 `pub fn create_stop_decoder(` 435 ` tokenizer: &Arc,` 453 ` tokenizer: &Arc,` 436 ` stop: Option<&StringOrArray>,` 454 ` stop: Option<&StringOrArray>,` 437 ` stop_token_ids: Option<&Vec>,` 455 ` stop_token_ids: Option<&Vec>,` 438 ` skip_special_tokens: bool,` 456 ` skip_special_tokens: bool,` 439 ` no_stop_trim: bool,` 457 ` no_stop_trim: bool,` 440 `) -> StopSequenceDecoder {` 458 `) -> StopSequenceDecoder {` 441 ` use crate::tokenizer::stop::StopSequenceDecoderBuilder;` 442 5 linesAll 32 lines5 lines 5 linesAll 32 lines5 lines 475 `}` 491 `}` 5 linesAll 644 lines5 lines 5 linesAll 644 lines5 lines # 6Miscellaneous 0 / 3 handlers.rssgl-router/src/routers/grpc/regular/responses +2−2 Mark as viewed 5 linesAll 53 lines5 lines 5 linesAll 53 lines5 lines 54 `use crate::{` 54 `use crate::{` All 4 lines All 4 lines 59 ` protocols::{` 59 ` protocols::{` 60 ` chat::{self, ChatCompletionStreamResponse},` 60 ` chat::{self, ChatCompletionStreamResponse},` 61 ` common,` 61 ` common::{self, ToolChoice},` 62 ` responses::{` 62 ` responses::{` 63 ` self, ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,` 63 ` self, ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,` 64 ` ResponseReasoningContent, ResponseStatus, ResponsesRequest, ResponsesResponse,` 64 ` ResponseReasoningContent, ResponseStatus, ResponsesRequest, ResponsesResponse,` 65 ` ResponsesUsage,` 65 ` ResponsesUsage,` 66 ` },` 66 ` },` 67 ` },` 67 ` },` All 9 lines All 9 lines 77 `};` 77 `};` 5 linesAll 422 lines5 lines 5 linesAll 422 lines5 lines 500 `impl StreamingResponseAccumulator {` 500 `impl StreamingResponseAccumulator {` 5 linesAll 87 lines5 lines 5 linesAll 87 lines5 lines 588 ` fn finalize(self) -> ResponsesResponse {` 588 ` fn finalize(self) -> ResponsesResponse {` 5 linesAll 54 lines5 lines 5 linesAll 54 lines5 lines 643 ` ResponsesResponse {` 643 ` ResponsesResponse {` 5 linesAll 15 lines5 lines 5 linesAll 15 lines5 lines 659 ` text: None,` 659 ` text: None,` 660 ` tool_choice: "auto".to_string(),` 660 ` tool_choice: ToolChoice::serialize_to_string(&self.original_request.tool_choice),` 661 ` tools: self.original_request.tools.clone().unwrap_or_default(),` 661 ` tools: self.original_request.tools.clone().unwrap_or_default(),` All 6 lines All 6 lines 668 ` }` 668 ` }` 669 ` }` 669 ` }` 670 `}` 670 `}` 5 linesAll 206 lines5 lines 5 linesAll 206 lines5 lines preparation.rssgl-router/src/routers/grpc/regular/stages/chat +1−1 Mark as viewed 5 linesAll 35 lines5 lines 5 linesAll 35 lines5 lines 36 `impl ChatPreparationStage {` 36 `impl ChatPreparationStage {` 37 ` async fn prepare_chat(` 37 ` async fn prepare_chat(` 38 ` &self,` 38 ` &self,` 39 ` ctx: &mut RequestContext,` 39 ` ctx: &mut RequestContext,` 40 ` request: &ChatCompletionRequest,` 40 ` request: &ChatCompletionRequest,` 41 ` ) -> Result<(), Response> {` 41 ` ) -> Result<(), Response> {` 42 ` // Step 1: Filter tools if needed` 42 ` // Step 1: Filter tools if needed` 43 ` let body_ref = utils::filter_tools_for_request(request);` 43 ` let body_ref = utils::filter_chat_request_by_tool_choice(request);` 5 linesAll 59 lines5 lines 5 linesAll 59 lines5 lines 103 ` }` 103 ` }` 104 `}` 104 `}` responses.rssgl-router/src/routers/grpc/harmony(2 of 2) +1−18 Mark as viewed 5 linesAll 50 lines5 lines 5 linesAll 50 lines5 lines 51 `use crate::{` 51 `use crate::{` 52 ` data_connector::{ResponseId, ResponseStorage},` 52 ` data_connector::{ResponseId, ResponseStorage},` 53 ` mcp::{self, McpManager},` 53 ` mcp::{self, McpManager},` 54 ` protocols::{` 54 ` protocols::{` 55 ` common::{Function, ToolCall, Usage},` 55 ` common::{Function, ToolCall, ToolChoice, ToolChoiceValue, Usage},` 56 ` responses::{` 56 ` responses::{` All 4 lines All 4 lines 61 ` },` 61 ` },` 62 ` },` 62 ` },` 5 linesAll 10 lines5 lines 5 linesAll 10 lines5 lines 73 `};` 73 `};` 5 linesAll 383 lines5 lines 5 linesAll 383 lines5 lines 457 `/// Serve Harmony Responses API with streaming (SSE)` 457 `/// Serve Harmony Responses API with streaming (SSE)` 5 linesAll 11 lines5 lines 5 linesAll 11 lines5 lines 469 ``/// - Handles errors with `response.failed``` 469 ``/// - Handles errors with `response.failed``` 470 `///` 471 `/// # Arguments` 472 `///` 473 ``/// * `ctx` - Harmony responses context with pipeline and dependencies`` 474 ``/// * `request` - Responses API request`` 475 `///` 476 `/// # Returns` 477 `///` 478 `/// SSE stream response with proper headers` 479 `pub async fn serve_harmony_responses_stream(` 470 `pub async fn serve_harmony_responses_stream(` 480 ` ctx: &HarmonyResponsesContext,` 471 ` ctx: &HarmonyResponsesContext,` 481 ` request: ResponsesRequest,` 472 ` request: ResponsesRequest,` 482 `) -> Response {` 473 `) -> Response {` 5 linesAll 54 lines5 lines 5 linesAll 54 lines5 lines 537 `}` 528 `}` 5 linesAll 560 lines5 lines 5 linesAll 560 lines5 lines 1098 `fn build_next_request_with_tools(` 1089 `fn build_next_request_with_tools(` All 5 lines All 5 lines 1104 `) -> Result> {` 1095 `) -> Result> {` 5 linesAll 92 lines5 lines 5 linesAll 92 lines5 lines 1192 ` Ok(request)` 1188 ` Ok(request)` 1193 `}` 1189 `}` 5 linesAll 19 lines5 lines 5 linesAll 19 lines5 lines 1213 `/// Convert MCP tools to Responses API tool format` 1209 `/// Convert MCP tools to Responses API tool format` 1214 `///` 1210 `///` 1215 `/// Converts MCP Tool entries (from rmcp SDK) to ResponseTool format so the model` 1211 `/// Converts MCP Tool entries (from rmcp SDK) to ResponseTool format so the model` 1216 `/// knows about available MCP tools when making tool calls.` 1212 `/// knows about available MCP tools when making tool calls.` 1217 `///` 1218 `/// # Arguments` 1219 `///` 1220 ``/// * `mcp_tools` - MCP tools from the MCP manager inventory (rmcp::model::Tool)`` 1221 `///` 1222 `/// # Returns` 1223 `///` 1224 `/// Vector of ResponseTool entries in MCP format` 1225 `pub fn convert_mcp_tools_to_response_tools(mcp_tools: &[mcp::Tool]) -> Vec {` 1213 `pub fn convert_mcp_tools_to_response_tools(mcp_tools: &[mcp::Tool]) -> Vec {` 5 linesAll 18 lines5 lines 5 linesAll 18 lines5 lines 1244 `}` 1232 `}` 5 linesAll 163 lines5 lines 5 linesAll 163 lines5 lines End of changesChat about this PR InfoChat ### 1 Potential bug Streaming tool loop never uses user's tool\_choice due to premature iteration increment Bugtool\_loop.rs:591 ### 5 Flags Tool filtering returns empty vec for AllowedTools with only non-function references Investigateutils.rs:285-293 Harmony preparation extracts both Function and MCP tools for constraint generation preparation.rs:131-140 MCP tool references bypass validation intentionally but may cause runtime errors responses.rs:776-793 Harmony router correctly switches tool\_choice to auto for subsequent iterations responses.rs:1183-1187 Non-streaming tool loop iteration tracking is correct tool\_loop.rs:278-301 ### ChecksPartial121/122 PR Test [pr-test-finishRequired3s](https://github.com/sgl-project/sglang/actions/runs/19090966620/job/54541467579) ### Reviewers3 KEkey4ng BYByronHsu SLslin1237 ### Assignees No assignees ### Labels4 enhancementhigh priorityrouterrun-ci ## Introducing Devin Review Devin Review is an intelligent PR review tool that helps you better understand your team's code before you ship it. 1 Intelligently organizes your diff Groups changes into logical sections with clear explanations → 2 Detects moved or copied code Keeps the diff clean by highlighting copy-pasted or moved code 3 Analyzes your PR Highlights potential bugs and flags areas that deserve extra careful review Ask questions or request edits in the chat. Ask Devin anything about this PR (Ctrl+I) Authorization Response Authorization Response