Home

Home

Wiki

Wiki

Review

Review

Docs

Docs

PR #12668

1ToolReference: Struct → Enum

3 files+122−34

common.rssgl-router/src/protocols

chat.rssgl-router/src/protocols

chat_completion.rssgl-router/tests/spec

Read explanation

The tool_choice parameter can reference different tool types. The old struct couldn't represent this:

rust

// BEFOREstruct ToolReference {    tool_type: String,  // "function"    name: String,}// AFTERenum ToolReference {    Function { name: String },    Mcp { server_label: String,           name: Option<String> },    FileSearch,    WebSearchPreview,    // ...}

Consequence → Chat API validation

Chat Completion API only supports function tools. Added validation to reject other types with a clear error message chat.rs:483-496

Consequence → Test updates

Tests updated to use new enum syntax.

2Responses API Validation

1 file+78

responses.rssgl-router/src/protocols

Read explanation

Added cross-field validation for tool_choice + tools in ResponsesRequest:

rust

fn validate_responses_cross_parameters(req) {    // If tool_choice references a function,    // verify it exists in tools array    if let Function { name } = tool_choice {        if !tools.contains(name) {            return Err("function not found")        }    }        // Validate AllowedTools mode is     // "auto" or "required"    if let AllowedTools { mode, .. } = tool_choice {        if mode != "auto" && mode != "required" {            return Err("invalid mode")        }    }}

3Harmony Router: tool_choice Constraint Generation

4 files+64−9

preparation.rssgl-router/src/routers/grpc/harmony/stages

request_building.rssgl-router/src/routers/grpc/harmony/stages

responses.rs(1/2)sgl-router/src/routers/grpc/harmony

sglang_scheduler.rssgl-router/src/grpc_client

Read explanation

The Harmony router uses "structural tags" to constrain model output. Added tool constraint generation for Responses API in the preparation stage:

preparation.rs:131-148

rust

// 1. Extract function + MCP toolslet tools = extract_tools_from_response_tools(    request.tools,     true  // include MCP);// 2. Filter by tool_choice if neededif let Some(filtered) =     filter_tools_by_tool_choice(&tools, &tool_choice) {    tools = filtered;}// 3. Generate Harmony structural tagslet constraints = generate_harmony_structural_tag(    &tools, &tool_choice)?;

The constraints are passed through request building request_building.rs:87 to the gRPC client, which now accepts tool_call_constraint for Responses API requests sglang_scheduler.rs:316

Infinite loop prevention: After receiving tool results, switch to "auto" so the model can choose to stop responses.rs:1183-1186

4Regular Router: tool_choice Pass-through

2 files+101−95

conversions.rssgl-router/src/routers/grpc/regular/responses

tool_loop.rssgl-router/src/routers/grpc/regular/responses

Read explanation

The Regular router converts Responses API requests to Chat API format. Now extracts function tools and passes tool_choice through:

conversions.rs:154-165

rust

// Extract ONLY function tools here// (MCP tools merged later by tool loop)let function_tools = extract_tools_from_response_tools(    req.tools,     false  // exclude MCP);// Pass through tool_choice unchangedOk(ChatCompletionRequest {    tools: Some(function_tools),    tool_choice: req.tool_choice.clone(),    // ...})

Tool loop: merging MCP tools

When MCP tools are present, the tool loop merges them with function tools before calling the chat pipeline. The iteration logic prevents infinite loops:

tool_loop.rs:70-92

rust

fn prepare_chat_tools_and_choice(    chat_request: &mut ChatCompletionRequest,    mcp_tools: &[Tool],    iteration: usize,) {    // Merge function + MCP tools    let mut all = chat_request.tools.unwrap_or_default();    all.extend(mcp_tools.iter().cloned());    chat_request.tools = Some(all);    // Iteration 0: use user's tool_choice    // Iteration 1+: force "auto" to avoid loops    chat_request.tool_choice = if iteration == 0 {        chat_request.tool_choice.clone()            .or(Some(ToolChoice::Value(Auto)))    } else {        Some(ToolChoice::Value(Auto))    };}

5Shared Utilities

2 files+88−25

utils.rssgl-router/src/routers/grpc/common/responses

utils.rssgl-router/src/routers/grpc

Read explanation

Extracted common logic to reduce duplication between routers:

extract_tools_from_response_tools() - Extracts function tools (and optionally MCP tools) from ResponseTools:

Harmony: include_mcp: true (MCP schemas pre-populated)
- Regular: include_mcp: false (MCP merged later in tool loop)

utils.rs:97-125

filter_tools_by_tool_choice() - Generic tool filtering extracted from the Chat-specific function. Now used by both Chat and Responses APIs:

utils.rs:276-305

rust

// Generic: returns filtered tools or Nonefn filter_tools_by_tool_choice(    tools: &[Tool],    tool_choice: &Option<ToolChoice>,) -> Option<Vec<Tool>>// Chat-specific: returns Cow<ChatCompletionRequest>fn filter_chat_request_by_tool_choice(    body: &ChatCompletionRequest,) -> Cow<'_, ChatCompletionRequest>

6Miscellaneous

3 files+4−21

handlers.rssgl-router/src/routers/grpc/regular/responses

preparation.rssgl-router/src/routers/grpc/regular/stages/chat

responses.rs(2/2)sgl-router/src/routers/grpc/harmony

Read explanation

Import changes and doc comment cleanup
- ToolChoice::serialize_to_string() helper for response serialization common.rs:186-196
- Function rename: filter_tools_for_request → filter_chat_request_by_tool_choice

Merged

sgl-project/sglang #12668

[router][grpc] Implement tool_choice support for Responses API

CACatherineSue

main chang/responses-fix

14 files

+457−184

DescriptionDiscussion2Commits3

Devin's AI analysis

This PR adds tool_choice support to the /v1/responses endpoint, allowing users to control tool calling behavior:

tool_choice: "none"      → prevent tool callstool_choice: "auto"      → let model decidetool_choice: "required"  → force tool calltool_choice: {function}  → force specific tooltool_choice: {allowed}   → restrict to subset

Key changes:

ToolReference becomes a tagged enum to properly represent different tool types (functions, MCP, hosted tools). This cascades to Chat API validation and tests.
Harmony router generates structural tag constraints from tool_choice in the preparation stage preparation.rs:131-148
Regular router passes tool_choice through to the chat pipeline. For MCP requests, tools are merged before constraint generation tool_loop.rs:70-92
Infinite loop prevention: When tool_choice: "required", the model would loop forever. After the first iteration, both routers switch to "auto"responses.rs:1183-1186

Motivation

The Responses API (/v1/responses) did not support the tool_choice parameter, which is a critical feature for controlling tool calling behavior according to OpenAI's Responses API specification. This limitation prevented users from:

Forcing the model to call specific tools (tool_choice: {type: "function", function: {name: "..."}})
Requiring at least one tool call (tool_choice: "required")
Restricting tool calls to a subset of available tools (tool_choice: {type: "allowed_tools", ...})
Preventing tool calls entirely (tool_choice: "none")

This PR implements full tool_choice support for the Responses API in both Harmony and Regular routers, following the same pattern as the existing Chat Completion API implementation.

Modifications

1. Harmony Router - Responses API tool_choice Support

File: src/routers/grpc/harmony/stages/preparation.rs

Added tool_choice constraint generation for Responses API requests
Extracts both Function and MCP tools from ResponseTools (schemas populated before pipeline)
Implements AllowedTools filtering to restrict tools based on user's tool_choice
Generates Harmony structural tags using triggered_tags format for tool constraints
Supports all tool_choice modes: none, auto, required, specific function, allowed_tools

2. Regular Router - Responses API tool_choice Support

Files:

src/routers/grpc/regular/responses/conversions.rs
src/routers/grpc/regular/responses/tool_loop.rs

Flow:

conversions.rs: Extracts function tools and passes through tool_choice unchanged
Without MCP: Function tools → chat pipeline → tool_choice constraints applied
With MCP: Tool loop merges function + MCP tools → chat pipeline → constraints applied to ALL tools
Loop iteration logic: Iteration 0 uses user's tool_choice, iteration 1+ uses "auto" to prevent infinite loops

3. ToolReference Protocol Enhancement

File: src/protocols/common.rs

Converted ToolReference from a simple struct to a tagged enum to properly support different tool types:

rust

// Before (struct)pub struct ToolReference {    pub tool_type: String,    pub name: String,}// After (enum)pub enum ToolReference {    Function { name: String },    Mcp { server_label: String, name: Option<String> },    FileSearch,    WebSearchPreview,    ComputerUsePreview,    CodeInterpreter,    ImageGeneration,}

Benefits:

Type-safe representation of different tool types
Supports MCP and hosted tools (not just functions)
Helper methods: identifier(), function_name()

4. Chat API Validation

File: src/protocols/chat.rs

Added validation that Chat Completion API ONLY accepts Function type ToolReference in tool_choice. Rejects MCP and hosted tools with clear error messages, enforcing the API contract.

5. Code Deduplication

File: src/routers/grpc/common/responses/utils.rs

Created shared utility function extract_tools_from_response_tools() to eliminate duplication:

Used by both Harmony preparation stage and Regular conversions
include_mcp parameter controls whether to extract MCP tools
Comprehensive documentation explains usage differences between routers

6. Helper Function for Chat Requests

File: src/routers/grpc/regular/responses/tool_loop.rs

Created prepare_chat_tools_and_choice() helper to:

Merge function tools from request with MCP tools
Set tool_choice based on iteration (user's choice on iteration 0, "auto" on iteration 1+)
Reduces duplication in tool loop

7. Bug Fixes

Fixed unnecessary pub use: Changed pub use crate::tokenizer::StopSequenceDecoder to regular use in utils.rs
Updated test cases: Fixed 6 test functions in tests/spec/chat_completion.rs to use new ToolReference enum syntax

8. Documentation

Added comprehensive comments explaining:

Tool extraction flow in conversions.rs (why MCP path "wastes" initial extraction)
Differences between Harmony and Regular router tool handling
Tool loop iteration behavior with tool_choice

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

Filter

gemini-code-assist177d

[!WARNING] You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

641 lines left

1ToolReference: Struct → Enum

0 / 3

common.rssgl-router/src/protocols

+78−4

Mark as viewed

5 linesAll 161 lines5 lines

162

/// Tool choice for both Chat Completion and Responses APIs

162

/// Tool choice for both Chat Completion and Responses APIs

163

#[derive(Debug, Clone, Deserialize, Serialize)]

163

#[derive(Debug, Clone, Deserialize, Serialize)]

164

#[serde(untagged)]

164

#[serde(untagged)]

165

pub enum ToolChoice {

165

pub enum ToolChoice {

5 linesAll 12 lines5 lines

178

}

178

}

179

180

impl Default for ToolChoice {

180

impl Default for ToolChoice {

181

fn default() -> Self {

181

fn default() -> Self {

182

Self::Value(ToolChoiceValue::Auto)

182

Self::Value(ToolChoiceValue::Auto)

183

}

183

}

184

}

184

}

185

186

impl ToolChoice {

187

/// Serialize tool_choice to string for ResponsesResponse

188

///

189

/// Returns the JSON-serialized tool_choice or "auto" as default

190

pub fn serialize_to_string(tool_choice: &Option<ToolChoice>) -> String {

191

tool_choice

192

.as_ref()

193

.map(|tc| serde_json::to_string(tc).unwrap_or_else(|_| "auto".to_string()))

194

.unwrap_or_else(|| "auto".to_string())

195

}

196

}

197

All 6 lines

192

/// Tool reference for ToolChoice::AllowedTools

204

/// Tool reference for ToolChoice::AllowedTools

205

///

206

/// Represents a reference to a specific tool in the allowed_tools array.

207

/// Different tool types have different required fields.

193

#[derive(Debug, Clone, Deserialize, Serialize)]

208

#[derive(Debug, Clone, Deserialize, Serialize)]

194

pub struct ToolReference {

209

#[serde(tag = "type")]

195

#[serde(rename = "type")]

210

#[serde(rename_all = "snake_case")]

196

pub tool_type: String, // "function"

211

pub enum ToolReference {

197

pub name: String,

212

/// Reference to a function tool

213

#[serde(rename = "function")]

214

Function { name: String },

215

216

/// Reference to an MCP tool

217

#[serde(rename = "mcp")]

218

Mcp {

219

server_label: String,

220

#[serde(skip_serializing_if = "Option::is_none")]

221

name: Option<String>,

222

},

223

224

/// File search hosted tool

225

#[serde(rename = "file_search")]

226

FileSearch,

227

228

/// Web search preview hosted tool

229

#[serde(rename = "web_search_preview")]

230

WebSearchPreview,

231

232

/// Computer use preview hosted tool

233

#[serde(rename = "computer_use_preview")]

234

ComputerUsePreview,

235

236

/// Code interpreter hosted tool

237

#[serde(rename = "code_interpreter")]

238

CodeInterpreter,

239

240

/// Image generation hosted tool

241

#[serde(rename = "image_generation")]

242

ImageGeneration,

243

}

244

245

impl ToolReference {

246

/// Get a unique identifier for this tool reference

247

pub fn identifier(&self) -> String {

248

match self {

249

ToolReference::Function { name } => format!("function:{}", name),

250

ToolReference::Mcp { server_label, name } => {

251

if let Some(n) = name {

252

format!("mcp:{}:{}", server_label, n)

253

} else {

254

format!("mcp:{}", server_label)

255

}

256

}

257

ToolReference::FileSearch => "file_search".to_string(),

258

ToolReference::WebSearchPreview => "web_search_preview".to_string(),

259

ToolReference::ComputerUsePreview => "computer_use_preview".to_string(),

260

ToolReference::CodeInterpreter => "code_interpreter".to_string(),

261

ToolReference::ImageGeneration => "image_generation".to_string(),

262

}

263

}

264

265

/// Get the tool name if this is a function tool

266

pub fn function_name(&self) -> Option<&str> {

267

match self {

268

ToolReference::Function { name } => Some(name.as_str()),

269

_ => None,

270

}

271

}

198

}

272

}

5 linesAll 148 lines5 lines

chat.rssgl-router/src/protocols

+36−14

Mark as viewed

5 linesAll 337 lines5 lines

338

fn validate_chat_cross_parameters(

338

fn validate_chat_cross_parameters(

339

req: &ChatCompletionRequest,

339

req: &ChatCompletionRequest,

340

) -> Result<(), validator::ValidationError> {

340

) -> Result<(), validator::ValidationError> {

5 linesAll 68 lines5 lines

409

if let Some(ref tool_choice) = req.tool_choice {

409

if let Some(ref tool_choice) = req.tool_choice {

5 linesAll 12 lines5 lines

422

if has_tools {

422

if has_tools {

423

let tools = req.tools.as_ref().unwrap();

423

let tools = req.tools.as_ref().unwrap();

424

425

match tool_choice {

425

match tool_choice {

5 linesAll 19 lines5 lines

445

ToolChoice::AllowedTools {

445

ToolChoice::AllowedTools {

446

mode,

446

mode,

447

tools: allowed_tools,

447

tools: allowed_tools,

448

..

448

..

449

} => {

449

} => {

5 linesAll 10 lines5 lines

460

// Validate that all referenced tool names exist in tools

460

// Validate that all ToolReferences are Function type (Chat API only supports function tools)

461

for tool_ref in allowed_tools {

461

for tool_ref in allowed_tools {

462

let tool_exists = tools.iter().any(|tool| {

462

match tool_ref {

463

tool.tool_type == tool_ref.tool_type

463

ToolReference::Function { name } => {

464

&& tool.function.name == tool_ref.name

464

// Validate that the function exists in tools array

465

});

465

let tool_exists = tools.iter().any(|tool| {

466

tool.tool_type == "function" && tool.function.name == *name

467

if !tool_exists {

467

});

468

let mut e =

468

469

validator::ValidationError::new("tool_choice_tool_not_found");

469

if !tool_exists {

470

e.message = Some(format!(

470

let mut e = validator::ValidationError::new(

471

"Invalid value for 'tool_choice.tools': tool '{}' not found in 'tools'.",

471

"tool_choice_tool_not_found",

472

tool_ref.name

472

);

473

).into());

473

e.message = Some(

474

return Err(e);

474

format!(

475

"Invalid value for 'tool_choice.tools': tool '{}' not found in 'tools'.",

476

name

477

)

478

.into(),

479

);

480

return Err(e);

481

}

482

}

483

_ => {

484

// Chat Completion API only supports function tools in tool_choice

485

let mut e = validator::ValidationError::new(

486

"tool_choice_invalid_tool_type",

487

);

488

e.message = Some(

489

format!(

490

"Invalid value for 'tool_choice.tools': Chat Completion API only supports function tools, got '{}'.",

491

tool_ref.identifier()

492

)

493

.into(),

494

);

495

return Err(e);

496

}

475

}

497

}

476

}

498

}

477

}

499

}

478

_ => {}

500

_ => {}

479

}

501

}

480

}

502

}

481

}

503

}

482

504

483

Ok(())

505

Ok(())

484

}

506

}

5 linesAll 197 lines5 lines

chat_completion.rssgl-router/tests/spec

+8−16

Mark as viewed

5 linesAll 332 lines5 lines

333

#[test]

333

#[test]

334

fn test_tool_choice_allowed_tools_invalid_mode() {

334

fn test_tool_choice_allowed_tools_invalid_mode() {

335

let req = ChatCompletionRequest {

335

let req = ChatCompletionRequest {

5 linesAll 14 lines5 lines

350

tool_choice: Some(ToolChoice::AllowedTools {

350

tool_choice: Some(ToolChoice::AllowedTools {

351

mode: "invalid_mode".to_string(),

351

mode: "invalid_mode".to_string(),

352

tools: vec![ToolReference {

352

tools: vec![ToolReference::Function {

353

tool_type: "function".to_string(),

354

name: "get_weather".to_string(),

353

name: "get_weather".to_string(),

355

}],

354

}],

356

tool_type: "function".to_string(),

355

tool_type: "function".to_string(),

357

}),

356

}),

358

..Default::default()

357

..Default::default()

359

};

358

};

All 9 lines

369

}

368

}

370

369

371

#[test]

370

#[test]

372

fn test_tool_choice_allowed_tools_valid_mode_auto() {

371

fn test_tool_choice_allowed_tools_valid_mode_auto() {

373

let req = ChatCompletionRequest {

372

let req = ChatCompletionRequest {

5 linesAll 14 lines5 lines

388

tool_choice: Some(ToolChoice::AllowedTools {

387

tool_choice: Some(ToolChoice::AllowedTools {

389

mode: "auto".to_string(),

388

mode: "auto".to_string(),

390

tools: vec![ToolReference {

389

tools: vec![ToolReference::Function {

391

tool_type: "function".to_string(),

392

name: "get_weather".to_string(),

390

name: "get_weather".to_string(),

393

}],

391

}],

394

tool_type: "function".to_string(),

392

tool_type: "function".to_string(),

395

}),

393

}),

396

..Default::default()

394

..Default::default()

397

};

395

};

398

396

399

let result = req.validate();

397

let result = req.validate();

400

assert!(result.is_ok(), "Should accept 'auto' mode");

398

assert!(result.is_ok(), "Should accept 'auto' mode");

401

}

399

}

402

400

403

#[test]

401

#[test]

404

fn test_tool_choice_allowed_tools_valid_mode_required() {

402

fn test_tool_choice_allowed_tools_valid_mode_required() {

405

let req = ChatCompletionRequest {

403

let req = ChatCompletionRequest {

5 linesAll 14 lines5 lines

420

tool_choice: Some(ToolChoice::AllowedTools {

418

tool_choice: Some(ToolChoice::AllowedTools {

421

mode: "required".to_string(),

419

mode: "required".to_string(),

422

tools: vec![ToolReference {

420

tools: vec![ToolReference::Function {

423

tool_type: "function".to_string(),

424

name: "get_weather".to_string(),

421

name: "get_weather".to_string(),

425

}],

422

}],

426

tool_type: "function".to_string(),

423

tool_type: "function".to_string(),

427

}),

424

}),

428

..Default::default()

425

..Default::default()

429

};

426

};

430

427

431

let result = req.validate();

428

let result = req.validate();

432

assert!(result.is_ok(), "Should accept 'required' mode");

429

assert!(result.is_ok(), "Should accept 'required' mode");

433

}

430

}

434

431

435

#[test]

432

#[test]

436

fn test_tool_choice_allowed_tools_tool_not_found() {

433

fn test_tool_choice_allowed_tools_tool_not_found() {

437

let req = ChatCompletionRequest {

434

let req = ChatCompletionRequest {

5 linesAll 14 lines5 lines

452

tool_choice: Some(ToolChoice::AllowedTools {

449

tool_choice: Some(ToolChoice::AllowedTools {

453

mode: "auto".to_string(),

450

mode: "auto".to_string(),

454

tools: vec![ToolReference {

451

tools: vec![ToolReference::Function {

455

tool_type: "function".to_string(),

456

name: "nonexistent_tool".to_string(),

452

name: "nonexistent_tool".to_string(),

457

}],

453

}],

458

tool_type: "function".to_string(),

454

tool_type: "function".to_string(),

459

}),

455

}),

460

..Default::default()

456

..Default::default()

461

};

457

};

All 9 lines

471

}

467

}

472

468

473

#[test]

469

#[test]

474

fn test_tool_choice_allowed_tools_multiple_tools_valid() {

470

fn test_tool_choice_allowed_tools_multiple_tools_valid() {

475

let req = ChatCompletionRequest {

471

let req = ChatCompletionRequest {

5 linesAll 25 lines5 lines

501

tool_choice: Some(ToolChoice::AllowedTools {

497

tool_choice: Some(ToolChoice::AllowedTools {

502

mode: "auto".to_string(),

498

mode: "auto".to_string(),

503

tools: vec![

499

tools: vec![

504

ToolReference {

500

ToolReference::Function {

505

tool_type: "function".to_string(),

506

name: "get_weather".to_string(),

501

name: "get_weather".to_string(),

507

},

502

},

508

ToolReference {

503

ToolReference::Function {

509

tool_type: "function".to_string(),

510

name: "get_time".to_string(),

504

name: "get_time".to_string(),

511

},

505

},

512

],

506

],

513

tool_type: "function".to_string(),

507

tool_type: "function".to_string(),

514

}),

508

}),

515

..Default::default()

509

..Default::default()

516

};

510

};

517

511

518

let result = req.validate();

512

let result = req.validate();

519

assert!(result.is_ok(), "Should accept all valid tool references");

513

assert!(result.is_ok(), "Should accept all valid tool references");

520

}

514

}

521

515

522

#[test]

516

#[test]

523

fn test_tool_choice_allowed_tools_one_invalid_among_valid() {

517

fn test_tool_choice_allowed_tools_one_invalid_among_valid() {

524

let req = ChatCompletionRequest {

518

let req = ChatCompletionRequest {

5 linesAll 25 lines5 lines

550

tool_choice: Some(ToolChoice::AllowedTools {

544

tool_choice: Some(ToolChoice::AllowedTools {

551

mode: "auto".to_string(),

545

mode: "auto".to_string(),

552

tools: vec![

546

tools: vec![

553

ToolReference {

547

ToolReference::Function {

554

tool_type: "function".to_string(),

555

name: "get_weather".to_string(),

548

name: "get_weather".to_string(),

556

},

549

},

557

ToolReference {

550

ToolReference::Function {

558

tool_type: "function".to_string(),

559

name: "nonexistent_tool".to_string(),

551

name: "nonexistent_tool".to_string(),

560

},

552

},

561

],

553

],

562

tool_type: "function".to_string(),

554

tool_type: "function".to_string(),

563

}),

555

}),

564

..Default::default()

556

..Default::default()

565

};

557

};

5 linesAll 12 lines5 lines

578

}

570

}

2Responses API Validation

0 / 1

responses.rssgl-router/src/protocols

+78

Mark as viewed

5 linesAll 448 lines5 lines

449

#[derive(Debug, Clone, Deserialize, Serialize, Validate)]

449

#[derive(Debug, Clone, Deserialize, Serialize, Validate)]

450

#[validate(schema(function = "validate_responses_cross_parameters"))]

450

pub struct ResponsesRequest {

451

pub struct ResponsesRequest {

5 linesAll 98 lines5 lines

549

550

5 linesAll 10 lines5 lines

560

pub stop: Option<StringOrArray>,

561

pub stop: Option<StringOrArray>,

561

562

5 linesAll 162 lines5 lines

Copy + paste725–800

725

/// Schema-level validation for cross-field dependencies

726

fn validate_responses_cross_parameters(

727

request: &ResponsesRequest,

728

) -> Result<(), validator::ValidationError> {

729

use super::common::{ToolChoice, ToolReference};

730

731

// Only validate if both tools and tool_choice are present

732

if let (Some(tools), Some(tool_choice)) = (&request.tools, &request.tool_choice) {

733

// Extract function tool names from ResponseTools

734

let function_tool_names: Vec<&str> = tools

735

.iter()

736

.filter_map(|t| match t.r#type {

737

ResponseToolType::Function => t.function.as_ref().map(|f| f.name.as_str()),

738

_ => None,

739

})

740

.collect();

741

742

match tool_choice {

743

ToolChoice::Function { function, .. } => {

744

// Validate the specific function exists

745

if !function_tool_names.contains(&function.name.as_str()) {

746

let mut e = validator::ValidationError::new("tool_choice_function_not_found");

747

e.message = Some(

748

format!(

749

"Invalid value for 'tool_choice': function '{}' not found in 'tools'.",

750

function.name

751

)

752

.into(),

753

);

754

return Err(e);

755

}

756

}

757

ToolChoice::AllowedTools {

758

mode,

759

tools: allowed_tools,

760

..

761

} => {

762

// Validate mode is "auto" or "required"

763

if mode != "auto" && mode != "required" {

764

let mut e = validator::ValidationError::new("tool_choice_invalid_mode");

765

e.message = Some(

766

format!(

767

"Invalid value for 'tool_choice.mode': must be 'auto' or 'required', got '{}'.",

768

mode

769

)

770

.into(),

771

);

772

return Err(e);

773

}

774

775

// Validate that all function tool references exist

InformationalR776-793

776

for tool_ref in allowed_tools {

777

if let ToolReference::Function { name } = tool_ref {

778

if !function_tool_names.contains(&name.as_str()) {

779

let mut e =

780

validator::ValidationError::new("tool_choice_tool_not_found");

781

e.message = Some(

782

format!(

783

"Invalid value for 'tool_choice.tools': tool '{}' not found in 'tools'.",

784

name

785

)

786

.into(),

787

);

788

return Err(e);

789

}

790

}

791

// Note: MCP and hosted tools don't need existence validation here

792

// as they are resolved dynamically at runtime

793

}

794

}

795

_ => {}

796

}

797

}

798

799

Ok(())

800

}

801

5 linesAll 236 lines5 lines

3Harmony Router: tool_choice Constraint Generation

0 / 4

preparation.rssgl-router/src/routers/grpc/harmony/stages

+24−5

Mark as viewed

All 7 lines

use crate::{

All 5 lines

routers::grpc::{

common::stages::PipelineStage,

common::{responses::utils::extract_tools_from_response_tools, stages::PipelineStage},

context::{PreparationOutput, RequestContext, RequestType},

error, utils,

},

};

All 5 lines

pub struct HarmonyPreparationStage {

builder: HarmonyBuilder,

}

5 linesAll 44 lines5 lines

impl HarmonyPreparationStage {

/// Prepare a chat completion request using Harmony encoding

async fn prepare_chat(

&self,

ctx: &mut RequestContext,

request: &ChatCompletionRequest,

) -> Result<Option<Response>, Response> {

All 6 lines

// Step 1: Filter tools if needed

let body_ref = utils::filter_tools_for_request(request);

let body_ref = utils::filter_chat_request_by_tool_choice(request);

5 linesAll 28 lines5 lines

117

});

117

});

118

119

Ok(None)

119

Ok(None)

120

}

120

}

121

122

/// Prepare a responses API request using Harmony encoding

122

/// Prepare a responses API request using Harmony encoding

123

///

123

///

124

/// For responses API, we build from conversation history using the same Harmony

124

/// For responses API, we build from conversation history using the same Harmony

125

/// encoding that the builder provides. This handles the MCP loop integration.

125

/// encoding that the builder provides. This handles the MCP loop integration.

126

pub async fn prepare_responses(

126

pub async fn prepare_responses(

127

&self,

127

&self,

128

ctx: &mut RequestContext,

128

ctx: &mut RequestContext,

129

request: &ResponsesRequest,

129

request: &ResponsesRequest,

130

) -> Result<Option<Response>, Response> {

130

) -> Result<Option<Response>, Response> {

InformationalR131-140

131

// Build via Harmony from responses API request

131

// Step 1: Extract function and MCP tools with schemas from ResponseTools

132

let mut function_tools = extract_tools_from_response_tools(request.tools.as_deref(), true);

133

134

// Step 2: Filter tools based on tool_choice (AllowedTools or Function)

135

// Note: Tool existence is already validated in ResponsesRequest::validate()

136

if let Some(filtered) =

137

utils::filter_tools_by_tool_choice(&function_tools, &request.tool_choice)

138

{

139

function_tools = filtered;

140

}

141

142

// Step 3: Generate Harmony structural tags from filtered tools

143

let tool_constraints = if !function_tools.is_empty() {

144

Self::generate_harmony_structural_tag(&function_tools, &request.tool_choice)

145

.map_err(|e| *e)?

146

} else {

147

None

148

};

149

150

// Step 3: Build via Harmony from responses API request

132

let build_output = self

151

let build_output = self

133

.builder

152

.builder

134

.build_from_responses(request)

153

.build_from_responses(request)

135

.map_err(|e| error::bad_request(format!("Harmony build failed: {}", e)))?;

154

.map_err(|e| error::bad_request(format!("Harmony build failed: {}", e)))?;

136

155

137

// Store results in preparation output

156

// Step 4: Store results with tool_constraints

138

ctx.state.preparation = Some(PreparationOutput {

157

ctx.state.preparation = Some(PreparationOutput {

139

original_text: None,

158

original_text: None,

140

token_ids: build_output.input_ids,

159

token_ids: build_output.input_ids,

141

processed_messages: None,

160

processed_messages: None,

142

tool_constraints: None,

161

tool_constraints,

143

filtered_request: None,

162

filtered_request: None,

144

harmony_mode: true,

163

harmony_mode: true,

145

selection_text: Some(build_output.selection_text),

164

selection_text: Some(build_output.selection_text),

146

harmony_messages: Some(build_output.harmony_messages),

165

harmony_messages: Some(build_output.harmony_messages),

147

harmony_stop_ids: Some(build_output.stop_token_ids),

166

harmony_stop_ids: Some(build_output.stop_token_ids),

148

});

167

});

149

168

150

Ok(None)

169

Ok(None)

151

}

170

}

5 linesAll 93 lines5 lines

245

}

264

}

request_building.rssgl-router/src/routers/grpc/harmony/stages

Mark as viewed

5 linesAll 17 lines5 lines

pub struct HarmonyRequestBuildingStage {

inject_pd_metadata: bool,

}

All 9 lines

impl PipelineStage for HarmonyRequestBuildingStage {

async fn execute(&self, ctx: &mut RequestContext) -> Result<Option<Response>, Response> {

5 linesAll 32 lines5 lines

let mut proto_request = match &ctx.input.request_type {

5 linesAll 15 lines5 lines

RequestType::Responses(request) => builder_client

.build_generate_request_from_responses(

request_id,

request.as_ref(),

placeholder_processed_text,

prep.token_ids.clone(),

prep.harmony_stop_ids.clone(),

prep.tool_constraints.clone(),

)

.map_err(|e| error::bad_request(format!("Invalid request parameters: {}", e)))?,

_ => unreachable!(),

};

5 linesAll 23 lines5 lines

114

}

115

}

All 4 lines

119

}

120

}

responses.rssgl-router/src/routers/grpc/harmony(1 of 2)

Mark as viewed

5 linesAll 1097 lines5 lines

1098

fn build_next_request_with_tools(

1089

fn build_next_request_with_tools(

1099

mut request: ResponsesRequest,

1090

mut request: ResponsesRequest,

1100

tool_calls: Vec<ToolCall>,

1091

tool_calls: Vec<ToolCall>,

1101

tool_results: Vec<ToolResult>,

1092

tool_results: Vec<ToolResult>,

1102

analysis: Option<String>,

1093

analysis: Option<String>,

1103

partial_text: String,

1094

partial_text: String,

1104

) -> Result<ResponsesRequest, Box<Response>> {

1095

) -> Result<ResponsesRequest, Box<Response>> {

5 linesAll 83 lines5 lines

1188

1179

1189

// Update request with new items

1180

// Update request with new items

1190

request.input = ResponseInput::Items(items);

1181

request.input = ResponseInput::Items(items);

1191

1182

InformationalR1183-1187

1183

// Switch tool_choice to "auto" for subsequent iterations

1184

// This prevents infinite loops when original tool_choice was "required" or specific function

1185

// After receiving tool results, the model should be free to decide whether to call more tools or finish

1186

request.tool_choice = Some(ToolChoice::Value(ToolChoiceValue::Auto));

1187

1192

Ok(request)

1188

Ok(request)

1193

}

1189

}

5 linesAll 214 lines5 lines

sglang_scheduler.rssgl-router/src/grpc_client

+34−4

Mark as viewed

5 linesAll 124 lines5 lines

125

impl SglangSchedulerClient {

125

impl SglangSchedulerClient {

5 linesAll 179 lines5 lines

305

/// Build a GenerateRequest from ResponsesRequest (OpenAI Responses API)

305

/// Build a GenerateRequest from ResponsesRequest (OpenAI Responses API)

306

///

307

/// NOTE: This is used by the Harmony router only. The Regular router uses

308

/// responses_to_chat() conversion and goes through the chat pipeline.

306

pub fn build_generate_request_from_responses(

309

pub fn build_generate_request_from_responses(

307

&self,

310

&self,

308

request_id: String,

311

request_id: String,

309

body: &ResponsesRequest,

312

body: &ResponsesRequest,

310

processed_text: String,

313

processed_text: String,

311

token_ids: Vec<u32>,

314

token_ids: Vec<u32>,

312

harmony_stop_ids: Option<Vec<u32>>,

315

harmony_stop_ids: Option<Vec<u32>>,

316

tool_call_constraint: Option<(String, String)>,

313

) -> Result<proto::GenerateRequest, String> {

317

) -> Result<proto::GenerateRequest, String> {

314

// Build sampling params from ResponsesRequest

318

// Build sampling params from ResponsesRequest

315

let mut sampling_params = self.build_grpc_sampling_params_from_responses(body)?;

319

let mut sampling_params =

320

self.build_grpc_sampling_params_from_responses(body, tool_call_constraint)?;

5 linesAll 22 lines5 lines

338

Ok(grpc_request)

343

Ok(grpc_request)

339

}

344

}

5 linesAll 101 lines5 lines

441

fn build_grpc_sampling_params_from_responses(

446

fn build_grpc_sampling_params_from_responses(

442

&self,

447

&self,

443

request: &ResponsesRequest,

448

request: &ResponsesRequest,

449

tool_call_constraint: Option<(String, String)>,

444

) -> Result<proto::SamplingParams, String> {

450

) -> Result<proto::SamplingParams, String> {

445

// ResponsesRequest doesn't have stop sequences in the same way

451

// ResponsesRequest doesn't have stop sequences in the same way

446

// Tools are handled externally by MCP loop, not via constraints

452

// For Harmony router: Tools are handled via structural_tag constraints

447

453

448

let max_new_tokens = request.max_output_tokens.map(|v| v as i32);

454

let max_new_tokens = request.max_output_tokens.map(|v| v as i32);

449

455

450

Ok(proto::SamplingParams {

456

Ok(proto::SamplingParams {

5 linesAll 14 lines5 lines

465

n: 1, // Responses API doesn't support n>1

471

n: 1, // Responses API doesn't support n>1

466

constraint: None, // No constraints - tools handled by MCP

472

constraint: self.build_constraint_for_responses(tool_call_constraint)?,

467

..Default::default()

473

..Default::default()

468

})

474

})

469

}

475

}

470

476

477

/// Build constraint for Responses API (simpler than Chat API's build_constraint)

478

///

479

/// Responses API doesn't support response_format, ebnf, or regex constraints,

480

/// so this only handles tool_call_constraint.

481

fn build_constraint_for_responses(

482

&self,

483

tool_call_constraint: Option<(String, String)>,

484

) -> Result<Option<proto::sampling_params::Constraint>, String> {

485

if let Some((constraint_type, constraint_value)) = tool_call_constraint {

Copy + paste486–494

486

let tool_constraint = match constraint_type.as_str() {

487

"structural_tag" => {

488

proto::sampling_params::Constraint::StructuralTag(constraint_value)

489

}

490

"json_schema" => proto::sampling_params::Constraint::JsonSchema(constraint_value),

491

"ebnf" => proto::sampling_params::Constraint::EbnfGrammar(constraint_value),

492

"regex" => proto::sampling_params::Constraint::Regex(constraint_value),

493

_ => return Err(format!("Unknown constraint type: {}", constraint_type)),

494

};

495

Ok(Some(tool_constraint))

496

} else {

497

Ok(None)

498

}

499

}

500

5 linesAll 101 lines5 lines

572

}

602

}

5 linesAll 167 lines5 lines

4Regular Router: tool_choice Pass-through

0 / 2

conversions.rssgl-router/src/routers/grpc/regular/responses

+56−71

Mark as viewed

All 9 lines

use crate::protocols::{

use crate::{

chat::{ChatCompletionRequest, ChatCompletionResponse, ChatMessage, UserMessageContent},

protocols::{

common::{FunctionCallResponse, StreamOptions, ToolCall, UsageInfo},

chat::{ChatCompletionRequest, ChatCompletionResponse, ChatMessage, UserMessageContent},

responses::{

common::{FunctionCallResponse, StreamOptions, ToolCall, ToolChoice, UsageInfo},

ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,

responses::{

ResponseReasoningContent::ReasoningText, ResponseStatus, ResponsesRequest,

ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,

ResponsesResponse, ResponsesUsage, StringOrContentParts,

ResponseReasoningContent::ReasoningText, ResponseStatus, ResponsesRequest,

ResponsesResponse, ResponsesUsage, StringOrContentParts,

},

routers::grpc::common::responses::utils::extract_tools_from_response_tools,

};

/// Convert a ResponsesRequest to ChatCompletionRequest for processing through the chat pipeline

///

/// # Conversion Logic

/// - `input` (text/items) → `messages` (chat messages)

/// - `instructions` → system message (prepended)

``/// - max_output_tokens → `max_completion_tokens```

/// - Tool-related fields are passed through

/// - `tools` → function tools extracted from ResponseTools

/// - `tool_choice` → passed through from request

/// - Response-specific fields (previous_response_id, conversation) are handled by router

pub fn responses_to_chat(req: &ResponsesRequest) -> Result<ChatCompletionRequest, String> {

5 linesAll 10 lines5 lines

// 2. Convert input to chat messages

match &req.input {

All 7 lines

ResponseInput::Items(items) => {

// Structured items → convert each to appropriate chat message

for item in items {

match item {

ResponseInputOutputItem::SimpleInputMessage { content, role, .. } => {

5 linesAll 18 lines5 lines

Move71–99

match role.as_str() {

messages.push(role_to_chat_message(role.as_str(), text));

"user" => {

messages.push(ChatMessage::User {

content: UserMessageContent::Text(text),

name: None,

});

}

"assistant" => {

messages.push(ChatMessage::Assistant {

content: Some(text),

name: None,

tool_calls: None,

reasoning_content: None,

});

}

"system" => {

messages.push(ChatMessage::System {

content: text,

name: None,

});

}

_ => {

// Unknown role, treat as user message

messages.push(ChatMessage::User {

content: UserMessageContent::Text(text),

name: None,

});

}

100

}

101

ResponseInputOutputItem::Message { role, content, .. } => {

102

// Extract text from content parts

103

let text = extract_text_from_content(content);

104

105

match role.as_str() {

messages.push(role_to_chat_message(role.as_str(), text));

106

"user" => {

107

messages.push(ChatMessage::User {

108

content: UserMessageContent::Text(text),

109

name: None,

110

});

111

}

112

"assistant" => {

113

messages.push(ChatMessage::Assistant {

114

content: Some(text),

115

name: None,

116

tool_calls: None,

117

reasoning_content: None,

118

});

119

}

120

"system" => {

121

messages.push(ChatMessage::System {

122

content: text,

123

name: None,

124

});

125

}

126

_ => {

127

// Unknown role, treat as user message

128

messages.push(ChatMessage::User {

129

content: UserMessageContent::Text(text),

130

name: None,

131

});

132

}

133

}

134

}

5 linesAll 61 lines5 lines

196

}

144

}

197

}

145

}

198

}

146

}

199

}

147

}

All 6 lines

206

// 3. Build ChatCompletionRequest

154

// 3. Extract function tools from ResponseTools

155

// Only function tools are extracted here (include_mcp: false).

156

// MCP tools are merged later by the tool loop (see tool_loop.rs:prepare_chat_tools_and_choice)

157

// before the chat pipeline, where tool_choice constraints are applied to ALL tools combined.

158

let function_tools = extract_tools_from_response_tools(req.tools.as_deref(), false);

159

let tools = if function_tools.is_empty() {

160

None

161

} else {

162

Some(function_tools)

163

};

164

165

// 4. Build ChatCompletionRequest

207

let is_streaming = req.stream.unwrap_or(false);

166

let is_streaming = req.stream.unwrap_or(false);

208

167

209

Ok(ChatCompletionRequest {

168

Ok(ChatCompletionRequest {

5 linesAll 19 lines5 lines

229

skip_special_tokens: true,

188

skip_special_tokens: true,

230

// Note: tools and tool_choice will be handled separately for MCP transformation

189

tools,

231

tools: None, // Will be set by caller if needed

190

tool_choice: req.tool_choice.clone(),

232

tool_choice: None, // Will be set by caller if needed

233

..Default::default()

191

..Default::default()

234

})

192

})

235

}

193

}

5 linesAll 14 lines5 lines

Move208–232

208

/// Convert role and text to ChatMessage

209

fn role_to_chat_message(role: &str, text: String) -> ChatMessage {

210

match role {

211

"user" => ChatMessage::User {

212

content: UserMessageContent::Text(text),

213

name: None,

214

},

215

"assistant" => ChatMessage::Assistant {

216

content: Some(text),

217

name: None,

218

tool_calls: None,

219

reasoning_content: None,

220

},

221

"system" => ChatMessage::System {

222

content: text,

223

name: None,

224

},

225

_ => {

226

// Unknown role, treat as user message

227

ChatMessage::User {

228

content: UserMessageContent::Text(text),

229

name: None,

230

}

231

}

232

}

233

}

234

250

/// Convert a ChatCompletionResponse to ResponsesResponse

235

/// Convert a ChatCompletionResponse to ResponsesResponse

All 7 lines

258

pub fn chat_to_responses(

243

pub fn chat_to_responses(

259

chat_resp: &ChatCompletionResponse,

244

chat_resp: &ChatCompletionResponse,

260

original_req: &ResponsesRequest,

245

original_req: &ResponsesRequest,

261

response_id_override: Option<String>,

246

response_id_override: Option<String>,

262

) -> Result<ResponsesResponse, String> {

247

) -> Result<ResponsesResponse, String> {

5 linesAll 76 lines5 lines

339

// Generate response

324

// Generate response

340

Ok(ResponsesResponse {

325

Ok(ResponsesResponse {

5 linesAll 14 lines5 lines

355

temperature: original_req.temperature,

340

temperature: original_req.temperature,

356

text: None,

341

text: None,

357

tool_choice: "auto".to_string(), // TODO: Map from original request

342

tool_choice: ToolChoice::serialize_to_string(&original_req.tool_choice),

358

tools: original_req.tools.clone().unwrap_or_default(),

343

tools: original_req.tools.clone().unwrap_or_default(),

All 6 lines

365

})

350

})

366

}

351

}

5 linesAll 63 lines5 lines

tool_loop.rssgl-router/src/routers/grpc/regular/responses

+45−24

Mark as viewed

All 8 lines

use axum::{

body::Body,

http::{header, StatusCode},

response::Response,

};

use bytes::Bytes;

use futures_util::StreamExt;

use serde_json::json;

use serde_json::{json, Value};

use tokio::sync::mpsc;

All 5 lines

use crate::{

mcp::{self, McpManager},

protocols::{

chat::{

ChatChoice, ChatCompletionMessage, ChatCompletionResponse, ChatCompletionStreamResponse,

ChatChoice, ChatCompletionMessage, ChatCompletionRequest, ChatCompletionResponse,

ChatCompletionStreamResponse,

},

All 6 lines

},

All 4 lines

};

5 linesAll 28 lines5 lines

/// Merge function tools from request with MCP tools and set tool_choice based on iteration

fn prepare_chat_tools_and_choice(

chat_request: &mut ChatCompletionRequest,

mcp_chat_tools: &[Tool],

iteration: usize,

) {

// Merge function tools from request with MCP tools

let mut all_tools = chat_request.tools.clone().unwrap_or_default();

all_tools.extend(mcp_chat_tools.iter().cloned());

chat_request.tools = Some(all_tools);

// Set tool_choice based on iteration

// - Iteration 0: Use user's tool_choice or default to auto

// - Iteration 1+: Always use auto to avoid infinite loops

chat_request.tool_choice = if iteration == 0 {

chat_request

.tool_choice

.clone()

.or(Some(ToolChoice::Value(ToolChoiceValue::Auto)))

} else {

Some(ToolChoice::Value(ToolChoiceValue::Auto))

};

}

5 linesAll 96 lines5 lines

165

fn build_mcp_list_tools_item(mcp: &Arc<McpManager>, server_label: &str) -> ResponseOutputItem {

190

fn build_mcp_list_tools_item(mcp: &Arc<McpManager>, server_label: &str) -> ResponseOutputItem {

166

let tools = mcp.list_tools();

191

let tools = mcp.list_tools();

167

let tools_info: Vec<McpToolInfo> = tools

192

let tools_info: Vec<McpToolInfo> = tools

168

.iter()

193

.iter()

169

.map(|t| {

194

.map(|t| McpToolInfo {

170

use serde_json::Value;

195

name: t.name.to_string(),

171

McpToolInfo {

196

description: t.description.as_ref().map(|d| d.to_string()),

172

name: t.name.to_string(),

197

input_schema: Value::Object((*t.input_schema).clone()),

173

description: t.description.as_ref().map(|d| d.to_string()),

198

annotations: Some(json!({

174

input_schema: Value::Object((*t.input_schema).clone()),

199

"read_only": false

175

annotations: Some(json!({

200

})),

176

"read_only": false

177

})),

178

}

179

})

201

})

180

.collect();

202

.collect();

All 6 lines

187

}

209

}

5 linesAll 29 lines5 lines

217

pub(super) async fn execute_tool_loop(

239

pub(super) async fn execute_tool_loop(

All 6 lines

224

) -> Result<ResponsesResponse, Response> {

246

) -> Result<ResponsesResponse, Response> {

5 linesAll 23 lines5 lines

248

// Get MCP tools and convert to chat format (do this once before loop)

270

// Get MCP tools and convert to chat format (do this once before loop)

249

let mcp_tools = ctx.mcp_manager.list_tools();

271

let mcp_tools = ctx.mcp_manager.list_tools();

250

let chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);

272

let mcp_chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);

251

debug!("Converted {} MCP tools to chat format", chat_tools.len());

273

debug!(

274

"Converted {} MCP tools to chat format",

275

mcp_chat_tools.len()

276

);

252

277

InformationalR278-301

253

loop {

278

loop {

254

// Convert to chat request

279

// Convert to chat request

255

let mut chat_request = conversions::responses_to_chat(&current_request)

280

let mut chat_request = conversions::responses_to_chat(&current_request)

256

.map_err(|e| error::bad_request(format!("Failed to convert request: {}", e)))?;

281

.map_err(|e| error::bad_request(format!("Failed to convert request: {}", e)))?;

257

282

258

// Add MCP tools to chat request so LLM knows about them

283

// Prepare tools and tool_choice for this iteration

259

chat_request.tools = Some(chat_tools.clone());

284

prepare_chat_tools_and_choice(&mut chat_request, &mcp_chat_tools, state.iteration);

260

chat_request.tool_choice = Some(ToolChoice::Value(ToolChoiceValue::Auto));

5 linesAll 180 lines5 lines

441

}

465

}

442

}

466

}

5 linesAll 73 lines5 lines

516

async fn execute_tool_loop_streaming_internal(

540

async fn execute_tool_loop_streaming_internal(

All 6 lines

523

) -> Result<(), String> {

547

) -> Result<(), String> {

5 linesAll 32 lines5 lines

556

// Get MCP tools and convert to chat format (do this once before loop)

580

// Get MCP tools and convert to chat format (do this once before loop)

557

let mcp_tools = ctx.mcp_manager.list_tools();

581

let mcp_tools = ctx.mcp_manager.list_tools();

558

let chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);

582

let mcp_chat_tools = convert_mcp_tools_to_chat_tools(&mcp_tools);

559

debug!(

583

debug!(

560

"Streaming: Converted {} MCP tools to chat format",

584

"Streaming: Converted {} MCP tools to chat format",

561

chat_tools.len()

585

mcp_chat_tools.len()

562

);

586

);

All 4 lines

Potential BugR591-662

567

loop {

591

loop {

5 linesAll 11 lines5 lines

579

if !mcp_list_tools_emitted {

603

if !mcp_list_tools_emitted {

580

let (output_index, item_id) =

604

let (output_index, item_id) =

581

emitter.allocate_output_index(OutputItemType::McpListTools);

605

emitter.allocate_output_index(OutputItemType::McpListTools);

582

606

583

// Build tools list for item structure

607

// Build tools list for item structure

584

let tool_items: Vec<_> = mcp_tools

608

let tool_items: Vec<_> = mcp_tools

585

.iter()

609

.iter()

586

.map(|t| {

610

.map(|t| {

587

use serde_json::Value;

588

json!({

611

json!({

589

"name": t.name,

612

"name": t.name,

590

"description": t.description,

613

"description": t.description,

591

"input_schema": Value::Object((*t.input_schema).clone())

614

"input_schema": Value::Object((*t.input_schema).clone())

592

})

615

})

593

})

616

})

594

.collect();

617

.collect();

5 linesAll 37 lines5 lines

632

}

655

}

633

656

634

// Convert to chat request

657

// Convert to chat request

635

let mut chat_request = conversions::responses_to_chat(&current_request)

658

let mut chat_request = conversions::responses_to_chat(&current_request)

636

.map_err(|e| format!("Failed to convert request: {}", e))?;

659

.map_err(|e| format!("Failed to convert request: {}", e))?;

637

660

638

// Add MCP tools to chat request so LLM knows about them

661

// Prepare tools and tool_choice for this iteration (same logic as non-streaming)

639

chat_request.tools = Some(chat_tools.clone());

662

prepare_chat_tools_and_choice(&mut chat_request, &mcp_chat_tools, state.iteration);

640

chat_request.tool_choice = Some(ToolChoice::Value(ToolChoiceValue::Auto));

5 linesAll 268 lines5 lines

909

}

931

}

910

932

911

Ok(())

933

Ok(())

912

}

934

}

913

935

914

/// Convert MCP tools to Chat API tool format

936

/// Convert MCP tools to Chat API tool format

915

fn convert_mcp_tools_to_chat_tools(mcp_tools: &[mcp::Tool]) -> Vec<Tool> {

937

fn convert_mcp_tools_to_chat_tools(mcp_tools: &[mcp::Tool]) -> Vec<Tool> {

916

use serde_json::Value;

917

mcp_tools

938

mcp_tools

918

.iter()

939

.iter()

919

.map(|tool_info| Tool {

940

.map(|tool_info| Tool {

All 8 lines

928

.collect()

949

.collect()

929

}

950

}

930

951

5 linesAll 141 lines5 lines

5Shared Utilities

0 / 2

utils.rssgl-router/src/routers/grpc/common/responses

+48−1

Mark as viewed

5 linesAll 10 lines5 lines

use crate::{

core::WorkerRegistry,

mcp::McpManager,

protocols::responses::{ResponseTool, ResponseToolType},

protocols::{

common::Tool,

responses::{ResponseTool, ResponseToolType},

},

routers::{grpc::error, openai::mcp::ensure_request_mcp_client},

};

5 linesAll 62 lines5 lines

/// Extract function tools (and optionally MCP tools) from ResponseTools

///

/// This utility consolidates the logic for extracting tools with schemas from ResponseTools.

/// It's used by both Harmony and Regular routers for different purposes:

///

/// - **Harmony router**: Extracts both Function and MCP tools (with `include_mcp: true`)

/// because MCP schemas are populated by convert_mcp_tools_to_response_tools() before the

/// pipeline runs. These tools are used to generate structural constraints in the

/// Harmony preparation stage.

///

/// - **Regular router**: Extracts only Function tools (with `include_mcp: false`) during

/// the initial conversion from ResponsesRequest to ChatCompletionRequest. MCP tools

/// are merged later by the tool loop before being sent to the chat pipeline, where

/// tool_choice constraints are generated for ALL tools (function + MCP combined).

pub fn extract_tools_from_response_tools(

response_tools: Option<&[ResponseTool]>,

include_mcp: bool,

100

) -> Vec<Tool> {

101

let Some(tools) = response_tools else {

102

return Vec::new();

103

};

104

105

tools

106

.iter()

107

.filter_map(|rt| {

108

match rt.r#type {

109

// Function tools: Schema in request

110

ResponseToolType::Function => rt.function.as_ref().map(|f| Tool {

111

tool_type: "function".to_string(),

112

function: f.clone(),

113

}),

114

// MCP tools: Schema populated by convert_mcp_tools_to_response_tools()

115

// Only include if requested (Harmony case)

116

ResponseToolType::Mcp if include_mcp => rt.function.as_ref().map(|f| Tool {

117

tool_type: "function".to_string(),

118

function: f.clone(),

119

}),

120

// Hosted tools: No schema available, skip

121

_ => None,

122

}

123

})

124

.collect()

125

}

utils.rssgl-router/src/routers/grpc

+40−24

Mark as viewed

5 linesAll 10 lines5 lines

use super::{error, ProcessedMessages};

pub use crate::tokenizer::StopSequenceDecoder;

use crate::{

5 linesAll 14 lines5 lines

tokenizer::{

cache::CachedTokenizer,

chat_template::{ChatTemplateContentFormat, ChatTemplateParams},

stop::StopSequenceDecoderBuilder,

traits::Tokenizer,

HuggingFaceTokenizer,

HuggingFaceTokenizer, StopSequenceDecoder,

},

tool_parser::{

ParserFactory as ToolParserFactory, PooledParser as ToolPooledParser, ToolParser,

},

};

5 linesAll 238 lines5 lines

276

/// Filter tools based on tool_choice (shared by both routers)

276

/// Filter tools based on tool_choice (generic helper)

277

/// Returns a reference to the original body if no filtering needed,

277

///

278

/// otherwise returns a cloned and filtered body

278

/// Returns filtered tools if filtering is needed, otherwise returns None.

279

pub fn filter_tools_for_request(

279

/// Used by both Chat API and Responses API (Harmony) for constraint generation.

280

body: &ChatCompletionRequest,

280

pub fn filter_tools_by_tool_choice(

281

) -> std::borrow::Cow<'_, ChatCompletionRequest> {

281

tools: &[Tool],

282

match &body.tool_choice {

282

tool_choice: &Option<ToolChoice>,

283

Some(ToolChoice::AllowedTools { tools: allowed, .. }) if body.tools.is_some() => {

283

) -> Option<Vec<Tool>> {

284

let mut filtered_body = body.clone();

284

match tool_choice {

InvestigateR285-293

285

let all_tools = filtered_body.tools.as_ref().unwrap();

285

Some(ToolChoice::AllowedTools { tools: allowed, .. }) => {

286

let allowed_names: std::collections::HashSet<&str> =

286

let allowed_names: std::collections::HashSet<&str> =

287

allowed.iter().map(|t| t.name.as_str()).collect();

287

allowed.iter().filter_map(|t| t.function_name()).collect();

288

let filtered_tools: Vec<Tool> = all_tools

288

let filtered: Vec<Tool> = tools

289

.iter()

289

.iter()

290

.filter(|t| allowed_names.contains(t.function.name.as_str()))

290

.filter(|t| allowed_names.contains(t.function.name.as_str()))

291

.cloned()

291

.cloned()

292

.collect();

292

.collect();

293

filtered_body.tools = Some(filtered_tools);

293

Some(filtered)

294

std::borrow::Cow::Owned(filtered_body)

295

}

294

}

296

Some(ToolChoice::Function { function, .. }) if body.tools.is_some() => {

295

Some(ToolChoice::Function { function, .. }) => {

297

let mut filtered_body = body.clone();

296

let filtered: Vec<Tool> = tools

298

let all_tools = filtered_body.tools.as_ref().unwrap();

299

let filtered_tools: Vec<Tool> = all_tools

300

.iter()

297

.iter()

301

.filter(|t| t.function.name == function.name)

298

.filter(|t| t.function.name == function.name)

302

.cloned()

299

.cloned()

303

.collect();

300

.collect();

301

Some(filtered)

302

}

303

_ => None, // No filtering needed

304

}

305

}

306

307

/// Filter ChatCompletionRequest by tool_choice

308

///

309

/// Returns a reference to the original request if no filtering needed,

310

/// otherwise returns a cloned request with filtered tools.

311

///

312

/// Note: Tool existence is validated earlier in ChatCompletionRequest::validate(),

313

/// so this function assumes tool_choice references valid tools.

314

pub fn filter_chat_request_by_tool_choice(

315

body: &ChatCompletionRequest,

316

) -> std::borrow::Cow<'_, ChatCompletionRequest> {

317

if let Some(tools) = &body.tools {

318

if let Some(filtered_tools) = filter_tools_by_tool_choice(tools, &body.tool_choice) {

319

let mut filtered_body = body.clone();

304

filtered_body.tools = Some(filtered_tools);

320

filtered_body.tools = Some(filtered_tools);

305

std::borrow::Cow::Owned(filtered_body)

321

return std::borrow::Cow::Owned(filtered_body);

306

}

322

}

307

_ => std::borrow::Cow::Borrowed(body), // No filtering needed, use original

308

}

323

}

324

325

// No filtering needed - return original request

326

std::borrow::Cow::Borrowed(body)

309

}

327

}

5 linesAll 124 lines5 lines

434

pub fn create_stop_decoder(

452

pub fn create_stop_decoder(

435

tokenizer: &Arc<dyn Tokenizer>,

453

tokenizer: &Arc<dyn Tokenizer>,

436

stop: Option<&StringOrArray>,

454

stop: Option<&StringOrArray>,

437

stop_token_ids: Option<&Vec<u32>>,

455

stop_token_ids: Option<&Vec<u32>>,

438

skip_special_tokens: bool,

456

skip_special_tokens: bool,

439

no_stop_trim: bool,

457

no_stop_trim: bool,

440

) -> StopSequenceDecoder {

458

) -> StopSequenceDecoder {

441

use crate::tokenizer::stop::StopSequenceDecoderBuilder;

442

5 linesAll 32 lines5 lines

475

}

491

}

5 linesAll 644 lines5 lines

6Miscellaneous

0 / 3

handlers.rssgl-router/src/routers/grpc/regular/responses

+2−2

Mark as viewed

5 linesAll 53 lines5 lines

use crate::{

All 4 lines

protocols::{

chat::{self, ChatCompletionStreamResponse},

common,

common::{self, ToolChoice},

responses::{

self, ResponseContentPart, ResponseInput, ResponseInputOutputItem, ResponseOutputItem,

ResponseReasoningContent, ResponseStatus, ResponsesRequest, ResponsesResponse,

ResponsesUsage,

},

All 9 lines

};

5 linesAll 422 lines5 lines

500

impl StreamingResponseAccumulator {

500

impl StreamingResponseAccumulator {

5 linesAll 87 lines5 lines

588

fn finalize(self) -> ResponsesResponse {

588

fn finalize(self) -> ResponsesResponse {

5 linesAll 54 lines5 lines

643

ResponsesResponse {

643

ResponsesResponse {

5 linesAll 15 lines5 lines

659

text: None,

659

text: None,

660

tool_choice: "auto".to_string(),

660

tool_choice: ToolChoice::serialize_to_string(&self.original_request.tool_choice),

661

tools: self.original_request.tools.clone().unwrap_or_default(),

661

tools: self.original_request.tools.clone().unwrap_or_default(),

All 6 lines

668

}

668

}

669

}

669

}

670

}

670

}

5 linesAll 206 lines5 lines

preparation.rssgl-router/src/routers/grpc/regular/stages/chat

+1−1

Mark as viewed

5 linesAll 35 lines5 lines

impl ChatPreparationStage {

async fn prepare_chat(

&self,

ctx: &mut RequestContext,

request: &ChatCompletionRequest,

) -> Result<(), Response> {

// Step 1: Filter tools if needed

let body_ref = utils::filter_tools_for_request(request);

let body_ref = utils::filter_chat_request_by_tool_choice(request);

5 linesAll 59 lines5 lines

103

}

103

}

104

}

104

}

responses.rssgl-router/src/routers/grpc/harmony(2 of 2)

+1−18

Mark as viewed

5 linesAll 50 lines5 lines

use crate::{

data_connector::{ResponseId, ResponseStorage},

mcp::{self, McpManager},

protocols::{

common::{Function, ToolCall, Usage},

common::{Function, ToolCall, ToolChoice, ToolChoiceValue, Usage},

responses::{

All 4 lines

},

5 linesAll 10 lines5 lines

};

5 linesAll 383 lines5 lines

457

/// Serve Harmony Responses API with streaming (SSE)

457

/// Serve Harmony Responses API with streaming (SSE)

5 linesAll 11 lines5 lines

469

``/// - Handles errors with `response.failed```

469

``/// - Handles errors with `response.failed```

470

///

471

/// # Arguments

472

///

473

/// * `ctx` - Harmony responses context with pipeline and dependencies

474

/// * `request` - Responses API request

475

///

476

/// # Returns

477

///

478

/// SSE stream response with proper headers

479

pub async fn serve_harmony_responses_stream(

470

pub async fn serve_harmony_responses_stream(

480

ctx: &HarmonyResponsesContext,

471

ctx: &HarmonyResponsesContext,

481

request: ResponsesRequest,

472

request: ResponsesRequest,

482

) -> Response {

473

) -> Response {

5 linesAll 54 lines5 lines

537

}

528

}

5 linesAll 560 lines5 lines

1098

fn build_next_request_with_tools(

1089

fn build_next_request_with_tools(

All 5 lines

1104

) -> Result<ResponsesRequest, Box<Response>> {

1095

) -> Result<ResponsesRequest, Box<Response>> {

5 linesAll 92 lines5 lines

1192

Ok(request)

1188

Ok(request)

1193

}

1189

}

5 linesAll 19 lines5 lines

1213

/// Convert MCP tools to Responses API tool format

1209

/// Convert MCP tools to Responses API tool format

1214

///

1210

///

1215

/// Converts MCP Tool entries (from rmcp SDK) to ResponseTool format so the model

1211

/// Converts MCP Tool entries (from rmcp SDK) to ResponseTool format so the model

1216

/// knows about available MCP tools when making tool calls.

1212

/// knows about available MCP tools when making tool calls.

1217

///

1218

/// # Arguments

1219

///

1220

/// * `mcp_tools` - MCP tools from the MCP manager inventory (rmcp::model::Tool)

1221

///

1222

/// # Returns

1223

///

1224

/// Vector of ResponseTool entries in MCP format

1225

pub fn convert_mcp_tools_to_response_tools(mcp_tools: &[mcp::Tool]) -> Vec<ResponseTool> {

1213

pub fn convert_mcp_tools_to_response_tools(mcp_tools: &[mcp::Tool]) -> Vec<ResponseTool> {

5 linesAll 18 lines5 lines

1244

}

1232

}

5 linesAll 163 lines5 lines

End of changesChat about this PR

InfoChat

1 Potential bug

Streaming tool loop never uses user's tool_choice due to premature iteration increment

Bugtool_loop.rs:591

5 Flags

Tool filtering returns empty vec for AllowedTools with only non-function references

Investigateutils.rs:285-293

Harmony preparation extracts both Function and MCP tools for constraint generation

preparation.rs:131-140

MCP tool references bypass validation intentionally but may cause runtime errors

responses.rs:776-793

Harmony router correctly switches tool_choice to auto for subsequent iterations

responses.rs:1183-1187

Non-streaming tool loop iteration tracking is correct

tool_loop.rs:278-301

ChecksPartial121/122

PR Test

pr-test-finishRequired3s

Reviewers3

KEkey4ng

BYByronHsu

SLslin1237

Assignees

No assignees

Labels4

enhancementhigh priorityrouterrun-ci

Introducing Devin Review

Devin Review is an intelligent PR review tool that helps you better understand your team's code before you ship it.

Intelligently organizes your diff

Groups changes into logical sections with clear explanations

→

Detects moved or copied code

Keeps the diff clean by highlighting copy-pasted or moved code

Analyzes your PR

Highlights potential bugs and flags areas that deserve extra careful review

Ask questions or request edits in the chat.

Ask Devin anything about this PR (Ctrl+I)

Authorization Response