mortred_model_server
About Model Sercver Configuration
All model server’s configurations are stored in $PROJECT_ROOT_DIR/conf/server folder.
Common Configuration
Use mobilenetv2’s model server configuration for example
host: server’s host address
port: server’s port
max_connections: server’s max connections. Old connections are kicked if no spare connection is left. Connections are refused if none can be kicked. Enlarge this when concurrency is large. See #issue463, #issue906 and tutorial-05-http_proxy.
peer_resp_timeout: the maximum duration for reading or sending out a block of data. The default setting is 15 seconds.
compute_threads: thread nums for computing. -1 mean default nums which equals your cpu core nums
handler_threads: thread nums for network process function and callback function
model_run_timeout: the time limit of a model’s inference process. inference process will be interruptted if cost more time.
server_uri: HTTP path on the model process (legacy gateway match). Example: /mortred_ai_server_v1/classification/mobilenetv2. Through the gateway prefer POST /v1/models/{id}/infer.
model / server_exe: catalog id and mortred-model-server.out.
model_config_file_path: lives in the [MODEL] table (not the *_SERVER table). See about_model_configuration.
Other Web Service Configuration
For other web service configuration you may find help at workflow_docs_about_global_configuration
Tune listen/timeout/worker settings in the matching conf/server/**/*.toml. Workflow-level defaults are applied once in src/server/base_server_impl.h for every model served by mortred-model-server.out --model <ID>.
Init Workflow Global Configuration Code Snappit
Overload Protection & Dynamic Batching (optional)
# max queued requests before the server answers 429 + Retry-After (0 = unlimited)
max_queue_depth=32
# dynamic batching: collect concurrent requests into one packed [N,...] run
# (default 1 = off). Works for HTTP single-session image models
# (classification/detection/segmentation/OCR/matting/enhancement/depth/
# SuperPoint). FastSAM / CLIP / LightGlue are bench-only and not HTTP-batched.
# Engines with dynamic batch (MNN) gain real throughput; TRT engines built
# with a static batch-1 profile fall back to per-item runs. Multi-session
# models (LightGlue / SAM prompt / CLIP) and diffusion samplers are not batchable.
max_batch_size=8
# batch collection window in ms: how long the first request waits for peers
max_batch_delay_ms=5