mortred_model_server

 Mortred-AI-Web-Server: A Noob Web Server for AI Models | [English](/mortred_model_server/) | [中文](/mortred_model_server/README.zh-cn.html) | [](https://github.com/MaybeShewill-CV/mortred_model_server/actions/workflows/ci.yml)

Mortred is a Linux process-per-model CV inference appliance: one catalog id is one OS process. Clients talk to mortred-gateway (:8080); the supervisor (:8787) owns the process tree. Inference backends are MNN, ONNX Runtime, and TensorRT, served over workflow. Weights are trained elsewhere (tensorflow / pytorch).

Do not hesitate to let me know if you find bugs here cause I’m a c-with-struct noob :upside_down_face:

The three major components are illustrated on the architecture picture below.

A quick overview and examples for both serving and model benchmarking are provided below. Detailed documentation and examples will be provided in the docs folder.

You’re welcomed to ask questions and help me to make it better!

All models and detectors can be downloaded from my Hugging Face Page.

Contents of this document

• Quick Start
• Benchmark
• Tutorials
• How To
• Web Server Configuration
• HTTP API Contract
• Long-task /jobs customer test
• Model Zoo

Quick Start

Linux is the only supported platform. Two deployment profiles exist and one switch drives everything (build, dependencies, model catalog, weight subset):

	gpu (default)	cpu
backends	MNN-CUDA / ORT-CUDA / TensorRT	MNN-CPU / ORT-CPU
hardware	NVIDIA GPU + CUDA 11/12	any x64 machine
models	full zoo	curated set (mobilenetv2, resnet50)

Three entries, one core (mortredctl): pick whichever fits; they all end at the same mortredctl doctor acceptance gate.

Entry 1: one-line bootstrap (fastest)

curl -fsSL https://raw.githubusercontent.com/MaybeShewill-CV/mortred_model_server/main/scripts/bootstrap.sh | bash

Detects your hardware (NVIDIA GPU → gpu, otherwise cpu). With Docker it prints the compose track. Without Docker it resolves the latest GitHub release tag and downloads mortred_model_server-<version>-<profile>-linux-x64.tar.gz (there is no ...-latest-... tarball filename). If no Release exists yet, it warns and prints the source-build path.

Entry 2: docker compose

git clone https://github.com/MaybeShewill-CV/mortred_model_server.git
cd mortred_model_server
python3 scripts/fetch_weights.py --profile cpu        # or: gpu
./scripts/mortredctl_init-trust.sh                    # three distinct tokens
set -a && . conf/local/trust.env && set +a
docker compose --profile cpu up -d                    # or: --profile gpu
curl -fs http://localhost:8787/api/v1/health

Entry 3: release tarball + systemd (bare metal)

Download mortred_model_server-<version>-<profile>-linux-x64.tar.gz from Releases, verify its .sha256, then:

mkdir unpack && tar -xzf mortred_model_server-*-linux-x64.tar.gz -C unpack
cd unpack                                          # archive is flat: install.sh, opt/, deploy/
sudo ./install.sh                                  # runtime deps + /opt/mortred + systemd
# install.sh writes a comment-only placeholder; --force fills real secrets
sudo /opt/mortred/bin/mortredctl.out init-trust --force --out /etc/mortred/supervisor.env
cd /opt/mortred && python3 scripts/fetch_weights.py --profile cpu
sudo systemctl start mortred-supervisor

First-hour core: mortredctl

mortredctl init [--profile cpu|gpu]   # detect hw, fetch weight subset, verify
mortredctl init-trust                 # write conf/local/trust.env (gitignored)
mortredctl init-edge --mode lan       # optional Nginx TLS on the host network
mortredctl prepare [--pack FILE]      # GPU: pack TensorRT engines on this card
mortredctl calibrate [--pack FILE]    # worker_nums report; --write-pack is opt-in
mortredctl doctor                     # live acceptance + non-fatal security warnings
mortredctl doctor --strict            # same, but security warnings fail the gate
mortredctl status | catalog           # runtime introspection

GPU note: convert pack TensorRT engines on this machine with mortredctl prepare (not the whole zoo). MORTRED_AUTO_BUILD_ENGINES=true still converts every engine and stays opt-in. See docs/deployment.md §10.

Building from source

# dependencies (version matrix + sha256 pinned + idempotent stamps)
./scripts/install_deps.sh --all              # gpu line (CUDA 11 default)
./scripts/install_deps.sh --cpu --all        # cpu line (no NVIDIA/TRT at all)

# configure + build (presets carry the profile)
cmake --preset full && cmake --build --preset full            # gpu
cmake --preset full-cpu && cmake --build --preset full-cpu    # cpu

# verify
./scripts/verify_deployment.sh --basic

Unit tests only (no engines needed; system packages or vcpkg):

cmake --preset tests-only && cmake --build --preset tests-only && ctest --preset tests-only

The complete operations manual - architecture diagrams, per-track walkthroughs, security checklist, upgrades, troubleshooting - lives in docs/deployment.md / 中文版.

Benchmark

The benchmark test environment is as follows：

OS: Ubuntu 20.04.5 LTS / 5.15.0-87-generic

MEMORY: 32G DIMM DDR4 Synchronous 2666 MHz

CPU: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz

GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

GPU: GeForce RTX 3080

CUDA: CUDA Version: 11.5

GPU Driver: Driver Version: 495.29.05

Model Inference Benchmark

All models loop several times to avoid the influence of gpu’s warmup and only model’s inference time has been counted.

Benchmark Code Snappit

• Details Of Model Inference Benchmark
• About Model Configuration

Tutorials

• Image Classification Model Server Tutorials
• Image Segmentation Model Server Tutorials
• Image Object Detection Model Server Tutorials
• Image Enhancement Model Server Tutorials
• Image Feature Point Model Server Tutorials

How To

• How To Add New Model :fire::fire:
• How To Add New Server :fire::fire:
• Model Developer Guide (task-oriented paths, contract / golden / debugging)
• Inference CI (hosted MNN smoke vs maintainer GPU golden)
• P4: Modern Model Developer Experience Plan (Chinese)

Model Zoo

HTTP-served (mortred-model-server.out --list / catalog id):

Task	Catalog id
Classification	MOBILENETV2 RESNET DENSENET
Detection	YOLOV5 YOLOV6 YOLOV7 YOLOV8 NANODET
Face	LIBFACE CENTER_FACE
OCR	DBNET
Segmentation	BISENETV2 PPHUMAN_SEG HRNET
Matting	MODNET PP_MATTING
Enhancement	ENLIGHTEN_GAN ATTENTIVE_GAN_DERAIN REAL_ESRGAN
Feature points	SUPERPOINT
Embedding	DINOV2
Depth	METRIC3D DEPTH_ANYTHING
SAM	SAM_AMG
Diffusion	DDPM DDIM CLS_COND_DDIM LDM

Bench-only (no HTTP catalog): OPENAI_CLIP, LIGHTGLUE, SAM_PREDICTOR, FAST_SAM, MSOCRNET.

Scaffold / not served: RTDETR. There is no MOT.

Web Server Configuration

• Description About Model Server
• HTTP API Contract (topology, auth, status mapping, overload behaviour)

Deployment

One-command dependency install

Build and install all third-party dependencies (MNN / WORKFLOW / ONNXRUNTIME / TensorRT / CUDA / fmt / header-only libs) into 3rd_party/{include,libs} with a single script — no manual compilation or copying:

./scripts/install_deps.sh --all     # build/install everything (CUDA 11 baseline)
./scripts/install_deps.sh --check   # verify integrity and print versions
./scripts/install_deps.sh --cuda-version 12   # switch to the CUDA 12 / TRT 10 line

Docker (fully automated build)

docker build -t mortred_model_server:gpu .
docker run --gpus all -p 127.0.0.1:8080:8080 -p 127.0.0.1:8787:8787 \
  -v $PWD/weights:/opt/mortred/weights \
  -e MORTRED_GATEWAY_AUTH_TOKEN=your-inference-token \
  -e MORTRED_API_TOKEN=your-management-token \
  -e MORTRED_METRICS_TOKEN=your-scrape-token \
  mortred_model_server:gpu
# or: docker compose --profile gpu up -d   (CPU: --profile cpu; see docker-compose.yml)

The image builds all deps + the full project, runs the unit/e2e tests, and ships the control plane. In-container topology: mortred-supervisor (management :8787, embedded web UI + REST API) supervises mortred-gateway (data plane :8080, the single inference entry) and all model servers; model processes bind loopback only and are no longer exposed port by port. The compose and docker run examples bind 8080/8787 to 127.0.0.1 on the host. External exposure must terminate TLS at Nginx on the host network (mortredctl init-edge, deploy/nginx); do not publish those ports on 0.0.0.0 without it (Bearer tokens would travel in the clear). Gateway GET /metrics always requires MORTRED_METRICS_TOKEN, including on loopback. Fail-closed also refuses a listener with no inference/management auth, a missing scrape token, a scrape token that matches those, and a wildcard bind unless MORTRED_EXPOSE=docker (containers) or unsafe (plaintext on metal). mortredctl doctor warns about non-loopback listeners, missing scrape token, and weak/identical tokens; doctor --strict fails on those warnings. TLS stays at Nginx.

TensorRT engine regeneration (hardware-adapted)

Engines are bound to this GPU / TensorRT. Day-to-day, convert the current pack (deployment guide §10):

mortredctl prepare --pack conf/packs/yolov8.toml

Zoo-wide convert remains scripts/convert_trt_engines.sh (trtexec via sudo ./scripts/install_deps.sh --nvidia). MORTRED_AUTO_BUILD_ENGINES=true converts every engine and stays off by default.

TODO

• Add more model into model zoo

Repo-Status

Star History

Visitor Count

Acknowledgement

mortred_model_server refers to the following projects:

• https://github.com/sogou/workflow
• https://github.com/alibaba/MNN
• https://github.com/PaddlePaddle/PaddleSeg
• https://github.com/Tencent/rapidjson
• https://github.com/ToruNiina/toml11

This site is open source. Improve this page.