mortred_model_server

How To Add New Model (Unified Backend Layer)

python scripts/new_model.py --list-tasks
python scripts/new_model.py --task object_detection --name rtdetr \
    --class RtdetrDetector --backend tensorrt --dry-run
python scripts/new_model.py --task object_detection --name rtdetr \
    --class RtdetrDetector --backend tensorrt

This generates the header, the .inl, the TOML config, an output contract test and a README, and prints the registration snippets it deliberately does not apply for you (catalog entry, test target, golden case). The scaffold compiles immediately; every unimplemented hook returns MODEL_NOT_IMPLEMENTED, so a half-finished model can never be served by accident. src/models/object_detection/rtdetr_detector.* is a checked-in example of exactly this output and doubles as the canary that keeps the templates compilable.

The rest of this document explains what the scaffold leaves for you to write.

All CV models now inherit from jinq::models::BackendCvModel<INPUT, OUTPUT>. The base class implements the full lifecycle:

init:      parse [SECTION.backend] -> create InferenceSession -> on_init([SECTION.params])
run_impl:  prepare_inputs -> session.run -> postprocess(context)

A standard single-image model only implements preprocess (cv::Mat to named tensors) and postprocess (named tensors plus request geometry to task output). Backend plumbing (MNN / ONNX Runtime / TensorRT session management, dtype & shape validation, dynamic shape handling, host/device copies) lives in src/models/backend/ and is never repeated per model.

Step 1: Pick the IO types

IO types live in src/models/io/, one header per task. common_input.h holds the shared inputs (mat_input, file_input, base64_input, pair_mat_input) and each task header holds its own std_*_output. Include only the task header you need - the old model_io_define.h still works but is a compatibility aggregate that pulls in every task. The loadable image inputs work with the default prepare_inputs path; task default outputs (std_*_output) are the recommended choice.

Step 2: Write the model class

Reference implementations (read these first):

template<typename INPUT, typename OUTPUT>
class MyModel : public jinq::models::BackendCvModel<INPUT, OUTPUT> {
  public:
    MyModel() : jinq::models::BackendCvModel<INPUT, OUTPUT>("MY_MODEL") {}

  private:
    // image -> named input tensors (required for image models)
    std::vector<jinq::models::backend::NamedTensor> preprocess(const cv::Mat& image) override;

    // named output tensors -> task output
    jinq::common::StatusCode postprocess(
        const std::vector<jinq::models::backend::NamedTensor>& outputs,
        const jinq::models::backend::InferenceContext& context,
        OUTPUT& output) override;

    // optional: read model specific keys from [MY_MODEL.params]
    jinq::common::StatusCode on_init(const toml::table& params) override;
};

Notes:

Step 3: Write the config

[MY_MODEL]
[MY_MODEL.backend]
type = "mnn"                # mnn | onnx | tensorrt
model_file_path = "../weights/my_model/model.mnn"
device = "gpu"             # cpu | gpu; omitted defaults to gpu; tensorrt forbids cpu
threads = 4
gpu_mem_limit_mb = 2048     # onnx+cuda only; 0 = unlimited; default 2048
input_layout = "nhwc"       # mnn only: auto | nhwc | nchw

[MY_MODEL.params]
score_threshold = 0.25

See about_model_configuration.md for the full key reference. Old BACKEND_DICT / XXX_TRT / XXX_ONNX / XXX_MNN three-section configs are gone; use scripts/migrate_model_config.py to migrate them (--dry-run first, --check in CI).

Step 4: Register in the task catalog

Every task owns an explicit catalog in src/factory/<task>_task.h. Adding a served model is now one row plus its creator - no hand-written server registration lambda, no copied CvServerSpec block:

// src/factory/my_task.h
template <typename INPUT, typename OUTPUT>
std::unique_ptr<BaseAiModel<INPUT, OUTPUT>> create_my_model(const std::string& name) {
    (void)name;
    return std::make_unique<MyModel<INPUT, OUTPUT>>();
}

using Output = jinq::models::io_define::my_task::std_my_task_output;
using Entry = jinq::factory::cv_catalog::CvModelEntry<Output>;

inline const std::vector<Entry>& catalog() {
    static const std::vector<Entry> entries = {
        Entry{"MY_MODEL", "My model display name", "MY_MODEL_SERVER",
              &create_my_model<jinq::server::Base64Input, Output>,
              &jinq::server::response::fill_my_task},
    };
    return entries;
}

factory::cv_catalog::create_server(catalog(), "MY_MODEL", server_name) does the rest: it registers the creator in ServerFactory<BaseAiServer> and builds the generic CvModelServer<Output>.

Two shapes exist on purpose:

If a task has more than one output contract, split it into one typed catalog per contract instead of type-erasing the list - see catalog() and face_catalog() in obj_detection_task.h.

test/model_catalog_unittest.cc fails the build when a catalog row references a TOML section or model_config_file_path that does not exist, or when the model or server section is duplicated across tasks.

Step 5: Verify

cmake --preset full && cmake --build --preset full
scripts/run_tests.sh build/full -R model_golden_test --output-on-failure

Register it with one macro from model_golden_registry.h - see the developer guide for the full list and the two-command baseline workflow. Prove a refactor changed nothing with golden_drift_check.py, and cover the rejection matrix with POSTPROCESS_CONTRACT_TEST.

Add a golden case with real weights to test/model_golden_test.cc (tolerances are per task: score/box-IoU for detection, fingerprint diff for dense outputs).

This site is open source. Improve this page.