SymetryML Release Notes

Version 7.0

Wire Protocol

Request signature

The secret key is no longer part of the string to sign; it is only the HMAC-SHA256 key. HMAC-SHA1 is no longer accepted. See Authentication Header. The same change applies to WebSocket messages.

Protocol version

Every REST request must send sym-version with major version 7 (for example sym-version: 7.0), and every WebSocket message carries the version as a fifth header string. A missing or older version is refused with 400 and a message asking for a client upgrade. See Protocol Version Header.

Encrypted bodies

Encrypted request bodies and response values (data source definitions, federation and fusion information) use a new format: v2: followed by Base64 of a random 12-byte nonce, the AES-GCM cipher text and its 16-byte tag. The fixed-IV AES/CBC format is refused with 400. The same format is used in Redis, on NATS and in exported files. See Encrypted Request and Response Bodies.

Authentication errors

The stringToSign entity of a 401 Invalid Signature response is sent only when the server runs with debug logging. A replayed request (same signature) is refused with 403 Duplicate request.

Security

TLS and SFTP verification on by default

Certificates of federation and fusion peers, HTTPS data sources and tls:// NATS servers are verified against the platform trust store with hostname verification, and SFTP host keys must be in known_hosts. New settings: sml.tls.trust_all, sml.tls.truststore, sml.tls.truststore.password and sml.sftp.known_hosts; sml.fed.admin.trust_all_certs is now a deprecated alias of sml.tls.trust_all. NATS opentls:// requires sml.tls.trust_all=1, and NATS credentials over plain nats:// can be refused with sml.nats.plaintext_credentials.allow=0. See SymetryML REST Configuration.

Local-file data sources off by default

The localfile data source type is refused unless the server sets rtlm.option.sml.local_ds.enable=1.

Kafka properties

Stream data sources may no longer set group.id, choose arbitrary deserializers or JAAS login modules, or name classes for Kafka to load. See Kafka Properties the Server Refuses.

Name validation

Names that were accepted before are now refused with 400:

• user names outside [A-Za-z0-9][A-Za-z0-9._@+-]{0,127} (REST user management and registration);
• federation names outside [A-Za-z0-9_-]{1,64} (NATS federations);
• NATS subjects that are not dot-separated tokens, or that contain whitespace, * or >;
• project names ending in -learn_merge_tmp or _rws_embedded;
• JDBC, Redshift and Snowflake database, schema, table and warehouse names with characters other than letters, digits, _ and $ (PostgreSQL-family table names are quoted instead).

JDBC and Redshift connections

A jdbc data source may only use a jdbcprefix and jdbcdriverclass pair from the server's sml.jdbc.drivers list (default: MySQL and PostgreSQL), with a plain host name or IP address and a numeric port. The server builds the connection URL itself, sends credentials as connection properties, and turns off MySQL local-file loading and object deserialization. A Redshift data source URL must be jdbc:redshift://host[:port][/database] with options only from sml.redshift.url_options (default tcpKeepAlive). Definitions that break these rules are refused with 400 when they are created. Generic-JDBC data sources using jdbc:redshift or another driver need the administrator to add the pair. See JDBC.

Federation

A peer joining a federation no longer receives the administrator's AWS key pair, and the files exchanged between peers no longer carry cloud or NATS credentials, the group key or the invite token. Every member of a federation must run 7.0. SMPC noise is never scaled below 2^8, whatever the peers announce.

Prediction output files

Prediction files written to a data source are RFC 4180 CSV, and text values that start with =, +, - or @ are prefixed with ' so that spreadsheets do not run them as formulas.

Web configuration

/webConfig returns only the three public web UI settings (sym.web.legal.notice, sym.web.auto.logoff.enable, sym.web.auto.logoff.time) and sml.local_ds.enable.

Drift Calculator

Persistence

A drift calculator created with drift_persist=true survives server restarts. A project it references cannot be deleted (409 CONFLICT) until the calculator is deleted or refit. See Persistence.

Exploration

Hypothesis tests corrected

z, t and F test results change: n−1 variances throughout, a two-sided p-value for the F-test conditioned on binary features, and several t-test fixes. Results differ from 6.x. See Various Hypothesis Test.

HTTP Status Codes

Client mistakes that returned a generic 500 now return 400 with a message, including encrypted payloads in the wrong format. In particular:

• Model information and project information return 404 for a model or project that does not exist (previously 400), and data source information returns 404 for an unknown data source.
• Every Drift Calculator endpoint returns 400 for an unknown calculator, unknown project, missing body, empty project list, or calculate before fit.
• SVD feature selection returns 400 for an unknown attribute or target before scheduling the job.
• Forget on a project with STSketches returns 400.
• A server whose Redis store has not been migrated to the 7.0 format answers every request with 503.

Removed

• Single sign-on (OIDC and LDAP): /smlaccess, /ssologin, /ssologout and all sso.* settings.
• Tasks API: /tasks, /tasks/create, /tasks/{taskid}, /tasks/{taskid}/start, /tasks/{taskid}/stop.
• Third-party (XGBoost) models: /tmodels, /tmodels/build, /tmodels/{modelid}, /tmodels/{modelid}/info, /tmodel/predict/{modelid}.
• EMR and Dataproc data sources: learning or forgetting from an emr or dataproc data source is refused with 400. Their server settings were removed.
• Server endpoints /logs, /metrics, /memoryClean, /updateServer and /projects/{pid}/status.
• Update check: the server no longer calls home for updates; sml.version.key and sml.version.host were removed. Updates ship as Docker images.
• ghmm and chmm model types.
• Settings sml.cat.multithread, sml.enable_bet.impute, sml.user.etl.load.max and sml.user.etl.notebook.dir. sml.task.max is no longer used.

Upgrade Notes

• Stored credentials must be migrated before the server can serve requests: contact SymetryML support before upgrading an existing installation. See Upgrading from 6.x to 7.0.
• Upgrade every client and every federation peer at the same time as the server.
• Provide trust material (truststore, known_hosts) for peers and data sources with self-signed or private-CA certificates, or set sml.tls.trust_all=1 on test systems.
• Set sml.local_ds.enable=1 if you use local-file data sources.
• Add any JDBC driver other than MySQL and PostgreSQL (including jdbc:redshift on the generic jdbc type) to sml.jdbc.drivers, and any Redshift URL option other than tcpKeepAlive to sml.redshift.url_options.
• Re-check hypothesis test thresholds and any client logic that relied on a 500 or a 400 for the cases listed under HTTP Status Codes.

Version 6.4

New REST API Endpoints

Drift Calculator

A stateful drift detector that is fit once against reference projects and then reused to score any number of projects, returning PCA reconstruction and Mahalanobis scores together. This complements the existing one-shot /projects/dataDrift endpoint, which recomputes from both projects on every call. See Drift Calculator API.

POST   /symetry/rest/{cid}/driftCalculator?id={id}
POST   /symetry/rest/{cid}/driftCalculator/{dcid}/fit
POST   /symetry/rest/{cid}/driftCalculator/{dcid}/calculate
GET    /symetry/rest/{cid}/driftCalculator/{dcid}/info
GET    /symetry/rest/{cid}/driftCalculators
DELETE /symetry/rest/{cid}/driftCalculator/{dcid}

Data Sources

Google Dataproc

A new dataproc data source type runs processing on a Google Cloud Dataproc cluster, reading data from Google Cloud Storage. Authentication resolves from the data source key, then a server setting, then Google Application Default Credentials — so on GCP no service-account JSON needs to be pasted in. Previewing a Dataproc data source reads GCS directly and does not launch a cluster. See Google Dataproc.

PostgreSQL and Amazon Redshift over JDBC

The jdbc data source type now supports the PostgreSQL family, selected by setting jdbcprefix to jdbc:postgresql or jdbc:redshift. The path takes the form database/table. Browsing lists databases through a maintenance connection and lists tables from the public schema. See PostgreSQL and Amazon Redshift over JDBC.

Snowflake Compute Pool Orchestrator

Snowpark Compute Pool processing is now enabled.

Field validation

Data source fields are now validated on create, update and sample preview, returning 400 BAD REQUEST with a message naming the missing field. In 6.3 these surfaced as generic 500 errors. Every data source type table now carries a Required column. See Field Validation.

HTTP and HTTPS data sources

Rewritten onto the JDK HTTP client. Redirects are now followed, a non-2xx response fails with HTTP GET <url> failed with status N, and a server that sends no Content-Length results in a reported size of -1.

Exploration

Per-request output precision

/explore takes a new precision query parameter — an integer in [0, 17], default 4. It replaces the server-wide sml.explore.double.format setting, which has been removed.

Partial failures

A bad MLContext no longer fails the whole /explore request. Each MLContext is evaluated independently and a failure becomes a map whose single key is the error message with value -1.0, inside a 200 OK. See Partial Failures.

Univariate quantiles

The uni metric accepts uni_q_<percentile> keys in extraParameters and returns a matching qtl_<percentile> entry.

useLocal on PCA, SVD, SVD feature selection and VIF

These four endpoints now accept the useLocal query parameter, matching /explore.

Prediction

Multiple models in one learnpredict call

The {modelid} path segment accepts a comma-separated list of model ids. Each model contributes its own columns to the returned DataFrame, named <model id>::<model key>.

Cross-Cutting Changes

Request body size limit

Request bodies are capped at 256 MB by default, configurable with request.body.max.size. A larger body is rejected with 413 Payload Too Large. See Request Body Size Limit.

CORS

The REST API can now emit CORS headers, letting browser pages call it directly. Off by default; enabled with -Dsml.rest.cors=true. See Cross-Origin Resource Sharing (CORS).

Improvements

JDBC and Snowflake values

Values from JDBC and Snowflake data sources are now carried through directly instead of passing through an intermediate CSV step. Column names and values that contain commas, quotes or newlines are handled correctly.

JDBC NULL handling

A SQL NULL now maps to the empty string, the same missing-value convention used elsewhere in SymetryML. Snowflake already behaved this way.

Federated exploration

Bivariate exploration, density estimates and conditional density estimates on a Federated project now use the correct project for each MLContext in a multi-context request.

Removed

• Legacy Spark data source types. Only spark<store>_map_reduce_4_1_0 remains; the unversioned sparks3_map_reduce and the 2.4.x / 3.0.x variants are rejected with Invalid ds type.
• sml.explore.double.format. Replaced by the per-request precision parameter.

Upgrade Notes

• Exploration results are no longer truncated to four decimals. Output now carries full precision unless precision is supplied. Any client parsing the old four-decimal output will see more digits.
• A 200 OK from /explore no longer means every MLContext succeeded. Check each returned map for a single entry whose value is -1.0.
• Request bodies larger than 256 MB now fail with 413. Raise request.body.max.size, or set it to 0 or less to disable the cap.
• Data sources missing a required field now fail at create time with a 400 instead of failing later.
• Re-load any JDBC or Snowflake data containing commas, quotes or newlines in column names or values — see Improvements above.
• Data sources using a legacy Spark type must be recreated with the _4_1_0 equivalent.

Version 6.3

New REST API Endpoints

LearnPredict

Combines learn and predict operations in a single call, enabling real-time learn-then-predict workflows without separate requests.

POST /symetry/rest/{cid}/projects/{pid}/learnpredict/{modelid}

Performance Estimation

Estimates supervised model performance on production data when ground truth labels are not yet available. Model-agnostic and label-free, it supports any binary classification model and estimates confusion matrix-based metrics (ROC curve, precision-recall curve, precision, recall, accuracy, F1 score, specificity). Based on the Probabilistic Adaptive Performance Estimation (PAPE) framework with a SymetryML-optimized single-pass implementation for streaming environments.

POST /symetry/rest/{cid}/projects/performanceEstimator

Data Drift Detection

Detects data drift between a baseline project and an analysis project using multiple drift metrics: PCA reconstruction error, global drift, marginal statistical, and marginal non-statistical methods. Integrates with Rolling Window projects for continuous drift monitoring.

POST /symetry/rest/{cid}/projects/dataDrift

EVT Wrapper Management

Creates or deletes Extreme Value Theory (EVT) wrappers on any model for dynamic anomaly thresholding. Instead of static thresholds, EVT wrappers track prediction outputs and use Extreme Value Theory to dynamically flag anomalies.

POST /symetry/rest/{cid}/projects/{pid}/{modelid}/evtwrapper
DELETE /symetry/rest/{cid}/projects/{pid}/{modelid}/evtwrapper

Conditional Density Estimates

Computes conditional density estimates (histograms) for pairs of attributes, useful for bivariate analysis with conditional distributions. Supports federated projects via the useLocal parameter.

POST /symetry/rest/{cid}/projects/{pid}/conditionalDensityEstimate

Survival Analysis

Cumulative Incidence Function (CIF)

New survival analysis model for competing risks. Unlike standard survival methods that can overestimate event probabilities by treating competing events as censored, CIF properly accounts for the fact that experiencing one event precludes another.

• Get Total Events -- POST /{cid}/projects/{pid}/cif/NtotEvents
• Predict At Risk -- POST /{cid}/projects/{pid}/cif/predictAtRiskCIF
• Predict Event -- POST /{cid}/projects/{pid}/cif/predictEvent

CIF project configuration parameters: cif_time_column, cif_event_column, cif_group_column, cif_event_of_interest, and sml_surv_type (set to cif).

Kaplan-Meier REST Endpoints

New REST endpoints for querying Kaplan-Meier survival models, including survival data frame retrieval and survival predictions with confidence intervals.

• Get Survival DataFrames -- GET /{cid}/projects/{pid}/km/getSurvivalFrame
• Kaplan-Meier Predict -- POST /{cid}/projects/{pid}/km/kaplanmeier

Federated Learning

Contract Management

New endpoints for managing data sharing contracts on federated projects:

• Add Contract -- POST /{cid}/fedml/{pid}/addContract
• Validate Contract -- POST /{cid}/fedml/{pid}/contractValidate

Federation Invitation

New alias endpoint for getting encrypted federation invitations:

POST /symetry/rest/{cid}/fedml/{pid}/getInvitation

The previous getEncrypted path is preserved for backward compatibility.

Exploration Enhancements

• Kurtosis: New univariate statistic (kurtosis) available in exploration results.
• Enhanced Bivariate Statistics: New metrics including conditional sum (condSum), conditional standard deviation (condStddev), and 95% confidence interval bounds (biCi95Down, biCi95Up).
• Bivariate Conditional Flags: New parameters need_bi_conditional, need_bi_conditional_mmm, and need_bi_conditional_ci95 to request conditional statistics in bivariate exploration.
• PCA Max Dimension: PCA results are now limited to 100 dimensions maximum by default. This value is configurable.
• Explore Endpoint: New returnProjectInfo query parameter to control whether project info is included in the response.
• Density Estimates: New useLocal parameter for federation support; improved threading.

Data Sources

Snowflake Data Source

New Snowflake data source type (snow) with support for:

• Account, user, role, and private key authentication
• Database, schema, and table configuration
• Snowpark Container Services (CP3O) compute pool configuration
• Job warehouse settings and data frame size limits

NATS Data Source

New NATS data source type (nats) with support for three data formats: Protobuf, JSON, and CSV. Protobuf format requires protobuf.schema and protobuf.msg.type.name parameters.

Spark 4.1.0

Spark support updated to version 4.1.0, required due to changes in the underlying JDK and Jetty server. Only Spark 4.1.0 is supported in this version.

Model Selection and AutoML

Genetic Algorithm Feature Selector (Experimental)

New evolutionary optimization approach for feature selection. Evolves a population of candidate feature sets over multiple generations using selection, crossover, and mutation to discover high-performing feature combinations. Use selector_type_genetic as the selector type.

Bayesian Optimization Feature Selector (Experimental)

New probabilistic modeling approach for efficient feature space search. Builds a surrogate model and uses an Upper Confidence Bound acquisition function to balance exploration and exploitation. Use selector_type_bayesian as the selector type.

MRMR Filter

New Minimum Redundancy Maximum Relevance filter (use_mrmr_filter) for feature pre-filtering during model selection.

Early Detection Assessment Metric

New early_detection assessment metric for binary classification models. Rewards models that return a positive prediction earlier in a sequence of observations. Designed for anomaly or malfunction detection on equipment where earlier detection yields a better score.

Feature Interaction Limit

New automl_feature_interaction_max_number parameter to control the maximum number of feature interactions during AutoML.

Rolling Window Projects

New project configuration for retaining statistics from only the most recent N rows of learned data:

• sml_project_rolling_window_size -- specifies the rolling window length
• sml_project_rolling_window_use_embedded -- creates an embedded project inside the main project for the rolling window
• The embedded project can be referenced in the Data Drift API using rolling_window_embedded_project as the analysis project name

Infrastructure Changes

• Logging Framework: Migrated from Log4j 1.x to Log4j 2.x
• Jetty Upgrade: Updated to Jetty 12.x (Jakarta EE)
• Explore Endpoint: Updated with returnProjectInfo parameter
• Density Estimates: Updated with federation support and improved threading

Version 6.2

New Features

SSO Authentication Support

SymetryML now supports Single Sign-On (SSO) authentication via OIDC (OpenID Connect) and LDAP. After SSO authentication, SymetryML credentials (username and secret key) are returned for API access. SSO can also auto-create SymetryML user accounts based on the SSO profile.

Server Logs API

A new REST endpoint allows retrieval of the last N lines from the server log file. The number of lines returned is controlled by the SML_LOG_TAIL_SIZE configuration parameter.

GET /symetry/rest/{cid}/logs

Infrastructure Changes

• Jakarta EE Migration: Migrated from javax.ws.rs to jakarta.ws.rs (JDK 11 support)
• All existing REST endpoints maintain full backward compatibility