Observability¶
Ella MOCN supports four observability pillars: Logs, Metrics, Traces and Profiles.
Logs¶
Ella MOCN writes its logs to standard output, one JSON object per line, at the level of logging.level in the configuration file: debug, info, warn or error.
| Field | Description |
|---|---|
ts |
Time of the event. |
level |
debug, info, warn or error. |
msg |
What happened. |
trace_id |
Trace ID of the span the event happened in, as 32 hex digits. Absent outside a span. |
span_id |
Span ID of the span the event happened in, as 16 hex digits. Absent outside a span. |
trace_flags |
W3C trace flags of that span, as 2 hex digits: 01 when sampled. Absent outside a span. |
Other fields depend on the event.
Audit events¶
Audit events are logs at level info with msg set to audit event.
| Field | Description |
|---|---|
action |
What was done, such as auth_login or create_operator. |
actor |
Email of the user who did it. Empty when unknown. |
ip |
Address the request came from. |
details |
What was done, in words. |
Metrics¶
Ella MOCN exposes Prometheus metrics at GET /api/v1/metrics.
| Metric | Type | Labels | Description |
|---|---|---|---|
ellamocn_radio_associations |
Gauge | interface, state (connected, up, setup_failed) |
Radios connected, by state: connected before the S1 or NG Setup, up after, setup_failed after Ella MOCN rejected the last S1 or NG Setup. |
ellamocn_core_associations |
Gauge | interface, operator, state (disconnected, connected, up, setup_failed) |
Associations to MMEs and AMFs, by operator and state. |
ellamocn_core_overloaded |
Gauge | interface, operator |
MMEs and AMFs reporting overload, by operator. |
ellamocn_setup_attempts_total |
Counter | interface, side (radio, core), result (success, failure, timeout, unknown_plmn, …) |
S1 and NG Setup attempts from radios (radio) and to MMEs and AMFs (core), by result. |
ellamocn_initial_ue_messages_total |
Counter | interface, result (relayed, unknown_plmn, no_mme, no_amf, rejected, invalid, error) |
New UE connections from radios, by result. |
ellamocn_paging_messages_total |
Counter | interface, result (relayed, no_radio, invalid) |
Paging messages from MMEs and AMFs, by result. |
ellamocn_overload_messages_total |
Counter | interface, procedure (overload_start, overload_stop), result (relayed, no_radio, ignored, invalid) |
Overload start and stop messages from MMEs and AMFs, by result. |
ellamocn_handover_messages_total |
Counter | interface, procedure (handover_request, path_switch_request), result (relayed, no_radio, no_mme, no_amf, invalid, error) |
Handover and path switch requests, by result. |
ellamocn_pws_messages_total |
Counter | interface, procedure (write_replace_warning, kill, pws_cancel, pws_restart_indication, pws_failure_indication), result (relayed, no_radio, timeout, …) |
Public warning requests from MMEs and AMFs and indications from radios, by result. |
ellamocn_ue_connections |
Gauge | interface, operator |
UE connections relayed, by operator. |
ellamocn_api_request_duration_seconds |
Histogram | method, route, code |
Duration of API requests. |
ellamocn_database_query_duration_seconds |
Histogram | pool (write, read) |
Duration of database queries. |
ellamocn_database_query_errors_total |
Counter | pool (write, read) |
Failed database queries. |
ellamocn_database_storage_bytes |
Gauge | file (main, wal) |
Size of the database on disk. |
ellamocn_build_info |
Gauge | version, revision, branch, goversion, goos, goarch, tags |
Build information. Always 1. |
go_* |
Gauge, Counter, Summary | Go runtime. | |
process_* |
Gauge, Counter | Process memory, CPU and file descriptors. |
User-plane metrics¶
With user-plane aggregation:
| Metric | Type | Labels | Description |
|---|---|---|---|
ellamocn_user_plane_enabled |
Gauge | 1 when ran.user_plane.enabled is true in the configuration file, 0 otherwise. |
|
ellamocn_user_plane_datapath_loaded |
Gauge | 1 while the user-plane eBPF program is loaded, 0 otherwise. | |
ellamocn_user_plane_state |
Gauge | operator, state (ready, unavailable, off) |
1 for the operator's current user-plane state, 0 for the other states. |
ellamocn_user_plane_unavailable |
Gauge | operator, reason (not_connected, no_route, route_mismatch, no_address, interface_error, too_many_operators) |
1 for the reason while the state is unavailable, 0 otherwise. |
ellamocn_user_plane_tunnels |
Gauge | operator, direction |
GTP-U tunnels relayed. |
ellamocn_user_plane_tunnel_setups_total |
Counter | operator, direction |
GTP-U tunnels set up. |
ellamocn_user_plane_tunnel_setup_failures_total |
Counter | operator, interface (s1, ng), procedure, reason (user_plane_unavailable, no_route, route_mismatch, no_address, no_teid, too_many_paths, invalid_address, unknown_endpoint, other) |
Procedures whose user-plane transport could not be relayed. The UE is released, or the handover or path switch is rejected. |
ellamocn_user_plane_packets_total |
Counter | operator, radio, direction |
G-PDUs relayed. End Marker and Tunnel Status messages are not counted. |
ellamocn_user_plane_bytes_total |
Counter | operator, radio, direction |
Octets of the T-PDUs of the G-PDUs relayed: the inner IP packet, without the GTP-U header and extension headers. |
ellamocn_user_plane_dropped_packets_total |
Counter | operator, path (fast, slow), reason |
GTP-U messages dropped. |
ellamocn_user_plane_slow_path_packets_total |
Counter | reason |
Packets on user-plane interfaces that the fast path left to the kernel. |
ellamocn_user_plane_slow_path_relayed_total |
Counter | operator, direction |
G-PDUs, End Markers and Tunnel Status messages relayed by the slow path. |
ellamocn_user_plane_error_indications_total |
Counter | origin, result |
GTP-U Error Indications. |
ellamocn_user_plane_peer_up |
Gauge | operator, side (radio, core), peer |
0 once 3 Echo Requests in a row, sent every 60 s, go unanswered, 3 to 4 minutes after the peer stops answering. 1 otherwise. Present while a tunnel of the operator relays to the peer. |
ellamocn_user_plane_bpf_map_entries |
Gauge | map (tunnels, tunnel_stats, ifaces, path_stats) |
Entries in a map of the user-plane datapath. |
ellamocn_user_plane_bpf_map_max_entries |
Gauge | map (tunnels, tunnel_stats, ifaces, path_stats) |
Capacity of a map of the user-plane datapath. |
ellamocn_user_plane_bpf_map_memory_bytes |
Gauge | map (tunnels, tunnel_stats, ifaces, config, path_stats, drops, passes) |
Kernel memory locked by a map of the user-plane datapath. |
Traces¶
Ella MOCN exports OpenTelemetry traces when telemetry.enabled is set in the configuration file.
| Property | Value |
|---|---|
| Export | OTLP over gRPC, without TLS |
| Service | ella-mocn |
| Sampling | Parent-based, always on for a trace the MOCN starts |
| Propagation | W3C Trace Context |
| S1AP, NGAP and database spans | No NAS, security keys, subscriber identity, SQL text or parameters |
Spans¶
| Span | Kind | Description |
|---|---|---|
{method} {route} |
Server | A request to the API, except /api/v1/livez, /api/v1/readyz, /api/v1/metrics and /api/v1/pprof/. Links to the caller's traceparent. |
s1ap/receive |
Server, Consumer | An S1AP message from a radio or an MME. |
s1ap/send |
Client, Producer | An S1AP message to a radio or an MME, a child of the s1ap/receive that caused it. |
ngap/receive |
Server, Consumer | An NGAP message from a radio or an AMF. |
ngap/send |
Client, Producer | An NGAP message to a radio or an AMF, a child of the ngap/receive that caused it. |
db.{call}, such as db.CreateUser |
Internal | A database call within a trace. |
RAN trace correlation¶
When an MME or AMF activates a trace for a UE, that UE's S1AP or NGAP spans join the core's trace.
| Item | Value |
|---|---|
| OpenTelemetry trace ID | 8 octets 0x00 ‖ E-UTRAN Trace ID or NG-RAN Trace ID (8 octets) |
Profiles¶
Ella MOCN exposes the net/http/pprof API at /api/v1/pprof/.