cx-telemetry-querying

Telemetry Querying Skill

Use this skill as the entry point for any investigation, debugging, or data question that may be answered from telemetry data. It helps you decide where the relevant signal lives (metrics, logs, traces, RUM) and tells you which reference files to load before querying.

Loading References

Before querying, load the reference files for the chosen pillar:

---

Safety

All query commands (cx logs, cx spans, cx metrics, cx dataprime, cx search-fields) are read-only and work in --read-only mode. They never modify data and can be run freely without --yes.

---

Quick Routing Guide

Use this table for obvious cases where one pillar is the clear first choice:

For ambiguous questions (e.g., "How much money did users spend last week?"), the signal could live in any pillar. Follow the Discovery Workflow below.

---

Discovery Workflow

When the answer could reside in multiple pillars, run discovery in parallel to find the best source.

Step 1: Search Metrics

Check if a relevant metric exists:

cx metrics search --name '*transaction*'
cx metrics search --name '*payment*'
cx metrics search --name '*revenue*'
cx metrics search --description "total purchase amount"

If a matching metric is found, load references/promql-guidelines.md + references/metrics-querying.md and continue.

Step 2: Search Log and Span Fields

Use semantic field search to find relevant DataPrime paths:

cx search-fields "transaction amount" --dataset logs
cx search-fields "payment total" --dataset spans
cx search-fields "purchase value" --dataset logs --limit 10

If you know a concrete value that should appear in the data but don't know which field holds it, use value search instead. It returns the matching field keys alongside sample values, which also lets you infer the field's type (string, numeric, enum, etc.):

cx search-fields "payment_failed" -s value --dataset logs
cx search-fields "grpc.status.UNAVAILABLE" -s value --dataset spans
cx search-fields "eu-west-1" -s value --dataset all

Requirements: cx search-fields needs a Coralogix API key or OAuth on the active profile. If credentials are missing, prompt the user to run cx profiles add <name>.

If matching fields are found:

• For logs: load references/dataprime-reference.md + references/logs-querying.md
• For spans: load references/dataprime-reference.md + references/spans-querying.md

Step 3: Search the Codebase

When discovery results are ambiguous or you need to validate what a metric/field actually represents, search the codebase:

• Look for metric registration code (e.g., prometheus.NewCounter, metrics.record)
• Look for log statements that emit the field (e.g., logger.info("transaction", ...))
• Look for span attributes (e.g., span.setAttribute("purchase.amount", ...))

This confirms the semantic meaning and helps you choose the right pillar.

Step 4: Choose and Query

Based on discovery results, pick the pillar with the clearest signal, load its reference files (see Loading References), then query.

---

Fallback and Pivoting

If your initial route yields no results, pivot to another pillar.

Example pivot paths:

• Metrics empty → try traces (per-request data) or logs (event records)
• Logs empty → try traces (structured span attributes) or metrics (aggregated counters)
• Traces empty → try logs (text-based debug output)

Do not stop after one failed attempt. Try at least two pillars before concluding the data does not exist.

---

CLI Commands Reference

---

Examples

Example 1: Business Question (Ambiguous Source)

Question: "How much money did people spend on the platform last week?"

Approach:

1. Search metrics: cx metrics search --name 'revenue' and cx metrics search --name 'transaction'
2. Search log fields: cx search-fields "transaction amount" --dataset logs
3. Search span fields: cx search-fields "payment total" --dataset spans
4. If a metric like payment_total_usd exists, load metrics references and run a range query
5. If only logs have the data, load logs references and use DataPrime aggregation
6. If traces have purchase.amount attribute, load spans references

Example 2: Latency Question (Clear First Choice)

Question: "What's the average latency of the checkout route?"

Approach:

1. First try metrics: cx metrics search --name 'checkoutlatency' or cx metrics search --name 'httpduration'
2. If a histogram metric exists, load metrics references and use histogram_quantile
3. If no metric, fall back to traces: load spans references and aggregate span durations

Example 3: Frontend Performance (RUM)

Question: "Why is the dashboard page loading slowly for users?"

Approach:

1. This is clearly a RUM question - load references/rum-querying.md + references/rum-fields.md + references/logs-querying.md + references/dataprime-reference.md
2. Query web vitals and page load times
3. If RUM shows backend calls are slow, pivot to spans references for the API calls

Example 4: Error Investigation (Logs + Traces)

Question: "Why are users getting 500 errors on the payment endpoint?"

Approach:

1. Check error rate metrics → load metrics references
2. Search for error logs → load logs references
3. Get traces for failed requests → load spans references
4. Cross-reference: find trace IDs in logs, then fetch full traces for root cause

---

Beyond Investigation

Not every question is answered by querying data. If the user's intent is operational rather than investigative, route to the appropriate workflow skill:

---

Update Notifications

When cx runs in --output agents mode, the output may include a trailing {"_meta": {"update": {...}}} JSON object. Check for it after every command:

{"_meta": {"update": {
  "binary": { "current": "1.2.0", "latest": "1.3.0", "command": "brew upgrade cx" }
}}}

If _meta.update is present, tell the user:

"A newer version of cx is available. Run <command> to upgrade."

Then offer to run the suggested command on their behalf. The same block appears in cx schema output so you can check it at startup.

---

Key Principles

• Load references before querying: check the Loading References table first
• Discover before querying: always run search/discovery to find the right source
• Parallel discovery: for ambiguous questions, search metrics, logs, and spans concurrently
• Validate with code: when unsure what a metric or field represents, check the codebase
• Pivot on failure: if one pillar is empty, try another before giving up
• Surface update notices: if _meta.update appears in any command output or in cx schema, inform the user and offer to run the upgrade command