We provide some tooling to make Sourcegraph.com instance easier to monitor and observe.

For general observability development, please refer to the observability development documentation instead, which includes links to useful how-to guides.

[data:image/svg+xml;base64,PHN2ZyBjbGFzcz0ib2N0aWNvbiBvY3RpY29uLWluZm8gbXItMiIgdmlld2JveD0iMCAwIDE2IDE2IiB2ZXJzaW9uPSIxLjEiIHdpZHRoPSIxNiIgaGVpZ2h0PSIxNiIgYXJpYS1oaWRkZW49InRydWUiPjxwYXRoIGQ9Ik0wIDhhOCA4IDAgMSAxIDE2IDBBOCA4IDAgMCAxIDAgOFptOC02LjVhNi41IDYuNSAwIDEgMCAwIDEzIDYuNSA2LjUgMCAwIDAgMC0xM1pNNi41IDcuNzVBLjc1Ljc1IDAgMCAxIDcuMjUgN2gxYS43NS43NSAwIDAgMSAuNzUuNzV2Mi43NWguMjVhLjc1Ljc1IDAgMCAxIDAgMS41aC0yYS43NS43NSAwIDAgMSAwLTEuNWguMjV2LTJoLS4yNWEuNzUuNzUgMCAwIDEtLjc1LS43NVpNOCA2YTEgMSAwIDEgMSAwLTIgMSAxIDAgMCAxIDAgMloiIC8+PC9zdmc+](data:image/svg+xml;base64,PHN2ZyBjbGFzcz0ib2N0aWNvbiBvY3RpY29uLWluZm8gbXItMiIgdmlld2JveD0iMCAwIDE2IDE2IiB2ZXJzaW9uPSIxLjEiIHdpZHRoPSIxNiIgaGVpZ2h0PSIxNiIgYXJpYS1oaWRkZW49InRydWUiPjxwYXRoIGQ9Ik0wIDhhOCA4IDAgMSAxIDE2IDBBOCA4IDAgMCAxIDAgOFptOC02LjVhNi41IDYuNSAwIDEgMCAwIDEzIDYuNSA2LjUgMCAwIDAgMC0xM1pNNi41IDcuNzVBLjc1Ljc1IDAgMCAxIDcuMjUgN2gxYS43NS43NSAwIDAgMSAuNzUuNzV2Mi43NWguMjVhLjc1Ljc1IDAgMCAxIDAgMS41aC0yYS43NS43NSAwIDAgMSAwLTEuNWguMjV2LTJoLS4yNWEuNzUuNzUgMCAwIDEtLjc1LS43NVpNOCA2YTEgMSAwIDEgMSAwLTIgMSAxIDAgMCAxIDAgMloiIC8+PC9zdmc+)

Looking for how to monitor Sourcegraph? See the observability documentation.

Metrics and alerting

For metrics and alerting, see the Sourcegraph monitoring guide.

Logging

Service logs are available in GCP logging in the sourcegraph-dev project. The quick-and-easy way is to go to the GCP console workloads page, select the workload of interest, and head over to the "Logs" tab.

Sourcegraph service logs follow a standardized JSON format - you can use this Logs Explorer view which is preconfigured with important attributes extracted to the log summary line, and uncomment the labels.k8s-pod/app filter to target your workload of choice. The resulting log filter should look something like this:

labels.k8s-pod/app="sourcegraph-frontend"
resource.type="k8s_container"
resource.labels.project_id="sourcegraph-dev"
resource.labels.location="us-central1-f"
resource.labels.cluster_name="cloud"
resource.labels.namespace_name="prod"

You can also use kubectl to work with service log output in the command line - see the Kubernetes guide to get started.

Tracing

Traces are available in Cloud Trace and an in-cluster Jaeger deployment. The latter is only accessible with site admin permissions - see Site-admin access to internal instances.

Trace spans meeting certain criteria are also exported to Honeycomb via our OpenTelemetry Collector deployment - see otel-collector.ConfigMap.yaml for our current configuration.

Also refer to how to use traces.

Cloudflare

Cloudflare Analytics is used to extract useful data about the performance of our WAF, as well as the overall traffic distribution to our instances. Note that the retention of analytics data is relatively short due to the limits on our plan.

This section gives a quick overview of how to access Cloudflare analytics, and how to interface with their GraphQL API. Note that in most cases, you'll be able to get much richer metrics by accessing our ‣ on our own internal monitoring.

GraphQL API

Cloudflare Analytics provides a somewhat limited API for retrieving monitoring data. Note that you can only retrieve relatively recent data, and have a limited number of operations.