Skip to main content
Version: 3.16

Tekton Long-Term Log Storage

Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller. This allows you to:

  • Provide custom Results metadata about your CI/CD workflows not available in the Tekton TaskRun/PipelineRun CRDs (for example: post-run actions).
  • Group related workloads together (e.g. bundle related TaskRuns and PipelineRuns into a single unit).
  • Make long-term result history independent of the Pipeline CRD controller, letting you free up etcd resources for Run execution.
  • Store logs produced by the TaskRuns/PipelineRuns so that completed Runs can be cleaned to save resources.

Long-Term Log Access Workflow​

The following diagram illustrates the workflow for accessing long-term logs for pipelines in the KubeRocketCI Portal:

Install Tekton Results​

Tekton Results is deployed as part of the Tekton Pipelines installation. For installing Tekton Pipelines, we recommend using the add-ons approach. Here's an example of how to configure values.yaml before installation:

tekton:
    createNamespace: true
    enable: true
    namespace: tekton-pipelines

Configuration​

note

We use Postgres operator to connect to Tekton results, but you can use any other external databases supported. Please align configuration accordingly to Tekton result documentation.

To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:

  1. Storage sizes in the results-pg.yaml file define the allocated volumes for the Tekton Results PostgreSQL cluster:

    • Database instance storage

      • Path: spec.instances.dataVolumeClaimSpec.resources.requests.storage
      • Default: 2Gi
      • Description: Specifies the size of the main PostgreSQL data volume.

    • Backup repository storage

      • Path: spec.backups.pgbackrest.repos.volume.volumeClaimSpec.resources.requests.storage
      • Default: 2Gi
      • Description: Specifies the size of the pgBackRest backup repository volume.

  2. Storage and retention settings in the results.yaml file define how Tekton Results result records are retained in the database and how related storage is configured:

    • Log storage size

      • Path: PersistentVolumeClaim.spec.resources.requests.storage
      • Default: 5Gi
      • Description: Specifies the allocated disk size for storing pipeline logs.

    • Retention period

      • Path: ConfigMap.data.defaultRetention in tekton-results-config-results-retention-policy
      • Default: "30" days
      • Description: Defines how long log files are retained before being automatically removed.

    • Cleanup schedule

      • Path: ConfigMap.data.runAt in tekton-results-config-results-retention-policy
      • Default: "0 18 * * *0*"
      • Description: Specifies the cron expression that determines when automated log cleanup runs (daily at 6:00 PM UTC).
warning

The retention policy agent removes only database records from PostgreSQL. It does not delete associated log files stored on persistent volumes, S3, or GCS backends. As a result, log data may remain on disk and consume storage even after the corresponding records are expired.

  1. To remove physical log files from PVCs (not just database records), use the settings in the results-clean-old-logs-cronjob.yaml file:

    • Cleanup schedule

      • Path: spec.schedule
      • Default: "0 18 * * *"
      • Description: Specifies the cron expression that determines when the CronJob runs (daily at 6:00 PM UTC).

    • Physical log retention (days)

      • Path: spec.jobTemplate.spec.template.spec.containers.command (the -mtime +N value in the find command)
      • Default: 30 days
      • Description: Physical log files in /tekton-results/logs/ older than this many days are deleted from the PVC. The value is the number in -mtime +N (e.g. +30 keeps files for 30 days).
note

In this example, the PGO (PostgreSQL Operator) is used for the Tekton Results database.

Authentication and TLS​

From KubeRocketCI 3.16.1, the results.yaml manifest turns on authentication and TLS for the Tekton Results API (tekton-results-api-service in tekton-pipelines, port 8080):

  • Authentication. The API validates each bearer token (TokenReview) and authorizes each request with Kubernetes RBAC on the results.tekton.dev API group, per namespace (SubjectAccessReview). The tekton-results-readonly ClusterRole grants read access to results, records, logs, and summaries, and aggregates to the view and edit ClusterRoles.
  • TLS. cert-manager issues the certificates, so it must be installed first. A self-signed CA (tekton-results-ca, valid for 5 years) signs the 1-year serving certificate in the tekton-results-tls Secret, which holds tls.crt, tls.key, and ca.crt. The watcher connects over TLS with its ServiceAccount token. The metrics port 9090 stays plain HTTP.
warning

Clients that call the API over http:// get 400 Client sent an HTTP request to an HTTPS server. Until the portal is configured as shown below, it shows no pipeline history, logs, or metrics from Tekton Results.

Configure the Portal for the Tekton Results API​

With an https:// URL, the portal authenticates to the API with its own ServiceAccount token, never with the token of the signed-in user.

  1. Copy the public CA certificate to a ConfigMap in the platform namespace:

    kubectl -n tekton-pipelines get secret tekton-results-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
    kubectl -n <platform-namespace> create configmap tekton-results-ca --from-file=ca.crt=ca.crt
    warning

    Copy only ca.crt. Never copy the tekton-results-ca-key-pair Secret: it holds the CA private key.

  2. Set the portal values under the krci-portal key of the edp-install values. If you already set volumes or volumeMounts, add these items to your lists:

    edp-install values.yaml
    krci-portal:
      configEnv:
        TEKTON_RESULTS_URL: https://tekton-results-api-service.tekton-pipelines.svc:8080
        NODE_EXTRA_CA_CERTS: /etc/tekton-results-ca/ca.crt
      volumes:
        - name: tekton-results-ca
          configMap:
            name: tekton-results-ca
      volumeMounts:
        - name: tekton-results-ca
          mountPath: /etc/tekton-results-ca
          readOnly: true
      tektonResults:
        rbac:
          enabled: true

    tektonResults.rbac.enabled binds the portal ServiceAccount to tekton-results-readonly in the platform namespace. The chart fails to render if the ServiceAccount is default or serviceAccount.automount is false; keep the default true.

Limitations:

  • The portal reads Tekton Results of the platform namespace only.
  • Every portal user can read the Tekton Results of the platform namespace through the portal. For the access model, see Sensitive Capabilities.

Certificate Renewal​

  • cert-manager renews the serving certificate about every 11 months, but the API and the watcher read it only at startup. After each renewal, restart both:

    kubectl -n tekton-pipelines rollout restart deploy/tekton-results-api deploy/tekton-results-watcher

  • cert-manager renews the CA at about two-thirds of its lifetime with a new private key. The serving certificate keeps the old CA until it is issued again. After a CA renewal, issue the serving certificate again, restart the API and the watcher, then update the portal ConfigMap and restart the portal:

    kubectl -n tekton-pipelines delete secret tekton-results-tls
    kubectl -n tekton-pipelines wait certificate tekton-results-serving-cert --for=condition=Ready --timeout=120s
    kubectl -n tekton-pipelines rollout restart deploy/tekton-results-api deploy/tekton-results-watcher
    kubectl -n tekton-pipelines get secret tekton-results-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
    kubectl -n <platform-namespace> create configmap tekton-results-ca --from-file=ca.crt=ca.crt \
      --dry-run=client -o yaml | kubectl apply -f -
    kubectl -n <platform-namespace> rollout restart deploy/krci-portal

Database Performance Indexes​

Tekton Results creates its schema through GORM auto-migration, which adds no indexes beyond the primary keys. As the history grows, the queries behind the KubeRocketCI Portal pipeline run history fall back to sequential scans and the history views become progressively slower to load. Create the indexes below to keep those queries on index scans.

note

CREATE INDEX CONCURRENTLY builds the index without locking the table for writes, so these statements are safe to run against a live database. They cannot run inside a transaction block, which is why each statement is applied separately.

  1. Find the primary PostgreSQL pod:

    PG_POD=$(kubectl get pods -n tekton-pipelines \
      -l postgres-operator.crunchydata.com/cluster=results,postgres-operator.crunchydata.com/role=master \
      -o jsonpath='{.items[0].metadata.name}')

  2. Create the index covering list queries that filter by parent and type and sort by time:

    kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \
      psql -U postgres -d results -c \
      "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_records_parent_type_created ON records (parent, type, created_time DESC);"

  3. Create the index covering JSONB containment queries (@>) used for codebase and name filtering:

    kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \
      psql -U postgres -d results -c \
      "CREATE INDEX CONCURRENTLY IF NOT EXISTS results_annotations ON results USING GIN (annotations jsonb_path_ops);"

  4. Create the index covering summary annotation filtering, which the DORA metrics views rely on:

    kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \
      psql -U postgres -d results -c \
      "CREATE INDEX CONCURRENTLY IF NOT EXISTS record_summary_annotations ON results USING GIN (recordsummary_annotations jsonb_path_ops);"

  5. Verify that the indexes exist:

    kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \
      psql -U postgres -d results -c "\di+"

  6. Confirm that queries use them. The plan should report an Index Scan or Bitmap Index Scan on idx_records_parent_type_created rather than a Seq Scan:

    kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \
      psql -U postgres -d results -c "
        EXPLAIN ANALYZE
        SELECT * FROM records
        WHERE parent LIKE 'tekton-pipelines/results/%'
          AND type = 'tekton.dev/v1.PipelineRun'
        ORDER BY created_time DESC
        LIMIT 10;
      "

To remove the indexes, drop them by name:

kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \
  psql -U postgres -d results -c "
    DROP INDEX IF EXISTS idx_records_parent_type_created;
    DROP INDEX IF EXISTS results_annotations;
    DROP INDEX IF EXISTS record_summary_annotations;
  "
Visit Main Site