## Summary
- Part of #213293
- Adds infrastructure for a new pipeline: `kibana / renovate helper`
- The pipeline and scripts are currently a no-op and will be finalized
and tested in a follow up PR against the provisioned pipeline.
## Plan
The goal is to automatically run a script when Renovate opens a PR to
dedupe and perform any other actions, then trigger the Kibana PR
pipeline. This can also be performed by commenting `buildkite run
renovate helper` in the instances where the PR has been rebased by
Renovate.
Additionally, this pipeline will trigger the Kibana PR pipeline
automatically for Renovate PRs when the commit author is not Renovate
bot. Currently a developer has to comment every time they want CI to run
on Renovate PRs. This will improve DX when merging upstream, debugging
the dep upgrade and auto-commits by `kibanamachine`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
With https://github.com/elastic/kibana/pull/207036 we introduced some
changes in the script so that it generates two PRs: one for the `main`
branch and one for `8.x` in order to keep previous versions with their
corresponding console definitions.
This PR updates the branches so the script generates PRs for `main`,
`9.0`, `8.19` and `8.18`
## Summary
Closes https://github.com/elastic/kibana/issues/38277
This PR ~attempts to take a stab at automating~ automates the process
for building the linux headless chromium used for reporting within
Kibana.
The Idea here is that one would simply only need create an issue within
the Kibana project that will then trigger the build for chromium, we'll
be leveraging GitHub actions for this, the Github workflow defined only
kicks in when an issue is created with the label
`trigger-chromium-build`, when such an issue is created, the body of the
issue is analysed; there's an expectation that the body would contain a
fenced codeblock of type json that specifies the property
`puppeteer_version`, if this is found this value is read and, we then
trigger a build on buildkite passing along this value, in the case where
there are multiple fenced code blocks specifying the expected property
`puppeteer_version` we leave a message that the issue body be modified,
also if the expected label was added but the expected property is not
found we also leave a message prompting that the expected property be
provided.
Once the build has commenced a message containing the link to the build
would be provided as initial feedback to the user, also on completion
another message is provided to the user, that provides a link to the PR
containing all the required changes to update puppeteer to the version
specified.
~It's also worth pointing out that this PR also, modifies the source for
`win`, `mac` chromium binaries to leverage the [JSON API
endpoint](https://github.com/GoogleChromeLabs/chrome-for-testing#json-api-endpoints)
provided by Google to get the required binaries for chromium headless in
a deterministic way, which in turns is part of what makes this
automation possible.~
## How to test this
- If you'd love to test the Github action too without the PR being
merging in just yet, you should consider setting up
[act](https://github.com/nektos/act), alongside it's companion [vscode
extension](https://sanjulaganepola.github.io/github-local-actions-docs/)
, we'll then want to create a payload file providing similar data that
github would return for our workflow trigger, more info about setting
this up
[here](https://sanjulaganepola.github.io/github-local-actions-docs/usage/settings/#payloads).
The payload file we'd want would be something along the following lines;
```json
{
"action": "labeled",
"label":{
"name": "trigger-chromium-build"
},
"issue": {
"number": 1,
"title": "Issue 1",
"author_association": "MEMBER",
"labels": [
{
"name": "trigger-chromium-build"
}
],
"body": "\n## Random text \n\n ~~~json\n{\n \"puppeteer_version\": \"24.6.1\" \n}\n~~~\n~~~json\n{\n \"some_random_value\": \"23.0.1\" \n}\n~~~"
}
}
```
- To test the actual build process it can be initiated through this
specific pipeline
https://buildkite.com/elastic/kibana-migration-pipeline-staging by
creating a custom build on `pull/212674/head` (this pull request) with
the env variable similar to this
```
TESTED_PIPELINE_PATH=.buildkite/pipelines/chromium_linux_build/build_chromium.yml
PUPPETEER_VERSION=24.6.1
GITHUB_ISSUE_NUMBER=212732
GITHUB_ISSUE_BASE_OWNER=elastic
GITHUB_ISSUE_BASE_REPO=kibana
GITHUB_ISSUE_TRIGGER_USER=eokoneyo
```
PS: Issue #212732 is an issue that's been created to test providing
feedback to the user about the build process, re-triggering a build on
an existing issue updates the comments in place.
<!-- ### Where to go from here?
- Ideas and thoughts welcome
-->
<!--
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Identify risks
Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.
- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
-->
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Introduces a new `security_solution/gen_ai_evals.yml` BuildKite pipeline
for automatically running our Assistant and Attack Discovery evaluation
suites weekly.
### To Run Locally:
Ensure you are authenticated with vault for LLM + LangSmith creds:
> See [internal
docs](https://github.com/elastic/infra/blob/master/docs/vault/README.md#login-with-your-okta)
for setup/login instructions.
Fetch Connectors and LangSmith creds:
> [!NOTE]
> In discussion with @elastic/kibana-operations it was preferred to use
the ci-prod secrets vault, so we cannot self-manage the secrets. To test
this locally though, you can grab the secrets and follow the
instructions in this [paste
bin](https://p.elstc.co/paste/q7k+zYOc#PN0kasw11u2J0XWC2Ls5PMNWreKzKTpgWA1wtsPzeH+).
```
cd x-pack/test/security_solution_api_integration
node scripts/genai/vault/retrieve_secrets.js
```
Navigate to api integration directory, load the env vars, and start
server:
```
cd x-pack/test/security_solution_api_integration
export KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)
yarn genai_evals:server:ess
```
Then in another terminal, load vars and run the tests:
```
cd x-pack/test/security_solution_api_integration
export KIBANA_SECURITY_TESTING_AI_CONNECTORS=$(base64 -w 0 < scripts/genai/vault/connector_config.json) && export KIBANA_SECURITY_TESTING_LANGSMITH_KEY=$(base64 -w 0 < scripts/genai/vault/langsmith_key.txt)
yarn genai_evals🏃ess
```
### To manually run on BuildKite:
Navigate to
[BuildKite](https://buildkite.com/elastic?filter=ftr-security-solution-gen-ai-evaluations)
and run `ftr-security-solution-gen-ai-evaluations` pipeline.
### To manually run on BuildKite for specific PR:
In `.buildkite/ftr_security_stateful_configs.yml`, temporarily move the
`genai/evaluations/trial_license_complete_tier/configs/ess.config.ts`
line down to the `enabled` section. Will see if we can do this without
requiring a commit. @elastic/kibana-operations is it possible to set a
buildkite env var that can be read in FTR tests when a specific GitHub
label is added to the PR? I.e. can I create a `SecurityGenAI:Run Evals`
label that when added will run this suite as part of the build?
> [!NOTE]
> Currently the connectors secrets only include `gpt-4o` and
`gpt-4o-mini`. Waiting on finalized list w/ credentials from @jamesspi
and @peluja1012 and then we can have ops update using the scripts
included in this PR.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <patryk.kopycinski@elastic.co>
This PR prepares the changes needed for removing 8.x branch and create
8.19.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
When VM image rebuild is triggered after ES promotion, only the cache
warmup should be built.
This PR also separates the daily full build to a daily base + cache
build (in case ES promotions are failing for some reason, we should
still have a daily cache refresh).
Requires: https://github.com/elastic/ci-agent-images/pull/1295
With this, we'd run a daily base image build and cache build (~40m +
25m) + cache warmups for every promotion (~4x 25m) instead of a full
build and promotion per build (~4x 55m). Ultimately not that much of a
gain 🤷 (4*55=220m => 40+5x25=165m)
## Summary
In #210425 we added Scout UI tests to be run for on-merge pipeline, but
test results are not ingested by default.
This PR enables Scout reporter for that pipeline
## Summary
This PR adds quick-checks regarding pipeline resource definitions.
- Adds validation/correction for `locations.yml`
- Adds validation for definitions (through
`docker.elastic.co/ci-agent-images/pipelib rre validate`).
- Also fixes a missing pipeline entry in `locations.yml`
Example for when a YML error bounces on schema validation:
https://buildkite.com/elastic/kibana-pull-request/builds/270652#0194a8d5-48df-4f1b-8ec7-1f076851d138
Example for auto-fixing missing location:
b2170ac53a
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
- Centralized Scout reporter settings
- Added owner area and config/test file information to reporter events
- Attempt to upload events at the end of a test run
- Enable Scout reporter test events upload for the `pull request` and
`on merge` pipelines
## Summary
Similar to https://github.com/elastic/kibana/pull/195581
Adds a pipeline that builds Kibana and starts cloud deployment without
going through the CI test suites (as in normal pull-request pipeline
runs). It can be useful if a developer would like to save time/compute
on re-building/re-testing the whole project before deploying to the
cloud.
Added labels (`ci:cloud-deploy / ci:cloud-redeploy`) are required
similarly to the usual CI flow.
Related to: https://github.com/elastic/kibana-operations/issues/121
https://github.com/elastic/kibana/pull/199071 introduced a daily
pipeline to run acceptance tests against the latest EPR image and to
promote the image if successful.
This updates our tests to start using the daily promotion except when
running the acceptance tests.
Previously reverted due to the number of statuses we were tracking, and
the subsequent rate limiting.
The number of updates we need to do after
https://github.com/elastic/kibana/pull/199073 is significantly lower.
We've already been using org-wide status checks for [on-merge commit
statuses](https://github.com/elastic/kibana/commits/main) in the interim
and are not seeing issues.
This adds a daily build that runs tests again
`docker.elastic.co/package-registry/distribution:lite` and if
successful, promotes the image to
`docker.elastic.co/kibana-ci/package-registry-distribution:lite`.
A follow up pull request will swap integration tests over to use the
promoted image, and add environment overrides to run tests against the
source image in this pipeline.
https://buildkite.com/elastic/kibana-package-registry-promote/builds/2
The number of steps in pull request builds has been causing GitHub API
rate limit issues. In particular, scenarios that cause all steps to fail
have proven to quickly trigger the rate limit.
The disables step statuses on pull requests. We will still have our
required kibana-ci check for the overall build, and the pull request
comment can be used as the source of individual step failures.
Adds a placeholder pipeline for `kibana / package registry promote`.
Initially, in a follow up PR, this will run a daily promotion of
`docker.elastic.co/package-registry/distribution:lite` to the kibana-ci
namespace. We can also run some verification steps if desired.
The distribution is a relatively large image, and nearly always running
uncached on CI due to the update frequency. This should help us balance
having an up to date image and avoiding cache misses.
## Summary
In #195581 we've added the option to deploy through the clickable
triggers. But in it's current state, it's broken in several aspects.
(1) It's not starting on click. Triggers was resulting in a 422 on
Buildkite's side, and after digging more into it, this was the error:
<img width="1019" alt="Screenshot 2024-10-16 at 16 53 13"
src="https://github.com/user-attachments/assets/f602dde9-2cc4-474f-b432-a3d4f9d5ae91">
Apparently, building PRs needs to be enabled on jobs that want to be
triggered through the PR bot.
(2) It is set up to run regardless of the labels
(3) There's no feedback on runs
## Changes
This PR:
- enables buildability in the pipeline's config
- exits early if deploy labels are missing
- adds a comment on the PR if a deploy job is started or finished
- removes the kibana build step, it's not needed, as we have a step to
build the docker image
TODO:
- [x] Add feedback about a started job (either through a non-required
check, or a github comment)
- [x] Early exit if a label is missing
There are several other builds started right now, because the logic that
would trigger a build on changing a draft to ready. To be fixed in
https://github.com/elastic/buildkite-pr-bot/issues/78
Tested after manually by enabling the option on the UI, and triggering
through the checkbox:
https://buildkite.com/elastic/kibana-deploy-project-from-pr/builds/23
The schedule adjustment is due to 7.17 verification taking longer than
other branches. Currently, we only rebuild once when main is updated.
This is a stopgap until a cache-only update can be introduced to avoid
continuous image rebuilds.
## Summary
There's a use-case where developers would like to work while testing
their deployments in a serverless environment, or they'd like to
redeploy only, without running the whole suite of tests before. Labels
are still required because project deployment requires the project kind
to be prescribed.
This PR adds the instruments to do so:
- pipeline resource definition to create a new pipeline
- pipeline implementation, that only builds and deploys
- `pull-requests.json` configuration, so the option shows up in the
'click to xy' section
With this, we present a cheaper, faster way to deploy serverless when
needed.
Closes: https://github.com/elastic/kibana-operations/issues/121
---------
Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>
