Commit graph

1238 commits

Author SHA1 Message Date
Jean-Louis Leysens
975eeed255
[HTTP/OAS] Commit OAS snapshot (#183338)
Close https://github.com/elastic/kibana/issues/181992

## Summary

First iteration of a CLI to capture an OAS snapshot.

## How to test

Run `node ./scripts/capture_oas_snapshot.js --update --include-path
/api/status` and see result in `oas_docs/bundle.json`.

If you have the [bump CLI](https://www.npmjs.com/package/bump-cli)
installed you can preview the hosted output with `bump preview
./oas_docs/bundle.json`

## Notes
* Added ability to filter by `version`, `access` (public/internal) and
excluding paths explicitly to the OAS generation lib
* Follows the same general pattern as our other "capture" CLIs like
`packages/kbn-check-mappings-update-cli`
* Result includes only `/api/status` for now, waiting for other paths to
add missing parts

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-30 06:02:19 -07:00
Khristinin Nikita
aa7ffc499f
Initial backfull group (#184144)
## Show backfill group for manual runs 

UX - copy is not final, there will be an additional ticket for that,
which we address later

After user execute manual rule API we will present backfill group.
Backfill group contains 1 to N scheduled entries. Each of the entry - is
associated with rule run (potential, in progress, completed)
Backfill group is removed after all task completed, so this why they
disappear from UI in the video



- Show amount of tasks
- Ability to cancel run
- There auto refresh - which is disabled by default, as backfills group
remove after completion



35823ac5-de20-4082-819f-030ccc524e82


### How to test 

1 . Enable feature flag - `manualRuleRunEnabled`
2. For you rule call schedule api
`/internal/alerting/rules/backfill/_schedule` `POST`
With this body (put your values for rule id and date range):
```
[{"rule_id":"58b4b926-6348-4c23-be1f-870a461fa342","start":"2024-05-21T13:00:00.000Z","end":"2024-05-21T14:05:00.000Z"}]
```

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-05-29 20:44:14 +02:00
Joe Reuter
c766dd33e2
Observability onboarding: Adjust codeowners (#184303)
This PR adjusts the codeowners to allow code reviews within the virtual
observability onboarding plugin
2024-05-29 10:15:32 -07:00
Yngrid Coello
d39739ee85
[Dataset quality] Move page to stack management > Data (#184122)
Relates to https://github.com/elastic/kibana/issues/183406.

## 📝  Summary
This PR creates a new plugin `data_quality` in order to register dataset
quality as a Stack management page under data section. For now there is
no reference to this new page in the sideNav in stateful or serverless.

In order to navigate to this new page you can use the url
`/app/management/data/data_quality`

Changes included in this PR:
- New plugin created
- Plugin registered in stack management, data section
- Dataset quality plugin is instantiated and the state is in sync with
URL
- Removed references to dataset quality in Logs explorer

## 🎥 Demo


501c9c47-4a1b-4f91-9be6-d022a821e88e

## 🙅🏼 Missing

- Dataset quality locator
- There are still references to logs explorer (table and flyout) that
will be handled in a follow up PR.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-29 07:06:43 -07:00
Sébastien Loix
9e2b401f2c
[Stateful sidenav] Register observabilty nav from inside plugin (#184215) 2024-05-29 03:36:59 -07:00
Drew Tate
cf1ff97c12
[ES|QL] add function definition sync to CI (#184225)
## Summary

Runs the script added in https://github.com/elastic/kibana/pull/179584
on the weekly schedule.

Successful run:
https://buildkite.com/elastic/kibana-es-ql-grammar-sync/builds/21#018fab3b-d051-4852-a03e-889cf156fb70/838

---------

Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
2024-05-28 11:02:20 -05:00
Faisal Kanout
4396bf6e2e
[OBS-UX-MNGMT] Move the Alerting comparators from TriggersActionsUI plugin to the alerting-types package (#181584)
## Summary
 It fixes #179633

Observability created a Comparator type/enum, when ResponseOps is
already exporting one and other rules using it.
The only difference is the wording of not in between [I put the two
types side by side to compare]
Currently, we import the one in triggers-actions-ui-plugin , and then
update the not in between to match our Comparator.

### Comparing the two enums:
![Screenshot 2024-04-23 at 18 17
23](16429ff9-e672-4c16-92ed-488a2f66007d)

## For reviewers 🧪 
- Everything should work as expected: Alert flyout, Alert reason
message, Rule creation flyout, etc.
- I kept the `outside` comparator (replaced by `NOT BETWEEN`) for
backward compatibility
2024-05-28 15:34:52 +02:00
Umberto Pepato
df15fe1b06
[ResponseOps][Security Solution] Transfer ownership of the alerts grouping package to ResponseOps (#184131)
## Summary

As agreed in the recent ResponseOps/Security/Observability meeting, the
ResponseOps team will take ownership of the alerts grouping package
(formerly `@kbn/securitysolution-grouping`) in order to provide a
solution-agnostic API surface for future usages in Observability and
Stack rules.

This PR implements this transfer, while also renaming the package to
`@kbn/grouping` to reflect the change in scope.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-28 11:15:04 +02:00
Steph Milovic
e2e1fb3504
[Security solution] AI Assistant, replace LLM with SimpleChatModel + Bedrock streaming (#182041) 2024-05-22 16:29:14 -06:00
Sébastien Loix
5b38b5a6c0
[Stateful sidenav] Dynamic links (#183157) 2024-05-22 16:55:16 +01:00
Jon
295b4dbc15
Update codeowners (#184021)
To label failing tests.
2024-05-22 09:20:20 -05:00
Alison Goryachev
542a3cee00
Update codeowners file (#183552) 2024-05-22 08:19:50 -04:00
Jon
9ef9a8481b
Add workflow to trigger alert on failed test (#183741) 2024-05-21 18:57:37 -05:00
Chris Cowan
7ae07f8913
[EEM][POC] The POC for creating entity-centric indices using entity definitions (#183205)
## Summary

This is a "proof of concept" for generating entity-centric indices for
the OAM. This exposes an API (`/api/entities`) for creating "asset
definitions" (`EntityDefinition`) that manages a transform and ingest
pipeline to produce documents into an index which could be used to
create a search experience or lookups for different services.

### Features
- Data schema agnostic, works with known schemas OR custom logs
- Supports defining multiple `identityFields` along with an
`identityTemplate` for formatting the `asset.id`
- Supports optional `identityFields` using `{ "field": "path-to-field",
"optional": true }` definition instead of a `string`.
- Supports defining key `metrics` with equations which are compatible
with the SLO product
- Supports adding `metadata` fields which will include multiple values. 
- Supports `metadata` fields can be re-mapped to a new destination path
using `{ "source": "path-to-source-field", "limit": 1000, "destination":
"path-to-destination-in-output" }` definition instead of a `string`
- Supports adding  `staticFields` which can also use template variables
- Support fine grain control over the frequency and sync settings for
the underlying transform
- Installs the index template components and index template settings for
the destination index
- Allow the user to configure the index patterns and timestamp field
along with the lookback
- The documents for each definition will be stored in their own index
(`.entities-observability.summary-v1.{defintion.id}`)

### Notes
- We are currently considering adding a historical index which will
track changes to the assets over time. If we choose to do this, the
summary index would remain the same but we'd add a second transform with
a group_by on the `definition.timestampField` and break the indices into
monthly indexes (configurable in the settings).
- We are looking into ways to add `firstSeenTimestamp`, this is a
difficult due to scaling issue. Essentially, we would need to find the
`minimum` timestamp for each entity which could be extremely costly on a
large datasets.
- There is nothing stopping you from creating an asset definition that
uses the `.entities-observability.summary-v1.*` index pattern to create
summaries of summaries... it can be very "meta".

### API
- `POST /api/entities/definition` - Creates a new asset definition and
starts the indexing. See examples below.
- `DELETE /api/entities/definition/{id}` - Deletes the asset definition
along with cleaning up the transform, ingest pipeline, and deletes the
destination index.
- `POST /api/entities/definition/{id}/_reset` - Resets the transform,
ingest pipeline, and destination index. This is useful for upgrading
asset definitions to new features.

## Example Definitions and Output

Here is a definition for creating services for each of the custom log
sources in the `fake_stack` dataset from `x-pack/packages/data-forge`.

```JSON
POST kbn:/api/entities/definition
{
  "id": "admin-console-logs-service",
  "name": "Services for Admin Console",
  "type": "service",
  "indexPatterns": ["kbn-data-forge-fake_stack.*"],
  "timestampField": "@timestamp",
  "lookback": "5m",
  "identityFields": ["log.logger"],
  "identityTemplate": "{{log.logger}}",
  "metadata": [
    "tags",
    "host.name"
  ],
  "metrics": [
    {
      "name": "logRate",
      "equation": "A / 5", 
      "metrics": [
        {
          "name": "A",
          "aggregation": "doc_count",
          "filter": "log.level: *"
        }
      ]
    },
    {
      "name": "errorRate",
      "equation": "A / 5", 
      "metrics": [
        {
          "name": "A",
          "aggregation": "doc_count",
          "filter": "log.level: \"ERROR\""
        }
      ]
    }
  ]
}
```
Which produces:
```JSON
{
  "host": {
    "name": [
      "admin-console.prod.020",
      "admin-console.prod.010",
      "admin-console.prod.011",
      "admin-console.prod.001",
      "admin-console.prod.012",
      "admin-console.prod.002",
      "admin-console.prod.013",
      "admin-console.prod.003",
      "admin-console.prod.014",
      "admin-console.prod.004",
      "admin-console.prod.015",
      "admin-console.prod.016",
      "admin-console.prod.005",
      "admin-console.prod.017",
      "admin-console.prod.006",
      "admin-console.prod.018",
      "admin-console.prod.007",
      "admin-console.prod.019",
      "admin-console.prod.008",
      "admin-console.prod.009"
    ]
  },
  "entity": {
    "latestTimestamp": "2024-05-10T22:04:51.481Z",
    "metric": {
      "logRate": 37.4,
      "errorRate": 1
    },
    "identity": {
      "log": {
        "logger": "admin-console"
      }
    },
    "id": "admin-console",
    "indexPatterns": [
      "kbn-data-forge-fake_stack.*"
    ],
    "definitionId": "admin-console-logs-service"
  },
  "event": {
    "ingested": "2024-05-10T22:05:51.955691Z"
  },
  "tags": [
    "infra:admin-console"
  ]
}
```

Here is an example of a definition for APM Services:

```JSON
POST kbn:/api/entities/definition
{
  "id": "apm-services",
  "name": "Services for APM",
  "type": "service", 
  "indexPatterns": ["logs-*", "metrics-*"],
  "timestampField": "@timestamp",
  "lookback": "5m",
  "identityFields": ["service.name", "service.environment"],
  "identityTemplate": "{{service.name}}:{{service.environment}}",
  "metadata": [
    "tags",
    "host.name"
  ],
  "metrics": [
    {
      "name": "latency",
      "equation": "A",
      "metrics": [
        {
          "name": "A",
          "aggregation": "avg",
          "field": "transaction.duration.histogram"
        }
      ]
    },
    {
      "name": "throughput",
      "equation": "A / 5",
      "metrics": [
        {
          "name": "A",
          "aggregation": "doc_count"
        }
      ]
    },
    {
      "name": "failedTransRate",
      "equation": "A / B",
      "metrics": [
        {
          "name": "A",
          "aggregation": "doc_count",
          "filter": "event.outcome: \"failure\""
        },
        {
          "name": "B",
          "aggregation": "doc_count",
          "filter": "event.outcome: *"
        }
      ]
    }
  ]
}
```
Which produces:
```JSON
{
  "host": {
    "name": [
      "simianhacker's-macbook-pro"
    ]
  },
  "entity": {
    "latestTimestamp": "2024-05-10T21:38:22.513Z",
    "metric": {
      "latency": 615276.8812785388,
      "throughput": 50.6,
      "failedTransRate": 0.0091324200913242
    },
    "identity": {
      "service": {
        "environment": "development",
        "name": "admin-console"
      }
    },
    "id": "admin-console:development",
    "indexPatterns": [
      "logs-*",
      "metrics-*"
    ],
    "definitionId": "apm-services"
  },
  "event": {
    "ingested": "2024-05-10T21:39:33.636225Z"
  },
  "tags": [
    "_geoip_database_unavailable_GeoLite2-City.mmdb"
  ]
}
```

### Getting Started 

The easiest way to get started is to use the`kbn-data-forge` config
below. Save this YAML to `~/Desktop/fake_stack.yaml` then run `node
x-pack/scripts/data_forge.js --config ~/Desktop/fake_stack.yaml`. Then
create a definition using the first example above.

```YAML
---
elasticsearch:
  installKibanaUser: false

kibana:
  installAssets: true
  host: "http://localhost:5601/kibana"

indexing:
  dataset: "fake_stack"
  eventsPerCycle: 50
  reduceWeekendTrafficBy: 0.5

schedule:
  # Start with good events
  - template: "good"
    start: "now-1d"
    end: "now-20m"
    eventsPerCycle: 50
    randomness: 0.8
  - template: "bad"
    start: "now-20m"
    end: "now-10m"
    eventsPerCycle: 50
    randomness: 0.8
  - template: "good"
    start: "now-10m"
    end: false
    eventsPerCycle: 50
    randomness: 0.8
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-17 12:03:42 -06:00
Aleh Zasypkin
b306f007fb
Use spec-compliant URL parser to parse next URL parameter. (#183521)
## Summary

The Node.js URL parser [has been
deprecated](https://nodejs.org/api/url.html#urlparseurlstring-parsequerystring-slashesdenotehost)
and we should start slowly switching to the [spec-compliant URL
parser](https://url.spec.whatwg.org/#url-parsing) to parse URLs in
Kibana starting from the `next` query string parameter.

__Flaky Test Runner:__ [x65 for every added functional
test](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6010)
2024-05-17 13:39:52 +02:00
Marco Vettorello
0be35b3af7
Remove GH project assignment for closed project (#183658)
## Summary

The `Lens` [GH Project](https://github.com/elastic/kibana/projects/32)
is closed and we no longer use it.
2024-05-16 09:58:53 -07:00
Ash
3227e4c4eb
[Security Solution][Endpoint][SentinelOne] Consolidated agent status component (#181632)
## Summary

Adds an agent status component that shows agent isolation status, os and
last checkin info for endpoint/non-endpoint agents.

Updates agent status based on `agentStatusClientEnabled` ff for

1. Endpoint list
2. Endpoint details
3. Endpoint/SentinelOne alerts
4. Endpoint/SentinelOne alerts -> timeline view
5. Endpoint/SentinelOne responder headers
6. Hosts page -> Endpoint Host overview

<details><summary>Screenshots</summary>
<h4>1. Endpoint list</h4>
<img
src="259f3053-7834-4c06-8b91-6519486386ad"/>

<h4>2. Endpoint details</h4>
<img
src="4a04817a-fa6d-47a6-a938-a4b640f3d039"/>

<h4>3. Endpoint alert</h4>
<img
src="4d9a6975-bed9-49a4-b6d8-2d7685692180"/>

<h4>3. SentinelOne Alert</h4>
<img
src="3616fb49-afc9-4584-880f-fab1e50ea4ee"/>

<h4>4. Endpoint alert -> Timeline view -> details flyout </h4>
<img
src="7d2ead35-5401-4f01-9d5d-89b2e4ff9d9b"
/>

<h4>4. SentinelOne alert -> Timeline view -> details flyout </h4>
<img
src="edd1839a-18ef-44d8-8688-35c181527a29"
/>

<h4>5. Endpoint responder</h4>
<img
src="eaeb7293-84ad-4195-9126-eb1e330f822f"/>

<h4>5. SentinelOne responder</h4>
<img
src="8ea1a50a-53d2-40ed-aeb4-3620d4e9ebfc"/>

<h4>6.  Host Page -> Endpoint overview</h4>
<img
src="c443331a-0cd8-4fa1-a59c-b254dbaae6d9"/>
</details> 

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
2024-05-15 20:37:57 +02:00
Marco Antonio Ghiani
a5dcc10076
[Stack Monitoring] Update code owners (#183279)
## 📓 Summary

Update ownership of `monitoring` and `monitoring_collection` plugins to
`@elastic/stack-monitoring` team.

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
2024-05-14 09:57:07 +02:00
Rodney Norris
98329a8879
[Search] Move TryInConsoleButton to it's own package (#183201)
## Summary

Moved the TryInConsoleButton from the search api panels package to its
own package. This should make it easier to import and encourage usage in
more parts of Kibana even outside of search.
2024-05-13 14:34:41 -05:00
Drew Tate
73864cc61e
[ES|QL] add AST inspector to examples (#182720)
## Summary

Something to get us started on the AST fun. At least until we contribute
ES|QL support to https://github.com/fkling/astexplorer :)



82c482b7-cd61-4440-b723-84f863c1b596

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-10 13:47:06 -05:00
Angela Chuang
517e4a4d39
[SecuritySolution] Security Solution API integration test files relocation (#181605)
## Summary

https://github.com/elastic/kibana/issues/181683

This PR moves the existing Security Solution API integration tests from
`x-pack/test/api_integration/apis` to
`x-pack/test/security_solution_api_integration` and apply tags for each
scenario.
(x-pack/test/timeline is not included in this PR as this PR is already
big)

## Todo in the follow up PR:
move `x-pack/test/timeline` to
`x-pack/test/security_solution_api_integration` (as this PR is already
big)

## What to review?
1. Please review if the codeowner is assigned  correctly.
2. Please review if the test cases are still valid.

## How to run the tests: 

Here we use explore/hosts with trial license as an example:
```
cd ./x-pack/test/security_solution_api_integration
```


**Start ESS server:**
```
node ./scripts/index.js server explore trial_license_complete_tier hosts ess

```

When the server is started, open another terminal

```
cd ./x-pack/test/security_solution_api_integration

node ../../../scripts/functional_test_runner --config=test_suites/explore/hosts/trial_license_complete_tier/configs/ess.config.ts

```





**Start Serverless server:**
```
node ./scripts/index.js server explore trial_license_complete_tier hosts serverless

```

When the server is started, open another terminal

```
cd ./x-pack/test/security_solution_api_integration

node ../../../scripts/functional_test_runner --config=test_suites/explore/hosts/trial_license_complete_tier/configs/serverless.config.ts

```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-10 13:29:25 +01:00
Sergi Massaneda
a3de2d4141
[Security Solution] Timeline cell actions migration (#182092)
## Summary

issue: https://github.com/elastic/kibana/issues/181849

This PR finishes the migration from the old hover actions components to
the new cell-actions package in the Security Solution application, the
latest parts still using the legacy hover actions were the Timeline
draggable and the field renderer components.

Besides the migration and cleaning, this PR refactors the cell-actions
package exports and the Security Solution actions registration, to
improve the bundling size.

### Changes:

#### Refactoring

- `packages/kbn-cell-actions/` -> Refactored to separate the "UI"
exports (components, hooks, context) which are used by multiple
components, from the "actions" export which is only used to register the
cell actions' execution logic on the plugin start. This change reduced
bundle size.

- `x-pack/plugins/security_solution/public/actions/` -> Moved under
`public/app/actions`. This directory contains the custom actions
implementation and registration logic, the directory has been relocated
for consistency since it was at the sub-plugin (app sections) directory
level, the `/app` directory fits much better it's purpose. All the logic
inside remains unchanged.

#### Migration

-
`x-pack/plugins/security_solution/public/common/components/drag_and_drop/draggable_cell_actions.tsx`
-> New `DraggableCellActions` component implemented using the new
cell-actions, replaces the usages of the old `WithHoverActions`
component in the `DraggableWrapper`.

-
`x-pack/plugins/security_solution/public/common/components/hover_popover/index.tsx`
-> New `HoverPopover` generic component that replaces "unconventional"
usages of the old `WithHoverActions` component, it is decoupled from
cell-actions.

-
`x-pack/plugins/security_solution/public/common/components/drag_and_drop/draggable_wrapper.tsx`
->
Implementation updated to use the new cell-actions (via
`DraggableCellActions`) instead of the legacy hover_actions.

-
`x-pack/plugins/security_solution/public/timelines/components/field_renderers/field_renderers.tsx`
-> Implementation updated using the new cell-actions instead of the
legacy hover_actions. The `scopeId` prop has been added to the component
it needs to be provided to the new cell-actions. The `sourcererScopeId`
prop has been removed since it can be derived from the new `scopeId`
prop.

The UX should be the same, there's only one small change, for
consistency, in the "Add to timeline" action title:

| Old | New |
|-|-|

|![old](8e4fe2a1-92e9-4c77-8e51-253d822f7a2c)|

|![old_filter_in](22dead4b-0014-42f2-a243-2ce78e10d934)|

|![old_filter_out](d0dc7b2c-cf30-4e1d-9818-69e12c529df0)|

|![old_add_timeline](ec5d5bcb-5575-4aaa-a0ad-c987ed7b27a5)|

|![old_top_n](d191c9ba-b605-4b2c-b298-092db94c4b07)|

|![old_copy](ce8db1a9-a73a-4515-91c9-4f0fc1429f73)|

#### Replacement

-
`x-pack/plugins/security_solution/public/entity_analytics/components/severity/common/index.tsx`
-> Unconventional usage of `WithHoverActions` has been replaced by the
new `HoverPopover` component.

Host risk level field hover:


![risk_score](21049777-659f-486c-b515-f7e29448e540)

-
`x-pack/plugins/security_solution/public/overview/components/recent_timelines/recent_timelines.tsx`
-> Unconventional usage of `WithHoverActions` has been replaced by the
new `HoverPopover` component.

Recent timelines hover:


![recent_timelines](bc91dbe9-99c3-4138-a2cc-6d8d58b8c035)

#### Cleaning

Deprecated hover_actions directories removed:
-
`x-pack/plugins/security_solution/public/common/components/with_hover_actions/*`
-
`x-pack/plugins/security_solution/public/common/components/hover_actions/*`

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-09 16:30:13 +02:00
Søren Louv-Jansen
1798e7b7ab
Add Observability AI Assistant test suites to CODEOWNERS file (#182716)
This adds the Obs AI Assistant team as code owners for the api and
functional tests related to Obs AI Assistant
2024-05-08 10:38:20 +02:00
Cauê Marcondes
f5854558d3
[Logs] Adding data access plugin (#181823)
closes https://github.com/elastic/kibana/issues/181813

- Adds new logs_data_access plugin
- Adds Logs rates services that returns:
```
[serviceName]: {
  logRate: number | null;
  logErrorRate: number | null;
}
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-08 09:31:09 +01:00
Victor Martinez
c6ff9caa6f
CODEOWNERS: GitHub actions for obs-robots automation (#182796)
As discussed in
https://github.com/elastic/kibana/pull/182773#issuecomment-2097968561
2024-05-07 15:22:46 +02:00
Victor Martinez
72bf25dba6
github-action: use pull_request_target to access the secrets (#182773)
Use `pull_request_target` to access the GitHub secrets to destroy the
oblt-test-env created previously.

It's secured enough; it does not run any changes from any PR but does
some other automation
2024-05-07 12:16:18 +02:00
Jon
8b015ebedd
[keystore] Add password support (#180414)
This adds support a password protected keystore. The UX should match
other stack products.

Closes https://github.com/elastic/kibana/issues/21756.

```
[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% bin/kibana-keystore create --password
A Kibana keystore already exists. Overwrite? [y/N] y
Enter new password for the kibana keystore (empty for no password): ********
Created Kibana keystore in /tmp/kibana-8.15.0-SNAPSHOT/config/kibana.keystore

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% bin/kibana-keystore add elasticsearch.username
Enter password for the kibana keystore: ********
Enter value for elasticsearch.username: *************

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% bin/kibana-keystore add elasticsearch.password
Enter password for the kibana keystore: ********
Enter value for elasticsearch.password: ********

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% bin/kibana
...
Enter password for the kibana keystore: ********
[2024-04-30T09:47:03.560-05:00][INFO ][root] Kibana is starting

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% bin/kibana-keystore has-passwd
Keystore is password-protected

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% ./bin/kibana-keystore show elasticsearch.username
Enter password for the kibana keystore: ********
kibana_system

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% ./bin/kibana-keystore remove elasticsearch.username
Enter password for the kibana keystore: ********

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% ./bin/kibana-keystore show elasticsearch.username
Enter password for the kibana keystore: ********
ERROR: Kibana keystore doesn't have requested key.

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% bin/kibana-keystore passwd
Enter password for the kibana keystore: ********
Enter new password for the kibana keystore (empty for no password):
[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% ./bin/kibana-keystore has-passwd
Error: Keystore is not password protected

[jon@mbpkbn1]/tmp/kibana-8.15.0-SNAPSHOT% ./bin/kibana
...
[2024-04-30T09:49:03.220-05:00][INFO ][root] Kibana is starting
```

## Password input

Environment variable usage is not consistent across stack products. I
implemented `KBN_KEYSTORE_PASSWORD_FILE` and `KBN_KEYSTORE_PASSWORD` to
be used to avoid prompts. @elastic/kibana-security do you have any
thoughts?


- `LOGSTASH_KEYSTORE_PASS` -
https://www.elastic.co/guide/en/logstash/current/keystore.html#keystore-password
- `KEYSTORE_PASSWORD` -
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-keystore-bind-mount
- `ES_KEYSTORE_PASSPHRASE_FILE` -
https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-running-systemd
- Beats discussion, unresolved:
https://github.com/elastic/beats/issues/5737


## Release note
Adds password support to the Kibana keystore.
2024-05-06 10:23:29 -05:00
Victor Martinez
63e29efbc6
github-action: undeploy serverless for closed PRs using ci:project-deploy-observability (#182374)
Automate the undeployment for all those Kibana PRs using the label
`ci:project-deploy-observability` once those PRs have been closed
(merged, closed).

This will help with tidying up all the ongoing deployments that were
created automatically as part of the recent automation with
https://github.com/elastic/kibana/pull/181851
2024-05-06 16:32:36 +02:00
Marco Antonio Ghiani
7a69fdd469
[Infra][Logs Explorer] Factor out shared custom hook into a package (#182336)
## 📓 Summary

This work is just the extraction of a utility hook implemented in the
`infra` plugin for re-usability, as there are some scenarios where it
becomes handy in the logs explorer plugin too.

For improved reusability in future, I extracted the useBoolean hook into
a react-hooks package, where additional stateful utility hooks can go in
future.

This also replaces the current usage of the hook in the `infra` plugin
and is used to refactor part of the logs_explorer top nav.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-06 16:17:39 +02:00
Stratoula Kalafateli
807da63c61
[ES|QL] Fetch the query columns utils (#182338)
## Summary

Revives this https://github.com/elastic/kibana/pull/181969

To do so, I had to create a new package `search-types` and move the
types I need there.

The Discovery team can take it from here.

Note: It also does a cleanup on the types I move, some of them were
declared twice.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-05-06 10:51:40 +02:00
Marco Antonio Ghiani
747ecea18a
[Discover] Create Discover Shared plugin and features registry (#181952)
## 📓 Summary

Closes #181528 
Closes #181976 

**N.B.** This work was initially reviewed on a separate PR at
https://github.com/tonyghiani/kibana/pull/1.

This implementation aims to have a stateful layer that allows the
management of dependencies between Discover and other plugins, reducing
the need for a direct dependency.

Although the initial thought was to have a plugin to register features
for Discover, there might be other cases in future where we need to
prevent cyclic dependencies.
With this in mind, I created the plugin as a more generic solution to
hold stateful logic as a communication layer for Discover <-> Plugins.

## Discover Shared

Based on some recurring naming in the Kibana codebase, `discover_shared`
felt like the right place for owning these dependencies and exposing
Discover functionalities to the external world.
It is initially pretty simple and only exposes a registry for the
Discover features, but might be a good place to implement other upcoming
concepts related to Discover.

Also, this plugin should ideally never depend on other solution plugins
and keep its dependencies to a bare minimum of packages and core/data
services.

```mermaid
flowchart TD

A(Discover) -- Get --> E[DiscoverShared]
B(Logs Explorer) -- Set --> E[DiscoverShared]
C(Security) -- Set --> E[DiscoverShared]
D(Any app) -- Set --> E[DiscoverShared]
```

## DiscoverFeaturesService

This service initializes and exposes a strictly typed registry to allow
consumer apps to register additional features and Discover and retrieve
them.

The **README** file explains a real use case of when we'd need to use it
and how to do that step-by-step.

Although it introduces a more nested folder structure, I decided to
implement the service as a client-service and expose it through the
plugin lifecycle methods to provide the necessary flexibility we might
need:
- We don't know yet if any of the features we register will be done on
the setup/start steps, so having the registry available in both places
opens the door to any case we face.
- The service is client-only on purpose. My opinion is that if we ever
need to register features such as server services or anything else, it
should be scoped to a similar service dedicated for the server lifecycle
and its environment.
It should never be possible to register the ObsAIAssistant
presentational component from the server, as it should not be permitted
to register a server service in the client registry.
A server DiscoverFeaturesService is not required yet for any feature, so
I left it out to avoid overcomplicating the implementation.

## FeaturesRegistry

To have a strictly typed utility that suggests the available features on
a registry and adheres to a base contract, the registry exposed on the
DiscoverFeaturesService is an instance of the `FeaturesRegistry` class,
which implements the registration/retrieval logic such that:
- If a feature is already registered, is not possible to override it and
an error is thrown to notify the user of the collision.
- In case we need to react to registry changes, is possible to subscribe
to the registry or obtain it as an observable for more complex
scenarios.

The FeaturesRegistry already takes care of the required logic for the
registry, so that `DiscoverFeaturesService`is left with the
responsibility of instantiating/exposing an instance and provide the set
of allowed features.

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
2024-05-03 11:27:32 +02:00
Steph Milovic
e306074aa9
[Security solution] Add additional properties to attack discovery telemetry (#182249) 2024-05-01 21:48:36 -07:00
Cauê Marcondes
97e0f419bf
[Assets] Creating Assets data access plugin (#182108)
Create an empty assets data access plugin so we can start registering
the services(APIs) that will be used to fetch assets type documents.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-04-30 08:15:23 -07:00
Coen Warmer
d83a7370f9
[Maintenance] Move Asset Manager, Metrics Data Access to /x-pack/plugins/observability_solution (#181970)
## Summary

Title says it all.
2024-04-30 05:38:30 -07:00
Victor Martinez
9035166aa8
github-actions: update command to run the docs validation (#181943)
Update the comment with how to run the docs validation
2024-04-29 15:27:32 +02:00
Marco Antonio Ghiani
600ca62bed
[Logs Explorer][Discover] Move Logs Overview into Discover codebase (#180262)
## 📓 Summary

Closes #180024 

This work aims to move the "Logs Overview" doc view created from the
logs-explorer app into the unified-doc-viewer plugin, creating and
registering this as a preset along the table and source doc views.

To keep control of whether this doc view should be displayed or not, an
`enabled` configuration flag is supported for every doc view and will be
used to determine whether a doc view should load or not in the view.
This `enabled` flag can be programmatically enabled/disabled by
`docView.id` using the 2 new methods added to the `DocViewsRegistry`,
the `enableById` and `disableById` ones.
The customization extension point does not register the content of the
tab, but is limited to enable/disable a preset overview tab.

To allow this change, some shared utilities between the flyout logic and
the smart fields feature have been copied into the `@kbn/discover-utils`
package. The utils will still live in the logs_explorer plugin and are
used by the smart fields feature until this is migrated too into
Discover.

## 💡 Reviewer hints

Although it seems a large PR, most of the changes are on moved files
from logs-explorer into unified-doc-viewer, which is logic already
reviewed. With these changes, there will be a follow-up PR to optimize
the shared parts with the other doc views.

## 🚶 Next steps

To keep the scope of this PR the smallest possible, here are some
changes I'll work out in upcoming PRs built on top of this one.

- [x] Implement a discover registry to enable registering external
features, such as the ObservabilityAIAssistant.
- [x] Integrate ObsAIAssistant with this work through the new discover
features registry.
- [x] Refactor the doc views to share duplicated logic.
- [x] Port the existing e2e tests for the logs overview tab into the
unified-doc-viewer plugin.

https://github.com/tonyghiani/kibana/pull/1

---------

Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Davis McPhee <davismcphee@hotmail.com>
2024-04-29 11:54:50 +02:00
Victor Martinez
2f5897a70c
labeller: enable serverless deployment for oblt automatically (#181851)
The GitHub label will automatically deploy a Serverless deployment with the Observability test data, see https://github.com/elastic/kibana/pull/181041
2024-04-29 07:33:11 +00:00
Nathan Reese
60ab2cfe95
PresentationPanel props hidePanelChrome (#181473)
### Changes
1. adds `hidePanelChrome` parameter to `ReactEmbeddableRenderer`. When
true, embeddable is rendered without `PresentationPanel` wrapper
2. Removes `embeddable-explorer` plugin
3. Moves Embeddable developer example into embeddable_examples plugin
4. Creates new examples that demonstrate how to use
`ReactEmbeddableRenderer`
<img width="600" alt="Screenshot 2024-04-23 at 5 19 18 PM"
src="5167a2a4-1968-42e6-92f7-4577c9cda3c6">

Follow-up work to narrow scope of this PR
1. add key concepts to embeddable overview
2. add "Register new embeddable type" tab that details how to create a
new embeddable and shows how you can add embeddable examples to
dashboard
3. group embeddable examples into a single item in "Add panel" menu - to
show best practices of how to keep menu clean
4. remove class based example embeddables

### Test instructions
1. start kibana with `yarn start --run-examples`
5. Open kibana menu and click "Developer examples"
6. Click "Embeddables" card and run examples

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-04-24 14:05:19 -06:00
Abdul Wahab Zahid
e1ec9ee17e
[Dataset quality] Flyout Summary section (#179479)
Closes https://github.com/elastic/kibana/issues/170492

## Summary

The PR Implements the Dataset Quality Flyout summary KPIs: "Docs count",
"Size", "Services", "Hosts" and "Degraded docs".

|Stateful|Serverless|
|:---:|:---|
|<img alt="Screenshot 2024-03-26 at 19 14 34"
src="d75de56f-0916-48a6-a101-fc3d8f084f4d">|<img
alt="Screenshot 2024-03-26 at 17 02 05"
src="46d58946-2ed5-4c21-b53c-be3d43e7b857">|

"Show all" links for "Services" and "Hosts" metrics depend on some
development in APM and Infra plugins and therefore will be implemented
in a follow up issue.

Note that "Size" metric is excluded on Serverless as the endpoint uses
ES's `_stats` endpoint which is not available on Serverless (at the time
of creation of this PR). The code contains some conditions and cases to
tackle this, which should be considered as a temporary measure. All of
the code related to these changes should either be removed or modified
once there's a way to calculate the size of an index/dataStream on
Serverless (see [related
](https://github.com/elastic/kibana/issues/178954)). The following
changes are made in particular:
 - Size UI component on the Flyout will be hidden on Serverless
- `dataset_quality/data_streams/{dataStream}/details` endpoint will
return `NaN` for size on Serverless
- Unit, integration and end-to-end tests on Serverless handle
"sizeBytes" property accordingly
2024-04-24 13:20:31 +02:00
Pierre Gayvallet
0f3ecf739e
[Security into Core] expose userProfile service from Core (#180372)
## Summary

Fix https://github.com/elastic/kibana/issues/178932

- Introduce the new `userProfile` core service, both on the browser and
server-side.

- Have the security plugin register its API to Core for re-exposition

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-04-24 01:20:05 -07:00
Victor Martinez
3c71d69bf9
ci: integrate observability deployments with the oblt-test-env (#181041) 2024-04-23 07:55:36 -07:00
Saarika Bhasi
439f4a8876
[ML] Support adding inference_id in a flyout (#180330)
## Summary

* Creates a new package - ` @kbn/inference_integration_flyout` 
* Above package has implementation for adding new inference endpoint id,
including Elastic, third party integration and instructions to upload
via Eland python client, in a flyout.
* The above package is used for supporting semantic_text feature and is
accessed via`add a new inference endpoint` button

## Screen Recording


dbd36634-bd4a-49f1-b1d5-d7b6c90444bc


### Checklist


- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-04-21 21:20:28 -04:00
Nathan L Smith
2d2649f13c
Remove @elastic/obs-ux-infra_services-team from Stack Monitoring CODEOWNERS (#180653)
The "virtual" @elastic/stack-monitoring-team owns this and reviews from
infra services team are not required.
2024-04-18 23:12:21 +00:00
Sébastien Loix
5be49cac39
[Stateful sidenav] Remove analytics IA group (#181132) 2024-04-18 16:12:16 +01:00
Jean-Louis Leysens
29fb11b358
[HTTP/OAS] @kbn/router-to-openapispec (#180683)
## Summary

Introduces a new package for generating OAS from Kibana's routers. This
first iteration includes:

* E2E conversion of Core's `Router` and `CoreVersionedRouter` routes
into a single OAS document (not written to disk or shared anywhere
yet...)
* Support for
[`$ref`](https://swagger.io/docs/specification/using-ref/?sbsearch=%24ref)
by introducing the `meta.id` field `@kbn/config-schema`'s base type.
This is intended to be used only response/request schemas initially.

## TODO

- [x] More unit tests

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-04-17 04:03:16 -07:00
Rachel Shen
9579635c25
[Share Modal Redesign] Reporting Refactor Modals (#180009)
## Summary

This PR refactors https://github.com/elastic/kibana/pull/179206 to have
each export type be registered in Reporting and then passed into the
share plugin.

This PR is focused on the redesign in terms of the export modals. Test
refactoring will be done in a separate PR.
Partially closes https://github.com/elastic/kibana-team/issues/753

- [x] Need to refactor this PR to include @eokoneyo's general modal
component
- [x] Lens needs to have Export with all three report type options - to
avoid circular dependencies move the Lens CSV stuff into the reporting
plugin vs having it in Lens
- [x] Canvas should not be affected by these changes (so the old
share/reporting code has to stay for canvas)
https://github.com/elastic/kibana/issues/151523 to keep in mind for the
redesign

Failed tests will be covered in this PR
https://github.com/elastic/kibana/pull/180406


### TO TEST 

Mark `share.new_version.enabled: true` in your kibana.dev.yml


### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))

---------

Co-authored-by: Eyo Okon Eyo <eyo.eyo@elastic.co>
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
Co-authored-by: Marco Liberati <dej611@users.noreply.github.com>
2024-04-16 15:01:44 -07:00
Dzmitry Lemechko
c4e40ea205
[CODEOWNERS] add owners for all svl FTR tests (#180802)
## Summary

Part of work to enable automatic Team notifications for serverless test
failures.

This PR adds missing code owners for FTR functions/api integration test
files, so that we can map failed test suite to responsible Team.

I was using `node scripts/check_ftr_code_owners.js` to find missing
owners.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2024-04-16 11:15:02 +02:00
Rodney Norris
308f514a45
[Search] Introduced Notebooks view for console (#180400)
## Summary

This PR adds the search-notebooks plugin and a python notebook renderer
to the persistent console.

### Screenshots
Console Closed
<img width="1418" alt="image"
src="8e2e2934-a19f-4204-8a31-1e8eab7fd20f">
Notebooks:
<img width="1418" alt="image"
src="bf9d40ad-352d-482e-8d84-f426c3026c69">
<img width="1418" alt="image"
src="fcf8cac2-4640-49e8-9bce-94a5a853383f">

Console View
<img width="1418" alt="image"
src="9230d1c2-3987-41f8-aa86-77a20509b8c0">

### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2024-04-15 11:10:28 -05:00
Sander Philipse
1fa7a69fb6
Move shared index management types to package (#179932)
## Summary

This moves a number of types that are used outside of the index
management plugin to a package so we can avoid cyclical dependencies in
the work we're doing to add semantic text as a mapping type. That will
depend on the ML plugin, which has dependencies that themselves depend
on a few types from index management. I split this into a separate PR
for ease of reviewing.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
2024-04-10 17:48:06 +02:00
Søren Louv-Jansen
fe507aa0b9
[Obs AI Assistant] Update codeowners (#180385)
This updates the code owners from `obs-knowledge-team` to
`obs-ai-assistant` for all of the ai assistant plugins
2024-04-10 17:18:23 +02:00