Your window into the Elastic Stack
Find a file
Tiago Vila Verde 5eb35a702e
[Entity Analytics][Privmon] Changing CSV bulk upsert to use scripted updates (#224721)
## Summary

This PR changes the "soft delete" mechanism in the CSV upload to use
scripted updates.
There were 2 main reasons for this:

1. We need to check if a privileged user has been added from some other
data source

In this case, not including the user in the CSV upload shouldn't delete
it, instead only remove the `"csv"` source label.

2. The soft delete needs to take into account the full list of users
being uploaded, and not only the current batch

This means we need to collapse the stream first and then run the soft
delete logic. Doing this allows the soft delete to search the Privmon
index for all users not included in the full csv file

## How to test


#### Prerequisite

Make sure you have a CSV file with usernames
Check
[here](https://gist.github.com/tiansivive/0be2f09e1bb380fdde6609a131e929ed)
for a little helper script

Create a few copies where some of the users are deleted, in order to
test soft delete.
Make sure to create files that are over the batch size (`100`) or change
the batch size in the code.

1. Start up kibana and ES
2. Navigate to Security > Entity Analytics > Privilege User Monitoring
3. Select the `File` option to add data
4. Add one of the CSV files to the open modal and upload
5. Repeat but now upload one of files with the omitted users 

Alternatively, testing only the backend only is possible by directly
hitting the API wit curl
```sh
curl -u elastic:changeme \
  -X POST "http://localhost:5601/api/entity_analytics/monitoring/users/_csv" \
  -H "kbn-xsrf: true" \
  -F "file=@test.csv;type=text/csv"
```

#### Verifying

Easiest way is to use the dev tools to `_search` the privmon users index
with:
```
GET .entity_analytics.monitoring.users-default/_search
```

Look for number of hits and/or use `query` to search for omitted users
like:
```json
{
   "query": {
      "bool": {
         "must": [
            { "term": { "labels.monitoring.privileged_users": "deleted" } }
          ]
      }
   }
}
```

Verify that the `"deleted"` users are _only_ the ones missing from the
last uploaded file.
2025-06-23 16:20:30 +01:00
.buildkite [Security][9.1] Security roles siemV3 migration for Global Artifact Management (#219566) 2025-06-23 16:14:13 +02:00
.devcontainer Upgrade to Storybook 8 (#195148) 2025-03-14 15:41:03 -07:00
.github docs-builder: add pull-requests: write permission to docs-build workflow (#224714) 2025-06-23 08:30:24 +00:00
api_docs [api-docs] 2025-06-23 Daily api_docs build (#224792) 2025-06-23 07:04:45 +01:00
config [Security][9.1] Security roles siemV3 migration for Global Artifact Management (#219566) 2025-06-23 16:14:13 +02:00
dev_docs Update anatomy_of_a_plugin.mdx (#222857) 2025-06-23 10:46:00 +02:00
docs Remove legacy time axis advanced setting (#221419) 2025-06-23 11:30:29 +02:00
examples [Search] Remove website search guided onboarding (#224538) 2025-06-19 19:34:32 +02:00
kbn_pm add es-cache to cleaned folders (#223373) 2025-06-12 10:58:05 +02:00
legacy_rfcs Optimize existing image assets with lossless compression (#223998) 2025-06-19 16:44:13 +02:00
licenses Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
oas_docs Security Entity Analytics - Privileged user monitoring - Privileged access detection support (#224008) 2025-06-22 00:36:09 -05:00
packages Create @kbn/css-utils package (#223933) 2025-06-20 22:48:44 +02:00
plugins
scripts [Inference] Instrument inference with OpenTelemetry (#218694) 2025-05-07 11:44:29 +02:00
src [ES|QL] Tuples followup cleanup (#224650) 2025-06-23 15:44:12 +01:00
typings chore: remove react-syntax-highlighter leftovers (#213076) 2025-03-04 15:35:34 +01:00
x-pack [Entity Analytics][Privmon] Changing CSV bulk upsert to use scripted updates (#224721) 2025-06-23 16:20:30 +01:00
.backportrc.json chore(NA): prepare replacement of 8.x branch with 8.19 (#218514) 2025-04-17 04:02:40 +01:00
.bazelignore Remove references to deleted .ci folder (#177168) 2024-02-20 19:54:21 +01:00
.bazeliskversion
.bazelrc
.bazelrc.common Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
.bazelversion chore(NA): revert bazel upgrade for v5.2.0 (#135096) 2022-06-24 03:57:21 +01:00
.browserslistrc Add Firefox ESR to browserlistrc (#184462) 2024-05-29 17:53:18 -05:00
.editorconfig
.eslintignore chore(fullstory): serve the snippet as an asset (#220368) 2025-05-07 23:06:24 +02:00
.eslintrc.js [ska] relocate guided_onboarding tests that import search solution src code (#224390) 2025-06-18 16:01:56 +02:00
.gitattributes
.gitignore Upgrade Node.js to 22.16.0 (#205983) 2025-06-18 08:55:18 -05:00
.i18nrc.json [Discover] Support Lens fetches across tabs (#218506) 2025-05-02 13:39:25 -03:00
.node-version Upgrade Node.js to 22.16.0 (#205983) 2025-06-18 08:55:18 -05:00
.npmrc [npmrc] Fix puppeteer_skip_download configuration (#177673) 2024-02-22 18:59:01 -07:00
.nvmrc Upgrade Node.js to 22.16.0 (#205983) 2025-06-18 08:55:18 -05:00
.prettierignore
.prettierrc
.puppeteerrc Add .puppeteerrc (#179847) 2024-04-03 09:14:39 -05:00
.stylelintignore
.stylelintrc Bump stylelint to ^14 (#136693) 2022-07-20 10:11:00 -05:00
.telemetryrc.json Sustainable Kibana Architecture: Move modules owned by @elastic/kibana-core (#201653) 2025-01-04 11:47:24 -07:00
.yarnrc
BUILD.bazel Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
catalog-info.yaml Configures PagerDuty Backstage Integration for kbn (#208440) 2025-01-27 23:29:11 +00:00
CODE_OF_CONDUCT.md
CONTRIBUTING.md Docs: fix broken links in CONTRIBUTING.md (#219158) 2025-04-24 17:25:00 -06:00
FAQ.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
fleet_packages.json [main] Sync bundled packages with Package Storage (#224663) 2025-06-23 10:54:41 +01:00
github_checks_reporter.json
kibana.d.ts Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
LICENSE.txt Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
NOTICE.txt [api-docs] 2025-01-01 Daily api_docs build (#205342) 2025-01-01 01:37:13 -06:00
package.json Upgrade Playwright to 1.53.1 (#224604) 2025-06-23 14:15:23 +02:00
preinstall_check.js Adds AGPL 3.0 license (#192025) 2024-09-06 19:02:41 -06:00
README.md
renovate.json Renovate - move immer redux toolkit owner, move csstype to react dependency owner (#223852) 2025-06-19 12:42:24 +02:00
RISK_MATRIX.mdx
run_fleet_setup_parallel.sh Sustainable Kibana Architecture: Move modules owned by @elastic/fleet (#202422) 2024-12-24 15:32:43 +01:00
SECURITY.md
sonar-project.properties [sonarqube] update memory, cpu (#190547) 2024-09-09 16:16:30 -05:00
STYLEGUIDE.mdx [styleguide] update path to scss theme (#140742) 2022-09-15 10:41:14 -04:00
tsconfig.base.json Create @kbn/css-utils package (#223933) 2025-06-20 22:48:44 +02:00
tsconfig.browser.json
tsconfig.browser_bazel.json
tsconfig.json Transpile packages on demand, validate all TS projects (#146212) 2022-12-22 19:00:29 -06:00
TYPESCRIPT.md Fix small typos in the root md files (#134609) 2022-06-23 09:36:11 -05:00
updatecli-compose.yaml deps(updatecli): bump all policies (#195865) 2024-10-15 07:37:12 -05:00
versions.json chore(NA): update versions after v8.17.8 bump (#222415) 2025-06-03 21:39:35 +01:00
WORKSPACE.bazel Upgrade Node.js to 22.16.0 (#205983) 2025-06-18 08:55:18 -05:00
yarn.lock Upgrade Playwright to 1.53.1 (#224604) 2025-06-23 14:15:23 +02:00

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 7.15.1 7.15.1 💚 OK
ES patch number is newer. 7.15.0 7.15.1 ⚠️ Logged warning
ES minor number is newer. 7.14.2 7.15.0 ⚠️ Logged warning
ES major number is newer. 7.15.1 8.0.0 🚫 Fatal error
ES patch number is older. 7.15.1 7.15.0 ⚠️ Logged warning
ES minor number is older. 7.15.1 7.14.2 🚫 Fatal error
ES major number is older. 8.0.0 7.15.1 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.