## Summary
Fix broken tests !!
These got broken due to changes on alerts overview page, i am also
expanding the scope to run on all observability plugin changes !!
**Resolves:** https://github.com/elastic/kibana/issues/214302
## Summary
This PR prevents showing rule upgrade confirmation modal on lower licenses where prebuilt rules customization is not allowed.
## Details
Users may see a rules upgrade confirmation modal when trying to upgrade prebuilt rules even if prebuilt rules customization is disabled due to insufficient license. It happens due to improper response from `upgrade/_perform` which doesn't respect `pick_version`. It's expected rule upgrade goes smoothly when `pick_version` is one of `BASE`, `CURRENT` or `TARGET`.
The fix makes sure dry run request isn't fired and a prebuilt rules upgrade confirmation modal isn't shown when running with insufficient for prebuilt rules customization license.
There is a [ticket](https://github.com/elastic/kibana/issues/214338) to address this issue in the API endpoint.
## Summary
The following changes are being done to Artifact Card's Menu (which
displays the option to Delete or Update the artifact) in support of
space awareness feature (currently behind Feature Flag:
`endpointManagementSpaceAwarenessEnabled`):
- Global Artifacts: If displaying a global artifact and user does not
have the new Global Artifact Management privilege - disable the Edit
menu icon and display a tooltip on hover
- Per-Policy Artifacts: if displaying a per-policy artifact in a space
other than one of the `ownerSpaceId` spaces that the artifact is
associated with and the user does not have the new Global Artifact
Management privilege - disable the Edit menu icon and display a tooltip
when the user hover over that button
> [!NOTE]
> Changes were **NOT** done to Endpoint Exceptions with this PR.
fixes [214167](https://github.com/elastic/kibana/issues/214167)
## Summary
Fixes a problem affecting the new service map api, causing the service
map to show duplicate exit span nodes
<img width="1476" alt="image"
src="https://github.com/user-attachments/assets/f7bf6035-17f5-4265-9950-5c764f1e7f0b"
/>
### how to test
- Enable the new service map api by adding
`xpack.apm.ui.serviceMapApiV2Enabled: true` to `kibana.dev.yml`
- Run the `service_map` synthtrace scenario
- Navigate to APM and view the service map for the `frontend-rum`
service
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
- addresses https://github.com/elastic/kibana/issues/197130
- allows deduplication of alerts for non-aggregating ES|QL queries, when
`mv_expand` used
- when `mv_expand` command used, id of alerts is constructed with
additional value from expanded field
- if expanded field(column) is not present in ES|QL response, we hash
the whole source object when generate alert id. Field can be absent,
because it was renamed or dropped. When field was renamed, it still
allow us to create unique alert id. When field is dropped, expanded
results would be identical. In this case, duplicated alerts won't be
created (refer to FTR tests)
## Summary
Currently we are using `globalSetup` [script in configuration
file](https://playwright.dev/docs/test-global-setup-teardown#option-2-configure-globalsetup-and-globalteardown)
to ingest Elasticsearch data before running the tests in parallel
against the same ES/Kibana instances.
This approach doesn't work well when you need to adjust `globalSetup`
logic based on specific condition, e.g. configuration file defining
where servers are hosted, its credentials, etc.
Not only global hook, but `ScoutConfig` fixture expects an argument to
define where servers configuration is defined:
cd502acea1/packages/kbn-scout/src/playwright/fixtures/worker/core_fixtures.ts (L65-L75)
`testInfo` is how Playwright exposes currently running configuration in
a form of `project` interface:
[projects](https://playwright.dev/docs/test-projects) can be used to
group tests, e.g. for specific envs or browsers.
Unfortunately `testInfo` is not exposed in global scripts, because in
Playwright project design `globalSetup` scripts are run before multiple
projects and projects can have its own `setup` hooks via
[dependencies](https://playwright.dev/docs/test-global-setup-teardown#option-1-project-dependencies):
```
{
name: 'setup',
testMatch: /global.setup\.ts/,
},
{
name: 'local',
use: { ...devices['Desktop Chrome'], configName: 'local' },
dependencies: 'setup',
},
```
We already use project API to get `serversConfigDir` path, where we plan
to store local and cloud server configurations. This PR proposes to
define projects as `local` and `cloud` (maybe even separate `cloud-mki`,
`cloud-ech`) as a way to provide playwright information about servers
configuration.
Advantages:
1. we can re-use existing fixtures as-is, without adding custom exported
helper functions for ES data ingestion
2. project dependency is displayed as `setup` in Playwright report
3. way better and simpler design for consumers:
```
import { globalSetupHook } from '@kbn/scout';
globalSetupHook('Ingest data to Elasticsearch', async ({ esArchiver, log }) => {
// add archives to load, if needed
const archives = [
testData.ES_ARCHIVES.LOGSTASH,
];
log.debug('[setup] loading test data (only if indexes do not exist)...');
for (const archive of archives) {
await esArchiver.loadIfNeeded(archive);
}
});
```
4. it is supported by VSCode Playwright plugin
<img width="1271" alt="Screenshot 2025-02-17 at 11 26 12"
src="https://github.com/user-attachments/assets/ba7eeb38-d39d-4785-9c11-18647599ec4a"
/>
I find it extremely useful because you don't need to change env var when
you want to switch b/w local or cloud run, all the configurations are
loaded automatically and you just tick the checkbox!
Disadvantages:
1. it is important to run `playwright test` with `--project` flag to use
the proper configuration
2. we have to define how `projects` are used for local and cloud
configuration, and make sure it meets requirements of multiple teams. We
can expose a way to pass custom project definitions in
`createPlaywrightConfig` function, but it might complicate the support
effort when every Team has too many custom projects.
3. `project` term is something we can't change and might be confusing
4. Since it is a Playwright feature, we might not have consistency with
API tests runner under Scout
For reviewers:
Playing with it locally might give a better understanding about the
pros/cons, especially with IDE playwright plugin installed.
Running servers with tests:
```
node scripts/scout.js run-tests --serverless=oblt --testTarget=local --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts
node scripts/scout.js run-tests --serverless=oblt --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts
```
Running test only requires passing `project` argument:
```
npx playwright test --project=local --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts
npx playwright test --project=local --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts
```
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: jennypavlova <jennypavlova94@gmail.com>
## Summary
Addresses https://github.com/elastic/kibana/issues/213706.
I suspect that this stopped working when #206523 removed the
`FullScreenContainer` from the inner StatefulEventsViewer component;
that change combined with the `display:flex` on this parent element
caused the table to no longer grow as expected.
Luckily, the fix here was simply not using flexbox on the container.
Since the table is the only element within this container, I saw no
reason why flexbox was needed here; testing has confirmed this.
### Screenshot (after fix; see bug for "before" image)
<kbd><img width="1709" alt="Screenshot 2025-03-11 at 5 35 43 PM"
src="https://github.com/user-attachments/assets/108be815-00e2-48b7-8821-e63566eefa09"
/></kbd>
Closes#197953
## Summary
This PR replaces the alert status filter with filter controls. In this
PR, I also covered backward compatibility when we have a `status` URL
parameter by passing that value to filters, as shown below:
|State|Screenshot|
|---|---|
|Before||
|After||
https://github.com/user-attachments/assets/86e82a19-f68e-4127-9fd8-e0efe0d41ece
I checked in Serverless and we have access to controls in viewer mode as
well:
https://github.com/user-attachments/assets/2c90ba3a-7d95-4682-b722-e5b327f7334d
### 🐞 Known issue
1. Privilege
In Stateful, if a user has Kibana privilege but not the `.alert*` es
privilege, then the controls do not work as expected. This issue will be
tackled in a separate ticket:
<details>
<summary>This is the error that we show in this scenario</summary>
</details>
2. Initial load
Related ticket: https://github.com/elastic/kibana/issues/183412
### 🗒️ Tasks
- [x] ~~Solving the permission issue~~ This issue does not happen in
Serverless and for stateful, we will fix it in a separate ticket:
https://github.com/elastic/kibana/issues/208225
- The main issue will be fixed in this
[PR](https://github.com/elastic/kibana/pull/191110)
- In the above [PR](https://github.com/elastic/kibana/pull/191110), we
remove controls if the user does not have the privilege for alert
indices, but we need to figure out how to adjust filter controls to
access the data based on Kibana privileges.
- [x] We should configure the filters to allow the selection of one item
for alert status but still show the other options
- [x] We need to see how we can make this work with the current status
field. Ideally, if there is a status field, we would apply it and remove
it from the URL.
- Fixed in
[c6cad2d](c6cad2dbe1)
- [x] Changing the URL does not update the page filters correctly. ~~It
might be related to https://github.com/elastic/kibana/issues/183412.~~
- [x] We need to make sure these adjustments work as expected in APM as
they use the observability alert search bar.
- [x] Check if the tags filter can be improved, and if not, whether it
makes sense to keep it in its current form.
- It works based on how array filtering works in ES, which seems like a
good start to me.
- [x] Check with Maciej: Do we need to disable changing control configs?
- Checked with Maciej: it is fine to keep the option of editing
controls.
- [x] Do we need to have a different local storage item for each page
(apm/rule details/alert details/alerts)?
- How can we disable syncing with the local storage?
- Added the possibility of disabling sync in
[24bab21](24bab210b0)
and disabled it for the rule details and alert details pages.
- Also, disabled it for the APM alert search bar.
- [x] Setting default status as active on the related alerts tab
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Shahzad <shahzad31comp@gmail.com>
## Summary
[META
issue](https://github.com/elastic/security-docs-internal/issues/57)
This PR improves the API documentation for timeline/notes/pinned events
as per the definition in the meta issue.
### Notes
#### Timeline API
- `eventType`:
- is always set to `all`
- looks like it's been superseded by dataView?
- I marked it as `deperecated`
- `eventIdToNoteIds`:
- there's a type mismatch between what the frontend expects and what is
returned
- also it does not seem to be used anymore?
- it needs to be investigated further
#### Notes API
- `GlobalNote`:
- looks like it's not used anymore
- I removed it, nothing broke
- `eventIngested`, `eventTimestamp`, `eventDataView` and `overrideOwner`
are all not used on the patch note endpoint, I removed them. The
`event*` ones I have never seen before to be honest.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/kibana/issues/214171**
## Summary
This PR fixes an issue with "references" field in -AB situations. When
the base version is missing we try to merge current and target arrays
which leads to old and potentially broken URLs from the current version
ending up in the result.
Now the behaviour is changed to always force merged version to be equal
to target (which always has correct URLs).
We need to add a `solution` field to rule types in order to provide a
root-level categorization based on the Kibana solution/area that owns
each rule type. This will serve as a more reliable alternative to
previous attempts at `featureId`-based filtering.
## Summary
- Adds a `solution` field to rule type objects, accepting the values
`observability`, `security` and `stack`.
- Categorizes existing rule types.
- Updates tests with the new field.
## Verification steps
1. Check that the rule types you own have been categorized under the
correct solution
2. Create rules in multiple areas of Kibana
3. Check that those rules show up correctly in their UIs and work
correctly
## References
Closes#212017
## Summary
- refines description for
- 9 `(win|mac|linux).advanced.events.hash.(md5|sha1|sha256)`:
9e7bbcf767
> Compute and include (MD5|SHA-1|SHA-256) hashes for processes and
libraries in events? This will increase CPU usage and event sizes. If
any user event filter or trustlists reference this hash type, Endpoint
will ignore this setting and automatically enable this hash type.
- 6 `(win|mac|linux).advanced.alerts.hash.(md5|sha1)`:
8fc0f51ab4
> Compute and include (MD5|SHA-1) hashes for processes and libraries in
alerts? This will increase CPU usage and alert sizes. If any user
exceptionlist, trustlist, or blocklists reference this hash type,
Endpoint will ignore this setting and automatically enable this hash
type.
- provides a 'history' for default behavior changes (e.g. `<=8.17
default: true, >=8.18 default: false`) for
- 12 `(win|mac|linux).advanced.(events|alerts).hash.(md5|sha1)`:
05b0ebe8ea
(note that events sha256 is not changed)
> <=8.17 default: true, >=8.18 default: false
- 3 `(win|mac|linux).advanced.events.aggregate_process`:
5984d8e90a
> <=8.17 default: false, >=8.18 default: true
- 3 `(win|mac|linux).advanced.events.set_extended_host_information`:
5da25a3592
> <=8.17 default: true, >=8.18 default: false
> [!IMPORTANT]
> The plan is to backport this PR to all open branches:
> - `8.18`/`8.x`/`9.0`/`main` will contain all modifications,
> - but `8.16`/`8.17` manual backports will only contain the description
refinement
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
## Summary
Resolves https://github.com/elastic/kibana/issues/213983
Adds gating logic to only display the rule upgrade related warning
callout on the rule editing page if the user has the appropriate license
level, otherwise the message is irrelevant as the user wouldn't be able
to edit a prebuilt rule in the way the message refers to
### Testing
**Steps to reproduce:**
1. Use a Platinum license or any tier that does not allow prebuilt rule
updates.
2. Navigate to the Rule Management page and select a prebuilt rule with
an available update.
3. Open Rule Details page -> Edit rule settings
4. Observe the callout message is not displayed at the top of the page
NOTE: When using the enterprise license, the callout should still be
displayed
## Summary
Changes how we use lookup indices so that they are now space aware, in
the form
```
`${LOOKUPS_INDEX_PREFIX}${this.spaceId}_${lookupName}`
```
### Checklist
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Fixes a problem [`Bulk adding tags to rules marks some rules as customized and doesn't mark other rules as customized. It looks like it depends on the existence of the base version.`](https://github.com/elastic/kibana/pull/212761#pullrequestreview-2675994950) discovered while smoke testing after enabling Prebuilt Rules Customization FF.
## Details
The problems manifests as some rules have `Modified` badge missing after modifying tags via bulk actions.
The root cause is that current bulk actions implementation expects unmodified rule's data in `paramsModifier()` callback. But Alerting Framework's Rules Client invokes `paramsModifier()` providing already modified rule. Alerting Framework managed fields like `rule.tags` have modified values.
The fix makes sure rule customizartion state is calculated by using unmodified rule data.
## Screenshots
Before:
https://github.com/user-attachments/assets/eeb65b18-c51f-4c5e-b0e6-6552e442994e
After:
https://github.com/user-attachments/assets/d18d8765-4f40-4513-95a1-2cd84ac2a0a9
## Summary
- Moving last Home plugin components from `js` to `tsx`
- Adding TS support
- Styling changes/ clean up
- Fixing bugs that were unnoticed without ts support
- Creating or updating unit tests
- shifting from `enzyme` to `testing-library/react`
- removing snapshots in favor of selectors
## Summary
Create monitor configs repository around monitor saved object to make
sure all operations are performed from same class.
This will be helpful when we create a new saved object to support
multiple-spaces !!
### Testing
All unit tests, api tests passing should be more than enough !!
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
It closes#210715
This PR introduces the No Data Found onboarding screen, which displays
the Asset Inventory Enabled success callout and the Integrations Card
Grid component from Security Onboarding.
Success Callout visibility is stored in local storage for persisting
visibility dismissal, also, the callout is only enabled for the user who
initiated the enablement.
This PR also includes:
- Moving the renderWithTestProvider into a reusable function
- Moving some onboarding test subj to the constants file
- Added the missing test file for the navigation onboarding screen.
**Note:** Customization of the Integrations Card Grid component for
asset inventory will be done in a separate task.
## Recording
https://github.com/user-attachments/assets/e8f4b363-7d31-4bd8-a2d3-d234f4a0b34b
## Summary
Error were not properly propagated to the user and instead of meaningful
message we were displaying just `API Error`.
<img width="1813" alt="Zrzut ekranu 2025-03-11 o 03 47 59"
src="https://github.com/user-attachments/assets/8d059159-f020-4944-a463-b10799e7fa46"
/>
Steps to reproduce, Thank you @andrew-goldstein 🙇
**Desk testing**
To reproduce:
1. In LM Studio, download the `MLX` variant (optimized for Mac) of
`Llama-3.2-3B-Instruct-4bit`:
```
mlx-community/Llama-3.2-3B-Instruct-4bit
```
2. Configure the model to have a context length of `131072` tokens, as
illustrated by the screenshot below:
3. Serve ONLY the model above in LM Studio. (Ensure no other models are
running in LM Studio), as illustrated by the screenshot below:
4. Configure a connector via the details in
<https://www.elastic.co/guide/en/security/current/connect-to-byo-llm.html>
but change:
```
local-model
```
to the name of the model when configuring the connector:
```
llama-3.2-3b-instruct
```
as illustrated by the screenshot below:
5. Generate Attack discoveries
**Expected results**
- Generation does NOT fail with the error described in the later steps
below.
- Progress on generating discoveries is visible in Langsmith, as
illustrated by the screenshot below:
Note: `Llama-3.2-3B-Instruct-4bit` may not reliably generate Attack
discoveries, so generation may still fail after `10` generation /
refinement steps.
6. In LM studio, serve a _second_ model, as illustrated by the
screenshot below:
7. Once again, generate Attack discoveries
**Expected results**
- Generation does NOT fail with the errors below
- Progress on generating discoveries is visible in Langsmith, though as
noted above, generation may still fail after `10` attempts if the model
does not produce output that conforms to the expected schema
**Actual results**
- Generation fails with an error similar to:
```
generate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 400. Message: API Error:
Bad Request: ActionsClientLlm: action result status is error: an error occurred while running the action - Status code: 400. Message: API Error: Bad Request,
```
or
```
generate node is unable to parse (openai) response from attempt 0; (this may be an incomplete response from the model): Status code: 404. Message: API Error: Not Found - Model "llama-3.2-3b-instruct" not found. Please specify a valid model.
```
as illustrated by the following screenshot:
**Addresses:** https://github.com/elastic/kibana/issues/180267
## Summary
This PR enables `prebuiltRulesCustomizationEnabled` feature flag.
## Details
Besides simply enabling `prebuiltRulesCustomizationEnabled` feature flag the following required changes were done
- failed tests due enabling the FF were fixed
- FF setting was removed from test configurations (integrations and Cypress tests)
- FF logic was removed from the codebase. Disabling the FF would require roll back test changes as well. So just in case we have to disable the FF it's simpler to roll back the PR's commit.
- Adds test for `execute_query` function
- Add `drop_null_columns` to esql query in order to avoid column
explosion
Depends on: https://github.com/elastic/kibana/pull/213231
---------
Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com>
## Summary
Adds extra labels and context for security rule executions. Follow up
work will add more context specific to the different security rule type.
This PR focuses on capturing information about the configuration in
logic shared across all rule types.
The info collected is generally intended for use to narrow down _why_ a
rule might be performing poorly. The `rule_id` param is collected so we
can aggregate and identify prebuilt rules that perform poorly across
many environments and ask the TRaDE team to help tune the rule
query/config.
## Testing
1. Spin up an Observability cluster on cloud.elastic.co
2. Setup your local cluster to send APM data to the cloud cluster
```
elastic:
apm:
active: true
serverUrl: <apm url from cloud console>
secretToken: <secret token>
```
To find the secret token, login to Kibana on your cloud obs cluster and
go to `Management -> Fleet -> Elastic cloud agent policy -> Elastic APM
-> Agent Authorization -> Secret token`
3. Run rules in your local test environment. Observe APM data in the
cloud cluster
## Uses
In addition to debugging, we can use this new apm data to create
dashboards like the screenshot below, showing the slowest rules by
`rule_id` (only one rule in the test environment, but in production this
would show the slowest rules across all apm enabled clusters).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR makes changes to force target version for critical rule fields upon upgrade when rule's base version isn't found.
## Details
In case of missing base version rule upgrade diff algorithm isn't able to detect rule customizations. The only thing it's able to determine is different current and target field values. It tries to merge mergeable fields whenever possible. In fact mergeable fields are only scalar arrays. There are 4 fields processed that way
- `tags`
- `references`
- `threat_index`
- `new_terms`
However, `threat_index` and `new_terms` are critical rule fields impacting what alerts rule is gonna generate. Auto-merged values have much higher chance to be improper.
This PR **changes** the current behavior for `threat_index` and `new_terms`. It forces target field version when base rule version isn't found.
## How to test
To test this fix an updated prebuilt rule asset should be created. Follow the next steps for that
- Start fresh ES and Kibana (clear any ES data before)
- Open Rules Management page to install the latest prebuilt rules package (`8.17.6` currently)
- Install a threat match rule (e.g. `Threat Intel Hash Indicator Match`) and a new terms rule (e.g. `AWS EC2 Route Table Modified or Deleted`)
- Create new prebuilt rule assets. You may copy and paste suggested queries below. Make sure to run them under `system_indicies_superuser`/`changeme` user.
<details>
<summary>PUT a new Threat Match prebuilt rule asset</summary>
```
PUT .kibana_security_solution/_doc/security-rule:aab184d3-72b3-4639-b242-6597c99d8bca_10
{
"security-rule": {
"author": [
"Elastic"
],
"description": "This rule is triggered when a hash indicator from the Threat Intel Filebeat module or integrations has a match against an event that contains file hashes, such as antivirus alerts, process creation, library load, and file operation events.",
"from": "now-65m",
"index": [
"auditbeat-*",
"endgame-*",
"filebeat-*",
"logs-*",
"winlogbeat-*"
],
"interval": "1h",
"language": "kuery",
"license": "Elastic License v2",
"name": "Threat Intel Hash Indicator Match",
"note": """## Triage and analysis
### Investigating Threat Intel Hash Indicator Match
Threat Intel indicator match rules allow matching from a local observation, such as an endpoint event that records a file hash with an entry of a file hash stored within the Threat Intel integrations index.
Matches are based on threat intelligence data that's been ingested during the last 30 days. Some integrations don't place expiration dates on their threat indicators, so we strongly recommend validating ingested threat indicators and reviewing match results. When reviewing match results, check associated activity to determine whether the event requires additional investigation.
This rule is triggered when a hash indicator from the Threat Intel Filebeat module or an indicator ingested from a threat intelligence integration matches against an event that contains file hashes, such as antivirus alerts, file operation events, etc.
> **Note**:
> This investigation guide uses the [Osquery Markdown Plugin](https://www.elastic.co/guide/en/security/current/invest-guide-run-osquery.html) introduced in Elastic Stack version 8.5.0. Older Elastic Stack versions will display unrendered Markdown in this guide.
#### Possible investigation steps
- Gain context about the field that matched the local observation. This information can be found in the `threat.indicator.matched.field` field.
- Investigate the hash , which can be found in the `threat.indicator.matched.atomic` field:
- Search for the existence and reputation of the hash in resources like VirusTotal, Hybrid-Analysis, CISCO Talos, Any.run, etc.
- Scope other potentially compromised hosts in your environment by mapping hosts with file operations involving the same hash.
- Identify the process that created the file.
- Investigate the process execution chain (parent process tree) for unknown processes. Examine their executable files for prevalence, whether they are located in expected locations, and if they are signed with valid digital signatures.
- Enrich the information that you have right now by determining how the file was dropped, where it was downloaded from, etc. This can help you determine if the event is part of an ongoing campaign against the organization.
- Retrieve the involved file and examine the host for derived artifacts that indicate suspicious activities:
- Analyze the process executable using a private sandboxed analysis system.
- Observe and collect information about the following activities in both the sandbox and the alert subject host:
- Attempts to contact external domains and addresses.
- Use the Elastic Defend network events to determine domains and addresses contacted by the subject process by filtering by the process' `process.entity_id`.
- Examine the DNS cache for suspicious or anomalous entries.
- !{osquery{"label":"Osquery - Retrieve DNS Cache","query":"SELECT * FROM dns_cache"}}
- Use the Elastic Defend registry events to examine registry keys accessed, modified, or created by the related processes in the process tree.
- Examine the host services for suspicious or anomalous entries.
- !{osquery{"label":"Osquery - Retrieve All Services","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services"}}
- !{osquery{"label":"Osquery - Retrieve Services Running on User Accounts","query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services WHERE\nNOT (user_account LIKE '%LocalSystem' OR user_account LIKE '%LocalService' OR user_account LIKE '%NetworkService' OR\nuser_account == null)\n"}}
- !{osquery{"label":"Osquery - Retrieve Service Unsigned Executables with Virustotal Link","query":"SELECT concat('https://www.virustotal.com/gui/file/', sha1) AS VtLink, name, description, start_type, status, pid,\nservices.path FROM services JOIN authenticode ON services.path = authenticode.path OR services.module_path =\nauthenticode.path JOIN hash ON services.path = hash.path WHERE authenticode.result != 'trusted'\n"}}
- Using the data collected through the analysis, scope users targeted and other machines infected in the environment.
### False Positive Analysis
- Adversaries often use legitimate tools as network administrators, such as `PsExec` or `AdFind`. These tools are often included in indicator lists, which creates the potential for false positives.
### Response and Remediation
- Initiate the incident response process based on the outcome of the triage.
- Isolate the involved host to prevent further post-compromise behavior.
- If the triage identified malware, search the environment for additional compromised hosts.
- Implement temporary network rules, procedures, and segmentation to contain the malware.
- Stop suspicious processes.
- Immediately block the identified indicators of compromise (IoCs).
- Inspect the affected systems for additional malware backdoors like reverse shells, reverse proxies, or droppers that attackers could use to reinfect the system.
- Remove and block malicious artifacts identified during triage.
- Run a full antimalware scan. This may reveal additional artifacts left in the system, persistence mechanisms, and malware components.
- Determine the initial vector abused by the attacker and take action to prevent reinfection through the same vector.
- Using the incident response data, update logging and audit policies to improve the mean time to detect (MTTD) and the mean time to respond (MTTR).
""",
"query": """file.hash.*:* or process.hash.*:* or dll.hash.*:*
""",
"references": [
"https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html",
"https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html",
"https://www.elastic.co/security/tip"
],
"required_fields": [
{
"ecs": false,
"name": "dll.hash.*",
"type": "unknown"
},
{
"ecs": false,
"name": "file.hash.*",
"type": "unknown"
},
{
"ecs": false,
"name": "process.hash.*",
"type": "unknown"
}
],
"risk_score": 99,
"rule_id": "aab184d3-72b3-4639-b242-6597c99d8bca",
"setup": """## Setup
This rule needs threat intelligence indicators to work.
Threat intelligence indicators can be collected using an [Elastic Agent integration](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#agent-ti-integration),
the [Threat Intel module](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#ti-mod-integration),
or a [custom integration](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#custom-ti-integration).
More information can be found [here](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html).
""",
"severity": "critical",
"tags": [
"OS: Windows",
"Data Source: Elastic Endgame",
"Rule Type: Threat Match",
"Resources: Investigation Guide"
],
"threat_filters": [
{
"$state": {
"store": "appState"
},
"meta": {
"disabled": false,
"key": "event.category",
"negate": false,
"params": {
"query": "threat"
},
"type": "phrase"
},
"query": {
"match_phrase": {
"event.category": "threat"
}
}
},
{
"$state": {
"store": "appState"
},
"meta": {
"disabled": false,
"key": "event.kind",
"negate": false,
"params": {
"query": "enrichment"
},
"type": "phrase"
},
"query": {
"match_phrase": {
"event.kind": "enrichment"
}
}
},
{
"$state": {
"store": "appState"
},
"meta": {
"disabled": false,
"key": "event.type",
"negate": false,
"params": {
"query": "indicator"
},
"type": "phrase"
},
"query": {
"match_phrase": {
"event.type": "indicator"
}
}
}
],
"threat_index": [
"logs-ti_*"
],
"threat_indicator_path": "threat.indicator",
"threat_language": "kuery",
"threat_mapping": [
{
"entries": [
{
"field": "file.hash.md5",
"type": "mapping",
"value": "threat.indicator.file.hash.md5"
}
]
},
{
"entries": [
{
"field": "file.hash.sha1",
"type": "mapping",
"value": "threat.indicator.file.hash.sha1"
}
]
},
{
"entries": [
{
"field": "file.hash.sha256",
"type": "mapping",
"value": "threat.indicator.file.hash.sha256"
}
]
},
{
"entries": [
{
"field": "dll.hash.md5",
"type": "mapping",
"value": "threat.indicator.file.hash.md5"
}
]
},
{
"entries": [
{
"field": "dll.hash.sha1",
"type": "mapping",
"value": "threat.indicator.file.hash.sha1"
}
]
},
{
"entries": [
{
"field": "dll.hash.sha256",
"type": "mapping",
"value": "threat.indicator.file.hash.sha256"
}
]
},
{
"entries": [
{
"field": "process.hash.md5",
"type": "mapping",
"value": "threat.indicator.file.hash.md5"
}
]
},
{
"entries": [
{
"field": "process.hash.sha1",
"type": "mapping",
"value": "threat.indicator.file.hash.sha1"
}
]
},
{
"entries": [
{
"field": "process.hash.sha256",
"type": "mapping",
"value": "threat.indicator.file.hash.sha256"
}
]
}
],
"threat_query": "@timestamp >= \"now-30d/d\" and event.module:(threatintel or ti_*) and (threat.indicator.file.hash.*:* or threat.indicator.file.pe.imphash:*) and not labels.is_ioc_transform_source:\"true\"",
"timeline_id": "495ad7a7-316e-4544-8a0f-9c098daee76e",
"timeline_title": "Generic Threat Match Timeline",
"timestamp_override": "event.ingested",
"type": "threat_match",
"version": 10
},
"type": "security-rule",
"references": [],
"managed": true,
"coreMigrationVersion": "8.8.0",
"updated_at": "2025-03-11T07:21:44.100Z",
"created_at": "2025-03-11T07:21:44.100Z"
}
```
</details>
<details>
<summary>PUT a new New Terms prebuilt rule asset</summary>
```
PUT .kibana_security_solution/_doc/security-rule:e7cd5982-17c8-4959-874c-633acde7d426_209
{
"security-rule": {
"author": [
"Elastic",
"Austin Songer"
],
"description": "Identifies AWS CloudTrail events where an EC2 route table or association has been modified or deleted. Route table or association modifications can be used by attackers to disrupt network traffic, reroute communications, or maintain persistence in a compromised environment. This is a [New Terms](https://www.elastic.co/guide/en/security/current/rules-ui-create.html#create-new-terms-rule) rule that detects the first instance of this behavior by the `aws.cloudtrail.user_identity.arn` field in the last 10 days.",
"false_positives": [
"Route Table could be modified or deleted by a system administrator. Verify whether the user identity, user agent, and/or hostname should be making changes in your environment. Route Table being modified from unfamiliar users should be investigated. If known behavior is causing false positives, it can be exempted from the rule. Also automated processes that use Terraform may lead to false positives."
],
"from": "now-9m",
"history_window_start": "now-10d",
"index": [
"filebeat-*",
"logs-aws.cloudtrail-*"
],
"language": "kuery",
"license": "Elastic License v2",
"name": "AWS EC2 Route Table Modified or Deleted",
"new_terms_fields": [
"test"
],
"note": """## Triage and Analysis
### Investigating AWS EC2 Route Table Modified or Deleted
This rule detects modifications or deletions of AWS route tables using actions such as `ReplaceRoute`, `ReplaceRouteTableAssociation`, `DeleteRouteTable`, `DeleteRoute`, or `DisassociateRouteTable`. These actions may indicate legitimate administrative activity, but they can also be abused by attackers to disrupt network traffic, reroute communications, or maintain persistence in a compromised environment.
#### Possible Investigation Steps
- **Review Request Parameters:**
- Check the `aws.cloudtrail.flattened.request_parameters` field. The sub-fields may vary depending on the `event.action` (e.g., `routeTableId` for `DeleteRouteTable`, `destinationCidrBlock` for `ReplaceRoute`).
- Validate the affected route table, routes, or associations based on the API call:
- For `ReplaceRoute`: Look for changes in specific routes using `destinationCidrBlock`.
- For `ReplaceRouteTableAssociation`: Review the new association details (e.g., subnet ID).
- For `DeleteRouteTable`: Confirm the `routeTableId` of the deleted table.
- For `DisassociateRouteTable`: Verify the disassociated resources.
- **Review User Context**:
- **User Identity**: Inspect the `aws.cloudtrail.user_identity.arn` field to determine the user or role initiating the action. Investigate whether this user is authorized to perform these operations.
- **Access Key ID**: Check the `aws.cloudtrail.user_identity.access_key_id` field to identify if the access key used was expected or potentially compromised.
- **Access Patterns**: Validate whether the user or role has a history of performing route table modifications and whether this aligns with their expected responsibilities.
- **Analyze Request Details**:
- **Action Type**: Verify the specific API call in the `event.action` field (e.g., `ReplaceRoute`, `DeleteRouteTable`) to understand the nature of the modification.
- **Source IP and Geolocation**: Examine the `source.address` and `source.geo` fields to confirm whether the request originated from a trusted location. Suspicious geolocations or IPs may indicate adversarial activity.
- **User Agent**: Review the `user_agent.original` field to determine the tool used for the request (e.g., AWS CLI, Terraform). Unusual or custom user agents may indicate malicious intent.
- **Correlate with Other Activity**:
- **Concurrent API Calls**: Look for related API calls (e.g., `CreateRoute`, `AuthorizeSecurityGroupIngress`, or `ModifyInstanceAttribute`) from the same user or IP to detect broader attack patterns.
- **IAM Changes**: Investigate whether any IAM policy updates or privilege escalation attempts preceded this activity.
- **Unusual Volume of Changes**: Check if the user has performed multiple route table modifications or deletions in a short timeframe.
- **Validate the Intent**:
- **Planned Changes**: Confirm with administrators whether the route table changes were part of a planned update or maintenance activity.
- **Permissions and Justification**: Ensure that the user or role has the least privilege necessary for these actions and that there is a valid reason for modifying the route table.
### False Positive Analysis
- **Routine Administration**: Route table modifications are often part of routine administrative tasks, such as creating new routes, updating associations, or removing unused resources.
- **Automation Tools**: Automated workflows, such as those executed by Terraform or CloudFormation, may trigger these events. Verify whether the `user_agent.original` field or source IP matches known automation tools.
- **Maintenance or Scaling**: Confirm whether these actions align with maintenance activities or scaling events (e.g., adding or removing subnets).
### Response and Remediation
- **Revoke Unauthorized Permissions**: If unauthorized, remove permissions for `ec2:ReplaceRoute`, `ec2:DeleteRouteTable`, or other related actions from the user or role.
- **Restore the Route Table**:
- If critical networking was impacted, restore the route table or reapply previous configurations from backups or Terraform state files.
- Verify connectivity to affected subnets or instances to ensure no disruptions to services.
- **Audit IAM Policies**:
- Limit route table modification permissions to specific trusted users, roles, or automation accounts.
- Implement conditions in IAM policies, such as source IP restrictions, to reduce the risk of unauthorized access.
- **Monitor and Alert**:
- Set up additional alerts for unexpected route table modifications or deletions.
- Use VPC flow logs and CloudTrail to monitor for related suspicious activity.
- **Secure Automation**: Ensure automation tools, such as Terraform or CloudFormation, are configured securely and that their credentials are stored in secure locations like AWS Secrets Manager.
""",
"query": """event.dataset: "aws.cloudtrail"
and event.provider: "ec2.amazonaws.com"
and event.action:(
"ReplaceRoute" or
"ReplaceRouteTableAssociation" or
"DeleteRouteTable" or
"DeleteRoute" or
"DisassociateRouteTable"
)
and event.outcome: "success"
and not source.address: (
"cloudformation.amazonaws.com" or
"servicecatalog.amazonaws.com" or
"fsx.amazonaws.com"
)
""",
"references": [
"https://github.com/easttimor/aws-incident-response#network-routing",
"https://docs.datadoghq.com/security_platform/default_rules/aws-ec2-route-table-modified/",
"https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceRoute.html",
"https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ReplaceRouteTableAssociation",
"https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteRouteTable.html",
"https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteRoute.html",
"https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DisassociateRouteTable.html"
],
"related_integrations": [
{
"integration": "cloudtrail",
"package": "aws",
"version": "^2.0.0"
}
],
"required_fields": [
{
"ecs": true,
"name": "event.action",
"type": "keyword"
},
{
"ecs": true,
"name": "event.dataset",
"type": "keyword"
},
{
"ecs": true,
"name": "event.outcome",
"type": "keyword"
},
{
"ecs": true,
"name": "event.provider",
"type": "keyword"
},
{
"ecs": true,
"name": "source.address",
"type": "keyword"
}
],
"risk_score": 21,
"rule_id": "e7cd5982-17c8-4959-874c-633acde7d426",
"severity": "low",
"tags": [
"Domain: Cloud",
"Data Source: AWS",
"Data Source: Amazon Web Services",
"Data Source: AWS EC2",
"Use Case: Network Security Monitoring",
"Resources: Investigation Guide",
"Tactic: Persistence"
],
"threat": [
{
"framework": "MITRE ATT&CK",
"tactic": {
"id": "TA0003",
"name": "Persistence",
"reference": "https://attack.mitre.org/tactics/TA0003/"
},
"technique": []
}
],
"timestamp_override": "event.ingested",
"type": "new_terms",
"version": 209
},
"type": "security-rule",
"references": [],
"managed": true,
"coreMigrationVersion": "8.8.0",
"updated_at": "2025-03-11T07:21:44.889Z",
"created_at": "2025-03-11T07:21:44.889Z"
}
```
</details>
- Remove the base versions
<details>
<summary>Remove Threat Match rule's base version</summary>
```
DELETE .kibana_security_solution/_doc/security-rule:aab184d3-72b3-4639-b242-6597c99d8bca_9
```
</details>
<details>
<summary>Remove New Terms rule's base version</summary>
```
DELETE .kibana_security_solution/_doc/security-rule:e7cd5982-17c8-4959-874c-633acde7d426_208
```
</details>
- Customize rule data source
- Open Prebuilt Rule Upgrade flyout and notice there are fields with solvable conflict
- Bulk update rules
- Notice Threat Match Index and New Terms Fields got target values. Data source value was set to target as well. Rules aren't marked as customized (there is not a "Modified" badge).
