Commit graph

67544 commits

Author SHA1 Message Date
Khristinin Nikita
0cf5ba15fe
Fix exceptions flyout disappearing (#166914)
## Summary
fix:
[https://github.com/elastic/kibana/issues/166616](https://github.com/elastic/kibana/issues/166616)

When we open exception flyout we do request a rule
Then in the rule details page, `alertDefaultFilters` was memoized based
on whole rule object
And if the rule changes it rerenders the whole alerts table. 

In the attached video it rule changes because of rule execution time.

I make `useMemo` and `use effect` for these cases really on rule
property, but not full object




eba7c3ce-84b9-47a7-8bc9-a15bc0179e2c
2023-09-26 08:01:59 +02:00
Kibana Machine
b3c1ba265f
[api-docs] 2023-09-26 Daily api_docs build (#167210)
Generated by
https://buildkite.com/elastic/kibana-api-docs-daily/builds/472
2023-09-26 00:42:16 -04:00
Lisa Cawley
78e887884c
[DOCS] Move preconfigured Tines connector details (#166217)
Co-authored-by: Ying Mao <ying.mao@elastic.co>
2023-09-25 17:58:53 -07:00
Alex Szabo
e81728ee96
Fix typecheck foundations (#167060)
## Summary
This PR is the core part of #166813. The original work seems to grow
large, and we'd like to enable a preventive check beforehand to prevent
more errors from entering the codebase.

The idea is to have a selective type check that would only check changed
files' projects.
- [x] when there's no extra label, run the selective type check only on
the diffing files' projects (success:
https://buildkite.com/elastic/kibana-pull-request/builds/161837)
- [x] when the label `ci:hard-typecheck` is present, run the regular
(but now, working) full typecheck (expected to fail: )

cc: @watson

---------

Co-authored-by: Brad White <brad.white@elastic.co>
Co-authored-by: Thomas Watson <w@tson.dk>
Co-authored-by: Thomas Watson <watson@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 16:28:56 -07:00
Søren Louv-Jansen
4662960980
[APM] Add permissions for "input-only" package (#166234)
Closes: https://github.com/elastic/kibana/issues/164936

This grants the necessary permissions to APM Server when running under
fleet.
2023-09-25 16:06:35 -07:00
Catherine Liu
1066eb3d59
[Dashboard] Focus on a single panel (#165417) 2023-09-25 15:37:30 -07:00
Melissa Alvarez
d4bb52b8b2
[ML] Data Frame Analytics Trained models flyout: update horizontal steps size (#167095)
## Summary

Related issue: https://github.com/elastic/kibana/issues/162831

- uses small size for EuiStepsHorizontal component

<img width="1354" alt="image"
src="cfe55fce-2fdf-4b6f-ac91-291c96c7c54c">



### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 16:19:25 -06:00
Quynh Nguyen (Quinn)
7a9a916260
[ML] Hide Exclude frozen data tier option in Transforms creation page (#166622)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 15:10:52 -07:00
Joseph McElroy
3043bed962
[Serverless Search] Serverless Getting Started UI Polish (#167118)
Fixes a long list of julian's UI bugs. Tested on both stateful and
serverless. See videos on visual fixes.


1a450bf6-7477-40a4-a020-a5172b56ef4c


92b40ecd-d888-4fd6-af91-045e81a1843f

Things to note:
- I had to adjust the asset path here as locally on main the images were
broken (the header for example).
2023-09-25 13:11:12 -07:00
Lisa Cawley
1c427a4a1e
Edit defaultModel generative AI connector description (#166405) 2023-09-25 13:09:30 -07:00
Nicolas Chaulet
f73f70ccdd
[Fleet] Improve config output validation for default output (#167085) 2023-09-25 15:01:56 -04:00
Patryk Kopyciński
da2695fef3
[security_solution] Skip search_filter.cy.ts (#167177)
## Summary

<img width="984" alt="image"
src="14775f02-5370-4e8f-90c0-267049c6ece4">
2023-09-25 13:34:47 -05:00
Kyle Pollich
c20d177a03
[Fleet] Increase package install max timeout + add concurrency control to rollovers (#166775)
Fixes https://github.com/elastic/kibana/issues/166761
Ref https://github.com/elastic/kibana/issues/162772

## Summary

- Increase overall timeout for waiting to retry "stuck" installations
from 1 minute to 30 minutes
- Add `pMap` concurrency control limiting concurrent `putMapping` +
`rollover` requests to mitigate ES load

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 11:05:03 -07:00
Devon Thomson
f23f2f49db
[Canvas] Remove Kui style sheet import (#167054)
removes import of the kui_light stylesheet from Canvas shareable runtime.
2023-09-25 14:04:31 -04:00
Devon Thomson
d941c4a565
unskip serverless dashboard import tests (#167161)
Unskips the serverless Dashboard import test.
2023-09-25 13:58:12 -04:00
Andrew Macri
077be69de1
[Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER (#167097)
## [Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER

This PR integrates [LangChain](https://www.langchain.com/) [Agents](https://js.langchain.com/docs/modules/agents/) and [Tools](https://js.langchain.com/docs/modules/agents/tools/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant).

These abstractions enable the LLM to dynamically choose whether or not to query, via [ELSER](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html), an [ES|QL](https://www.elastic.co/blog/elasticsearch-query-language-esql) knowledge base. Context from the knowledge base is used to generate `ES|QL` queries, or answer questions about `ES|QL`.

Registration of the tool occurs in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:

```typescript
  const tools: Tool[] = [
    new ChainTool({
      name: 'esql-language-knowledge-base',
      description:
        'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
      chain,
    }),
  ];
```

The `tools` array above may be updated in future PRs to include, for example, an `ES|QL` query validator endpoint.

### Details

The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:

1. Creates a `RetrievalQAChain` from an `ELSER` backed `ElasticsearchStore`, which serves as a knowledge base for `ES|QL`:

```typescript
  // ELSER backed ElasticsearchStore for Knowledge Base
  const esStore = new ElasticsearchStore(esClient, KNOWLEDGE_BASE_INDEX_PATTERN, logger);
  const chain = RetrievalQAChain.fromLLM(llm, esStore.asRetriever());
```

2. Registers the chain as a tool, which may be invoked by the LLM based on its description:

```typescript
  const tools: Tool[] = [
    new ChainTool({
      name: 'esql-language-knowledge-base',
      description:
        'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
      chain,
    }),
  ];
```

3. Creates an Agent executor that combines the `tools` above, the `ActionsClientLlm` (an abstraction that calls `actionsClient.execute`), and memory of the previous messages in the conversation:

```typescript
  const executor = await initializeAgentExecutorWithOptions(tools, llm, {
    agentType: 'chat-conversational-react-description',
    memory,
    verbose: false,
  });
```

Note: Set `verbose` above to `true` to for detailed debugging output from LangChain.

4. Calls the `executor`, kicking it off with `latestMessage`:

```typescript
    await executor.call({ input: latestMessage[0].content });
```

### Changes to `x-pack/packages/kbn-elastic-assistant`

A client side change was required to the assistant, because the response returned from the agent executor is JSON. This response is parsed on the client in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx`:

```typescript
  return assistantLangChain ? getFormattedMessageContent(result) : result;
```

Client-side parsing of the response only happens when then `assistantLangChain` feature flag is `true`.

## Desk testing

Set

```typescript
assistantLangChain={true}
```

in `x-pack/plugins/security_solution/public/assistant/provider.tsx` to enable this experimental feature in development environments.

Also (optionally) set `verbose` to `true` in the following code in ``x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts``:

```typescript
  const executor = await initializeAgentExecutorWithOptions(tools, llm, {
    agentType: 'chat-conversational-react-description',
    memory,
    verbose: true,
  });
```

After setting the feature flag and optionally enabling verbose debugging output, you may ask the assistant to generate an `ES|QL` query, per the example in the next section.

### Example output

When the Elastic AI Assistant is asked:

```
From employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. "September 2019". Only show the query
```

it replies:

```
Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:

FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5
```

Per the screenshot below:

![ESQL_query_via_langchain_agents_and_tools](c5cc75da-f7aa-4a12-9078-ed531f3463e7)

The `verbose: true` output from LangChain logged to the console reveals that the prompt sent to the LLM includes text like the following:

```
Assistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.
```

along with instructions for "calling" the tool like a function.

The debugging output also reveals the agent selecting the tool, and returning results from ESLR:

```
[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  "tool": "esql-language-knowledge-base",
  "toolInput": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "log": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:ChainTool] Entering Tool run with input: "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] Entering Chain run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] Entering Retriever run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] [115ms] Exiting Retriever run with output: {
  "documents": [
    {
      "pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
```

The documents containing `ES|QL` examples, retrieved from ELSER, are sent back to the LLM to answer the original question, per the abridged output below:

```
[llm/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index,
```

### Complete (verbose) LangChain output from the example

The following `verbose: true` output from LangChain below was produced via the example in the previous section:

```
[chain/start] [1:chain:AgentExecutor] Entering Chain run with input: {
  "input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
  "chat_history": []
}
[chain/start] [1:chain:AgentExecutor > 2:chain:LLMChain] Entering Chain run with input: {
  "input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
  "chat_history": [],
  "agent_scratchpad": [],
  "stop": [
    "Observation:"
  ]
}
[llm/start] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "[{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"SystemMessage\"],\"kwargs\":{\"content\":\"Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. However, above all else, all responses must adhere to the format of RESPONSE FORMAT INSTRUCTIONS.\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nOutput a JSON markdown code snippet containing a valid JSON object in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": string, // The action to take. Must be one of [esql-language-knowledge-base]\\n    \\\"action_input\\\": string // The input to the action. May be a stringified object.\\n}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly and conversationally to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": \\\"Final Answer\\\",\\n    \\\"action_input\\\": string // You should put what you want to return to use here and make sure to use valid json newline characters.\\n}\\n```\\n\\nFor both options, remember to always include the surrounding markdown code snippet delimiters (begin with \\\"```json\\\" and end with \\\"```\\\")!\\n\\n\\nUSER'S INPUT\\n--------------------\\nHere is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n\\n\\n\\n\\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \\\"September 2019\\\". Only show the query\",\"additional_kwargs\":{}}}]"
  ]
}
[llm/end] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:ActionsClientLlm] [3.08s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 2:chain:LLMChain] [3.09s] Exiting Chain run with output: {
  "text": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  "tool": "esql-language-knowledge-base",
  "toolInput": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "log": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:ChainTool] Entering Tool run with input: "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] Entering Chain run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] Entering Retriever run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] [115ms] Exiting Retriever run with output: {
  "documents": [
    {
      "pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_format.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_trunc.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/source_commands/from.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/processing_commands/where.asciidoc"
      }
    }
  ]
}
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain] Entering Chain run with input: {
  "question": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "input_documents": [
    {
      "pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_format.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_trunc.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/source_commands/from.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/processing_commands/where.asciidoc"
      }
    }
  ],
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] Entering Chain run with input: {
  "question": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "context": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n\n\n[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n"
}
[llm/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n\n\n[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n\n\nQuestion: Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\nHelpful Answer:"
  ]
}
[llm/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] [2.23s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] [2.23s] Exiting Chain run with output: {
  "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain] [2.23s] Exiting Chain run with output: {
  "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] [2.35s] Exiting Chain run with output: {
  "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[tool/end] [1:chain:AgentExecutor > 4:tool:ChainTool] [2.35s] Exiting Tool run with output: "FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5"
[chain/start] [1:chain:AgentExecutor > 10:chain:LLMChain] Entering Chain run with input: {
  "input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
  "chat_history": [],
  "agent_scratchpad": [
    {
      "lc": 1,
      "type": "constructor",
      "id": [
        "langchain",
        "schema",
        "AIMessage"
      ],
      "kwargs": {
        "content": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```",
        "additional_kwargs": {}
      }
    },
    {
      "lc": 1,
      "type": "constructor",
      "id": [
        "langchain",
        "schema",
        "HumanMessage"
      ],
      "kwargs": {
        "content": "TOOL RESPONSE:\n---------------------\nFROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5\n\nUSER'S INPUT\n--------------------\n\nOkay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.",
        "additional_kwargs": {}
      }
    }
  ],
  "stop": [
    "Observation:"
  ]
}
[llm/start] [1:chain:AgentExecutor > 10:chain:LLMChain > 11:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "[{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"SystemMessage\"],\"kwargs\":{\"content\":\"Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. However, above all else, all responses must adhere to the format of RESPONSE FORMAT INSTRUCTIONS.\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nOutput a JSON markdown code snippet containing a valid JSON object in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": string, // The action to take. Must be one of [esql-language-knowledge-base]\\n    \\\"action_input\\\": string // The input to the action. May be a stringified object.\\n}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly and conversationally to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": \\\"Final Answer\\\",\\n    \\\"action_input\\\": string // You should put what you want to return to use here and make sure to use valid json newline characters.\\n}\\n```\\n\\nFor both options, remember to always include the surrounding markdown code snippet delimiters (begin with \\\"```json\\\" and end with \\\"```\\\")!\\n\\n\\nUSER'S INPUT\\n--------------------\\nHere is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n\\n\\n\\n\\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \\\"September 2019\\\". Only show the query\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"AIMessage\"],\"kwargs\":{\"content\":\"```json\\n{\\n    \\\"action\\\": \\\"esql-language-knowledge-base\\\",\\n    \\\"action_input\\\": \\\"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\\\"\\n}\\n```\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOL RESPONSE:\\n---------------------\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\\n\\nUSER'S INPUT\\n--------------------\\n\\nOkay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.\",\"additional_kwargs\":{}}}]"
  ]
}
[llm/end] [1:chain:AgentExecutor > 10:chain:LLMChain > 11:llm:ActionsClientLlm] [6.47s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "```json\n{\n    \"action\": \"Final Answer\",\n    \"action_input\": \"Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\\n\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\"\n}\n```"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 10:chain:LLMChain] [6.47s] Exiting Chain run with output: {
  "text": "```json\n{\n    \"action\": \"Final Answer\",\n    \"action_input\": \"Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\\n\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\"\n}\n```"
}
[chain/end] [1:chain:AgentExecutor] [11.91s] Exiting Chain run with output: {
  "output": "Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\n\nFROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
```
2023-09-25 11:55:19 -06:00
Paul Tavares
3f03264dc0
[Security Solution][Endpoint] Refactor Cypress login task and ensure consistent use of users across ESS and Serverless tests (#166958)
## Summary

- Cypress `login` task refactored:
- `login(user?)` : logs use in using the default `user` or one of the
users supported by security solution and endpoint management tests
- `login.with(username, password)` : Logs a user in by using `username`
and `password`
- `login.withCustomRole(role)` : creates the provided `role`, creates a
user for it by the same role name and logs in with it
- The Cypress process for loading users into Kibana only applies to
non-serverless (at the moment). For serverless, it only validates that
the `username` being used is one of the approved user names that applies
to serverless
- FYI: the creation/availability of serverless roles/users for testing
is an ongoing effort by the kibana ops team
- New generic `RoleAndUserLoader` class. Is initialized with an map of
`Roles` and provide a standard interface for loading them.
- A sub-class (`EndpointSecurityTestRolesLoader`) was also created for
the endpoint security test users, which uses the existing set of role
definitions
- The `resolver_generator_script` was also updated to use the new
`EndpointSecurityTestRolesLoader` class for handling the `--rbacUser`
argument

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 13:46:35 -04:00
Stratoula Kalafateli
93ce98831a
[ES|QL] Hides system indices (#166909)
## Summary

Closes https://github.com/elastic/kibana/issues/166874

Hides indices starting with . (and considered as system from the
autocomplete)

<img width="785" alt="image"
src="9c4cce79-c844-41b6-a30e-06dad49f7c52">


Followed the exact pattern that the dataview management page is using.
2023-09-25 10:38:22 -07:00
Kevin Delemme
5b24da796d
fix(slo): date range filter format (#166989) 2023-09-25 13:09:03 -04:00
Alex Szabo
7f82102d72
[Ops] ES Serverless image verification pipeline (#166054)
## Summary
Prepares the serverless FTR tests to be runnable with a custom ES image.
(`--esServerlessImage` cli arg)
Creates a pipeline for testing and promoting ES Serverless docker
releases.

The job can be triggered here:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote
The three main env variables it takes:
- BUILDKITE_BRANCH: the kibana branch to test with (maybe not as
important)
 - BUILDKITE_COMMIT: the kibana commit to test with
- ES_SERVERLESS_IMAGE: the elasticsearch serverless image, or tag to use
from this repo:
`docker.elastic.co/elasticsearch-ci/elasticsearch-serverless`

## TODOS:
 - [x] set `latest_verified` with full img path as default
- [x] ~~find other CLIs that might need the `esServerlessImage` argument
(if the docker runner has multiple usages)~~ | I confused the `yarn es
docker` with this, because I thought we only run ES serverless in a
docker container, but `elasticsearch` can also be run in docker.
- [x] set `latest-compatible` or similar flag in a manifest in gcs for
Elastic's use-case
- [ ] ensure we can only verify "forward" (ie.: to avoid a
parameterization on old versions to set our pointers back) [on a second
thought, this might be kept as a feature to roll back (if we should ever
need that)]


There are two confusing things I couldn't sort out just yet:
#### Ambiguity in --esServerlessImage 
We can either have 2 CLI args: one for an image tag, one for an image
repo/image url, or we can have one (like I have it now) and interpret
that in the code, it can be either the image url, or the tag. It's more
flexible, but it's two things in one. Is it ok this way, or is it too
confusing?
e.g.:
```
node scripts/functional_tests --esFrom serverless --esServerlessImage docker.elastic.co/elasticsearch-ci/elasticsearch-serverless:git-8fc8f941bd4d --bail --config x-pack/test_serverless/functional/test_suites/security/config.ts

# or
node scripts/functional_tests  --esFrom serverless --esServerlessImage latest --bail --config x-pack/test_serverless/functional/test_suites/security/config.ts
```

#### Ambiguity in the default image path
The published ES Serverless images will sit on this image path:
`docker.elastic.co/elasticsearch-ci/elasticsearch-serverless`, however,
our one exception is the `latest-verified` which we will be tagging
under a different path, where we have write rights:
`docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified`.

Is it okay, that by default, we're searching in the `elasticsearch-ci`
images for any tags as parameters (after all, all the new images will be
published there), however our grand default will ultimately be
`docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified`.


## Links
Buildkite:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote
eg.:
https://buildkite.com/elastic/kibana-elasticsearch-serverless-verify-and-promote/builds/24
Closes: https://github.com/elastic/kibana/issues/162931

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 09:49:20 -07:00
Paulo Henrique
778dbf26b9
[Cloud Security] [Misconfigurations] Test coverage for the Alerts workflow (#166788) 2023-09-25 09:33:13 -07:00
Paulo Henrique
f9c35e4971
[Cloud Security] [Dashboard Navigation] Fix edit filter when navigating from dashboard (#166500) 2023-09-25 09:32:53 -07:00
Paulo Henrique
8c853b6ca7
[Cloud Security] [CSPM] Update cloud native deployment instructions (#166419) 2023-09-25 09:32:12 -07:00
Kevin Lacabane
88fdebdc81
[apm] allow retrieval of metric indices (#167041)
### Summary

Closes https://github.com/elastic/kibana/issues/166961

`/internal/apm/services/{serviceName}/infrastructure_attributes` route
was disabled in serverless as it relied on an infra API to function.
Since the infra plugin dependency was removed in
https://github.com/elastic/kibana/pull/164094 we can reenable the route

### Testing
I used a ccs cluster connected to edge-oblt and had to update the apm
indices to also search the remote_cluster
```
xpack.apm.indices.metric: remote_cluster:metrics-apm*,remote_cluster:apm*,metrics-apm*,apm*
xpack.apm.indices.transaction: remote_cluster:traces-apm*,remote_cluster:apm*,traces-apm*,apm*
xpack.apm.indices.span: remote_cluster:traces-apm*,remote_cluster:apm*,traces-apm*,apm*
xpack.apm.indices.error: remote_cluster:logs-apm*,remote_cluster:apm*,logs-apm*,apm*
```
- start serverless kibana
- navigate to Applications -> Services, we need to select a [service
linked to a
container](https://github.com/elastic/kibana/blob/main/x-pack/plugins/apm/server/routes/infrastructure/get_host_names.ts#L23)
to fully trigger the route logic (you can pick `quoteservice` if
connected to edge-oblt data)
- navigate to Logs tab
- call to `/infrastructure_attributes` is successful
2023-09-25 18:28:09 +02:00
Dima Arnautov
2bce7bbcbe
[ML] Shared service for elastic curated models (#167000)
## Summary

Adds a shared service for elastic curated models. The first use case is
to provide a default/recommended ELSER version based on the hardware of
the current cluster.

#### Why?
In 8.11 we'll provide a platform-specific version of the ELSER v2
alongside the portable one. At the moment several solutions refer to
ELSER for download/inference purposes with a `.elser_model_1` constant.
Starting 8.11 the model ID will vary, so using the `ElastcModels`
service allows retrieving the recommended version of ELSER for the
current cluster without any changes by solution teams in future
releases. It is still possible to request an older version of the model
if necessary.

#### Implementation 
- Adds a new Kibana API endpoint `/trained_models/model_downloads` that
provides a list of model definitions, with the `default` and
`recommended` flags.
- Adds a new Kibana API endpoint `/trained_models/elser_config` that
provides an ELSER configuration based on the cluster architecture.
- `getELSER` method is exposed from the plugin `setup` server-side as
part of our shared services and plugin `start` client-side.

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-09-25 09:15:35 -07:00
Aurélien FOUCRET
2f1b6ac896
[Guided onboarding] Updating search solution items. (#166953) 2023-09-25 18:02:33 +02:00
Lisa Cawley
e74ec69a0f
[DOCS] Move preconfigured Torq connector details (#166218) 2023-09-25 08:55:25 -07:00
Jonathan Budzenski
212bc53b8e skip failing test suite (#167076) 2023-09-25 10:42:29 -05:00
Sander Philipse
e4105331d3
[Search] Fix type errors (#167138)
## Summary

Fix type issues in the Search plugin.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 08:40:30 -07:00
Dario Gieselaar
13e5c076d5
[Observability AI Assistant] ES|QL query generation (#166041)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 08:39:34 -07:00
Quynh Nguyen (Quinn)
4957d87a66
[Transform] Improve loading behavior of Transform if stats takes too long or is not available (#166320)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 10:39:15 -05:00
Cauê Marcondes
2dcabd2b82
[Profiling] TS fixes (#167129)
I ran the ts check on all three projects owned by profiling (`profiling`
/ `profiling_data_access` / `kbn_profiling-utils`) and all passed now.

---------

Co-authored-by: Thomas Watson <watson@elastic.co>
2023-09-25 08:33:03 -07:00
Bruno Mercier Costa
18555d5dc4
[DOCS] Fix typo in Upgrade Assistant docs (#167031)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-09-25 10:19:27 -04:00
Vadim Kibana
dfcf60ae33
Remove deprecated EUI components in kibana_react (#166225)
## Summary

Closes https://github.com/elastic/kibana/issues/161422

In this PR:

- [x] Removes `KibanaPageTemplate` utilities from `kibana_react`
- [x] Migrates remaining users of `KibanaPageTemplate` from
`kibana_react` to the one from `@kbn/shared-ux-page-kibana-template`
- [x]
`src/plugins/home/public/application/components/tutorial_directory.js`
- [x]
`src/plugins/home/public/application/components/tutorial/tutorial.js`
  - [x] `x-pack/plugins/osquery/public/components/empty_state.tsx`

## Visual Changes

Below are the main visual differences this PR introduces.

Sample data page before:

<img width="1419" alt="Screenshot 2023-09-13 at 12 12 11"
src="cbad5bd2-9c51-486c-a48d-ac44a80cdbf7">

Sample data page now:

<img width="1456" alt="Screenshot 2023-09-13 at 12 12 18"
src="4118db78-89e8-41ed-a3d3-51a221dde857">

Tutorials page before:

<img width="1402" alt="Screenshot 2023-09-13 at 13 09 58"
src="b51af548-211a-4a59-9143-b82f2ff55678">

Tutorials page now:

<img width="1439" alt="Screenshot 2023-09-13 at 13 10 04"
src="f71590a3-7c3d-4f60-a460-47aacdfcbf09">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 16:04:28 +02:00
Alexi Doak
a71242686a
[ResponseOps] [Alerting] Removing skip on x-pack/test/alerting_api_integration/security_and_spaces/group2/tests/telemetry/alerting_and_actions_telemetry·ts (#166985)
Resolves https://github.com/elastic/kibana/issues/140973

## Summary

Failure were caused by
```
Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read',
  response: undefined
}
```
Flaky test runner x 250
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3210
2023-09-25 07:02:23 -07:00
Alex Szabo
ab2881feef
Further test failures (#167148)
## Summary
2023-09-25 15:58:28 +02:00
Maryam Saeidi
258e9848d1
Fix Errors rules link on observability alert page (#167027)
Fixes #167004

## Summary

This PR fixes the Errors stat link and improves the code using the Rules
locator.


![image](bb836360-3ee9-4b12-b079-7fa9ad26bbf7)

## 🧪 How to test
- Please check all the statuses (Disabled, Snoozed, Errors) to make sure
links work as expected.
2023-09-25 15:49:12 +02:00
Gerard Soldevila
e4a088c7eb
[Migrations] Ensure individual migrator failures do not break consistency (#166924)
## Summary

Tackles https://github.com/elastic/kibana/issues/158818

The goal of the PR is to introduce failures in single migrators at some
of the crucial steps of the migration, testing that consistency is
maintained, and that subsequent migration attempts can successfully
complete the upgrade.

This is done by _proxying_ the `Client` class, which uses the
elasticsearch-js library underneath to perform all calls to ES.
Inspired on https://github.com/elastic/kibana/pull/158995.
2023-09-25 13:24:42 +02:00
Shahzad
d8b80901d7
[Uptime] Fixes monitor details overview (#166736) 2023-09-25 12:50:57 +02:00
natasha-moore-elastic
62e087a8a8
[DOCS] Makes shards optional in Create pack API (#166639)
## Summary

* Resolves https://github.com/elastic/security-docs/issues/3822.  

* Updates the `shards` object in Create pack API to optional for 8.10.1
and 8.11.0 onwards, per https://github.com/elastic/kibana/pull/166178

* Related to changes made in
https://github.com/elastic/kibana/pull/166363.
2023-09-25 11:38:41 +01:00
Kibana Machine
e44362feae skip failing test suite (#151854) 2023-09-25 06:24:07 -04:00
Cauê Marcondes
9cbd597ebb
[Profiling] Fix set up process (#167067)
So clients reported that they got stuck in the set up screen In the
Universal Profling UI. And when the set up button was clicked an error
happened:

```
An integration policy with the name elastic-universal-profiling-collector already exists. Please rename it or choose a different name.
```

This happens because when we were checking if the Collector and
Symbolizer integrations were installed we weren't taking into
consideration that the Fleet API is paginated. So if neither integration
was available on the first page we just assumed the Profiling wasn't set
up.

This PR fixes it by adding a kuery filter in the Fleet API call to only
look for out integrations. So we don't need to worry about paginating.
2023-09-25 10:51:03 +01:00
Maxim Kholod
e6cbe509a0
[Cloud Security][Fleet] fix broken k8s manifest link (#167059)
## Summary

fixes:
- https://github.com/elastic/kibana/issues/166931

The issue was introduced in
https://github.com/elastic/kibana/pull/165127/files#diff-276f84c47e09954d668b83d633d87edc09406b69603dac7e63964b70e2342af1R120

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-09-25 11:17:10 +02:00
Alex Szabo
d601747e3a
fix: Fix hash on changed sha key (#167116)
## Summary
Infra updated the GPG keys for elastic, so we need to adjust the hash of
the gpg key.

more context:
https://elastic.slack.com/archives/C0D8P2XK5/p1695621791855459
2023-09-25 11:16:45 +02:00
Dmitrii Shevchenko
38e6b76640
[Security Solution] Extract OpenAPI codegen to a package (#166269) 2023-09-25 10:51:40 +02:00
Maxim Kholod
c7f49c200c
[Cloud Security] fix ingest pipeline for benchmark scores index (#166966)
## Summary

during the [ILM fix](https://github.com/elastic/kibana/pull/165317) for
serverless the default pipeline for the scores index was also changed by
mistake. Reverting this change


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-09-25 09:46:51 +02:00
Ido Cohen
48d293f29a
[Cloud Security] Fix wrong fleet url in cloudFormation (#166130)
solves:
- https://github.com/elastic/security-team/issues/7482

choose the correct Fleet URL in case there is more than one fleet
server.

---------

Co-authored-by: Maxim Kholod <maxim.kholod@elastic.co>
2023-09-25 09:44:55 +02:00
Stratoula Kalafateli
14e4987ae6
[ES|QL] Fixes words wrapping (#167025)
## Summary

Fixes https://github.com/elastic/kibana/issues/166648

<img width="2083" alt="image"
src="0374c577-dc52-4453-b220-a6dfc12d79e3">
2023-09-25 00:38:46 -07:00
Gerard Soldevila
b98b6d08cd
Fix flakiness on custom time range saved searches (#165454)
A bunch of tests on dashboards are customising some of the panels
settings and providing custom time ranges:

<img width="409" alt="image"
src="c869c1a3-f7db-4ccd-ad00-c5403f2b4201">

Currently, the logic is not waiting for the quick toggle animation to
complete, before proceeding to select a time range.
This can cause a flaky behavior if the logic tries to customize the
range before the button is actually available, as seen on [this failed
test](018a4c46-0e7a-4b69-9a3d-9c54c27165b0_4fcbc47e71644919129e320eea8bb3bc.html?response-content-type=text%2Fhtml&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LZWZ5UB5F%2F20230901%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230901T094837Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEB0aCXVzLWVhc3QtMSJGMEQCIGCyKcVLGPUawZubNzZdt5oZNb5v0saiIuPqXwI7rmwlAiAsOj%2Fiep94v%2BYZJtLY3Gw0m%2FmK5mJw2IcIBdNKFXgK%2BCr6Awjm%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAAaDDAzMjM3OTcwNTMwMyIMXOd1Hm6ks%2FNE37V0Ks4DgMUso7syv87hnPcC%2BB1soxvFFnj4JnNZc6ZgkLUe93z99iPFBUsqH%2BRbUTfSbjVOEJYBKGYuvp32xvSWsYNVPXKmcej18LC0yNi%2BBzoG2X%2Bj80g%2BbGMm6YfTncjPhOE0CHHqOWXts9nQ8WpDy8XOl0zfMtuiPjzOXHo9lvw2mgYDZIJIMV72FYB9JGg8FPbLQtD3rysLGNE0VDKgl5LCnYwhY1pwRCRHnVW41QfV0pwK%2FbjNf9HjdK31LQvMY%2FGPuB3M6O2CUZLsvLGfWBeGYHtkqb0hrL9ijO1Uo28ZSS1FytPftEdF0e1kAC9C5zD56HtYm55aktOWtaaC0XPWLdWWGUq%2FKQzhxSCiXK6ovATU3zI3yPNoZs92YBYmIPMOpEI40dCCpksjPwAMCiQd%2F9gMNKP5Qp5CbYd2Khy%2FeXaT8J7HOZCueN63O0j%2FtX1tbwfznhbr74lAcRQjueRYmwboZaGSDZUQ33lSSmyZk1V9WF9eJyt88oHvIx0q9bIjvOlW05DiNKfEFWYwfBywdGuvRU6eGMs1QcDNu33Lb%2BhymudM2JZmQKIjZOcb2l3Fzctp614owH4JcRlmF4%2BIa4xHeBdRlTMysS8bTIsgMK7axacGOqYBzIpC1wgZWJ1kZ0agLWCNaMIdUl%2B4xrr7w%2Fz0843WWMhRrvbJhDTHqk5UclF%2FSROAMe0FH2XEXiQ65ILyUPlrUMels5tfQ3Pp%2FJWPi9NsQJUQ1n9uLN%2BFPDOoMo8Uxg4%2FkG2O7yTkrIdArfA6pWN9I21gFMW%2BFZy9BMYltt5T65ZKOyYAIFGpLhgfBySIBCUMgwR1kusfDhf1%2FRTvtDKD2sJKN5a0IA%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=35fabe908aa7514e4a92de0ed12973af85ccfb439984fc3bdd7ef3bb8fe3419b).
(part of this [failed CI
build](https://buildkite.com/elastic/kibana-pull-request/builds/155285#018a4c46-0e7a-4b69-9a3d-9c54c27165b0))

The goal of this PR is to add a small waiting period, to make sure the
toggle animation has completed, and that the time range controls are
visible and clickable.

I used the opportunity to cleanup some "await delay millis" calls,
reusing existing logic instead.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 09:32:57 +02:00
Kibana Machine
823a1d542b
[api-docs] 2023-09-25 Daily api_docs build (#167109)
Generated by
https://buildkite.com/elastic/kibana-api-docs-daily/builds/471
2023-09-25 00:54:39 -04:00