Commit graph

33992 commits

Author SHA1 Message Date
Melissa Alvarez
d4bb52b8b2
[ML] Data Frame Analytics Trained models flyout: update horizontal steps size (#167095)
## Summary

Related issue: https://github.com/elastic/kibana/issues/162831

- uses small size for EuiStepsHorizontal component

<img width="1354" alt="image"
src="cfe55fce-2fdf-4b6f-ac91-291c96c7c54c">



### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 16:19:25 -06:00
Quynh Nguyen (Quinn)
7a9a916260
[ML] Hide Exclude frozen data tier option in Transforms creation page (#166622)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 15:10:52 -07:00
Joseph McElroy
3043bed962
[Serverless Search] Serverless Getting Started UI Polish (#167118)
Fixes a long list of julian's UI bugs. Tested on both stateful and
serverless. See videos on visual fixes.


1a450bf6-7477-40a4-a020-a5172b56ef4c


92b40ecd-d888-4fd6-af91-045e81a1843f

Things to note:
- I had to adjust the asset path here as locally on main the images were
broken (the header for example).
2023-09-25 13:11:12 -07:00
Lisa Cawley
1c427a4a1e
Edit defaultModel generative AI connector description (#166405) 2023-09-25 13:09:30 -07:00
Nicolas Chaulet
f73f70ccdd
[Fleet] Improve config output validation for default output (#167085) 2023-09-25 15:01:56 -04:00
Patryk Kopyciński
da2695fef3
[security_solution] Skip search_filter.cy.ts (#167177)
## Summary

<img width="984" alt="image"
src="14775f02-5370-4e8f-90c0-267049c6ece4">
2023-09-25 13:34:47 -05:00
Kyle Pollich
c20d177a03
[Fleet] Increase package install max timeout + add concurrency control to rollovers (#166775)
Fixes https://github.com/elastic/kibana/issues/166761
Ref https://github.com/elastic/kibana/issues/162772

## Summary

- Increase overall timeout for waiting to retry "stuck" installations
from 1 minute to 30 minutes
- Add `pMap` concurrency control limiting concurrent `putMapping` +
`rollover` requests to mitigate ES load

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 11:05:03 -07:00
Devon Thomson
f23f2f49db
[Canvas] Remove Kui style sheet import (#167054)
removes import of the kui_light stylesheet from Canvas shareable runtime.
2023-09-25 14:04:31 -04:00
Devon Thomson
d941c4a565
unskip serverless dashboard import tests (#167161)
Unskips the serverless Dashboard import test.
2023-09-25 13:58:12 -04:00
Andrew Macri
077be69de1
[Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER (#167097)
## [Security Solution] [Elastic AI Assistant] LangChain Agents and Tools integration for ES|QL query generation via ELSER

This PR integrates [LangChain](https://www.langchain.com/) [Agents](https://js.langchain.com/docs/modules/agents/) and [Tools](https://js.langchain.com/docs/modules/agents/tools/) with the [Elastic AI Assistant](https://www.elastic.co/blog/introducing-elastic-ai-assistant).

These abstractions enable the LLM to dynamically choose whether or not to query, via [ELSER](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html), an [ES|QL](https://www.elastic.co/blog/elasticsearch-query-language-esql) knowledge base. Context from the knowledge base is used to generate `ES|QL` queries, or answer questions about `ES|QL`.

Registration of the tool occurs in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:

```typescript
  const tools: Tool[] = [
    new ChainTool({
      name: 'esql-language-knowledge-base',
      description:
        'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
      chain,
    }),
  ];
```

The `tools` array above may be updated in future PRs to include, for example, an `ES|QL` query validator endpoint.

### Details

The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts`:

1. Creates a `RetrievalQAChain` from an `ELSER` backed `ElasticsearchStore`, which serves as a knowledge base for `ES|QL`:

```typescript
  // ELSER backed ElasticsearchStore for Knowledge Base
  const esStore = new ElasticsearchStore(esClient, KNOWLEDGE_BASE_INDEX_PATTERN, logger);
  const chain = RetrievalQAChain.fromLLM(llm, esStore.asRetriever());
```

2. Registers the chain as a tool, which may be invoked by the LLM based on its description:

```typescript
  const tools: Tool[] = [
    new ChainTool({
      name: 'esql-language-knowledge-base',
      description:
        'Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.',
      chain,
    }),
  ];
```

3. Creates an Agent executor that combines the `tools` above, the `ActionsClientLlm` (an abstraction that calls `actionsClient.execute`), and memory of the previous messages in the conversation:

```typescript
  const executor = await initializeAgentExecutorWithOptions(tools, llm, {
    agentType: 'chat-conversational-react-description',
    memory,
    verbose: false,
  });
```

Note: Set `verbose` above to `true` to for detailed debugging output from LangChain.

4. Calls the `executor`, kicking it off with `latestMessage`:

```typescript
    await executor.call({ input: latestMessage[0].content });
```

### Changes to `x-pack/packages/kbn-elastic-assistant`

A client side change was required to the assistant, because the response returned from the agent executor is JSON. This response is parsed on the client in `x-pack/packages/kbn-elastic-assistant/impl/assistant/api.tsx`:

```typescript
  return assistantLangChain ? getFormattedMessageContent(result) : result;
```

Client-side parsing of the response only happens when then `assistantLangChain` feature flag is `true`.

## Desk testing

Set

```typescript
assistantLangChain={true}
```

in `x-pack/plugins/security_solution/public/assistant/provider.tsx` to enable this experimental feature in development environments.

Also (optionally) set `verbose` to `true` in the following code in ``x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts``:

```typescript
  const executor = await initializeAgentExecutorWithOptions(tools, llm, {
    agentType: 'chat-conversational-react-description',
    memory,
    verbose: true,
  });
```

After setting the feature flag and optionally enabling verbose debugging output, you may ask the assistant to generate an `ES|QL` query, per the example in the next section.

### Example output

When the Elastic AI Assistant is asked:

```
From employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. "September 2019". Only show the query
```

it replies:

```
Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:

FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5
```

Per the screenshot below:

![ESQL_query_via_langchain_agents_and_tools](c5cc75da-f7aa-4a12-9078-ed531f3463e7)

The `verbose: true` output from LangChain logged to the console reveals that the prompt sent to the LLM includes text like the following:

```
Assistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.
```

along with instructions for "calling" the tool like a function.

The debugging output also reveals the agent selecting the tool, and returning results from ESLR:

```
[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  "tool": "esql-language-knowledge-base",
  "toolInput": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "log": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:ChainTool] Entering Tool run with input: "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] Entering Chain run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] Entering Retriever run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] [115ms] Exiting Retriever run with output: {
  "documents": [
    {
      "pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
```

The documents containing `ES|QL` examples, retrieved from ELSER, are sent back to the LLM to answer the original question, per the abridged output below:

```
[llm/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index,
```

### Complete (verbose) LangChain output from the example

The following `verbose: true` output from LangChain below was produced via the example in the previous section:

```
[chain/start] [1:chain:AgentExecutor] Entering Chain run with input: {
  "input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
  "chat_history": []
}
[chain/start] [1:chain:AgentExecutor > 2:chain:LLMChain] Entering Chain run with input: {
  "input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
  "chat_history": [],
  "agent_scratchpad": [],
  "stop": [
    "Observation:"
  ]
}
[llm/start] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "[{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"SystemMessage\"],\"kwargs\":{\"content\":\"Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. However, above all else, all responses must adhere to the format of RESPONSE FORMAT INSTRUCTIONS.\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nOutput a JSON markdown code snippet containing a valid JSON object in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": string, // The action to take. Must be one of [esql-language-knowledge-base]\\n    \\\"action_input\\\": string // The input to the action. May be a stringified object.\\n}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly and conversationally to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": \\\"Final Answer\\\",\\n    \\\"action_input\\\": string // You should put what you want to return to use here and make sure to use valid json newline characters.\\n}\\n```\\n\\nFor both options, remember to always include the surrounding markdown code snippet delimiters (begin with \\\"```json\\\" and end with \\\"```\\\")!\\n\\n\\nUSER'S INPUT\\n--------------------\\nHere is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n\\n\\n\\n\\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \\\"September 2019\\\". Only show the query\",\"additional_kwargs\":{}}}]"
  ]
}
[llm/end] [1:chain:AgentExecutor > 2:chain:LLMChain > 3:llm:ActionsClientLlm] [3.08s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 2:chain:LLMChain] [3.09s] Exiting Chain run with output: {
  "text": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  "tool": "esql-language-knowledge-base",
  "toolInput": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "log": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```"
}
[tool/start] [1:chain:AgentExecutor > 4:tool:ChainTool] Entering Tool run with input: "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] Entering Chain run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] Entering Retriever run with input: {
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[retriever/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 6:retriever:VectorStoreRetriever] [115ms] Exiting Retriever run with output: {
  "documents": [
    {
      "pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_format.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_trunc.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/source_commands/from.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/processing_commands/where.asciidoc"
      }
    }
  ]
}
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain] Entering Chain run with input: {
  "question": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "input_documents": [
    {
      "pageContent": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_format.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/functions/date_trunc.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/source_commands/from.asciidoc"
      }
    },
    {
      "pageContent": "[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n",
      "metadata": {
        "source": "/Users/andrew.goldstein/Projects/forks/spong/kibana/x-pack/plugins/elastic_assistant/server/knowledge_base/esql/docs/processing_commands/where.asciidoc"
      }
    }
  ],
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'."
}
[chain/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] Entering Chain run with input: {
  "question": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "query": "Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.",
  "context": "[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n\n\n[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n"
}
[llm/start] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n[[esql-date_format]]\n=== `DATE_FORMAT`\nReturns a string representation of a date in the provided format. If no format\nis specified, the `yyyy-MM-dd'T'HH:mm:ss.SSSZ` format is used.\n\n[source,esql]\n----\nFROM employees\n| KEEP first_name, last_name, hire_date\n| EVAL hired = DATE_FORMAT(hire_date, \"YYYY-MM-dd\")\n----\n\n\n[[esql-date_trunc]]\n=== `DATE_TRUNC`\nRounds down a date to the closest interval. Intervals can be expressed using the\n<<esql-timespan-literals,timespan literal syntax>>.\n\n[source,esql]\n----\nFROM employees\n| EVAL year_hired = DATE_TRUNC(1 year, hire_date)\n| STATS count(emp_no) BY year_hired\n| SORT year_hired\n----\n\n\n[[esql-from]]\n=== `FROM`\n\nThe `FROM` source command returns a table with up to 10,000 documents from a\ndata stream, index, or alias. Each row in the resulting table represents a\ndocument. Each column corresponds to a field, and can be accessed by the name\nof that field.\n\n[source,esql]\n----\nFROM employees\n----\n\nYou can use <<api-date-math-index-names,date math>> to refer to indices, aliases\nand data streams. This can be useful for time series data, for example to access\ntoday's index:\n\n[source,esql]\n----\nFROM <logs-{now/d}>\n----\n\nUse comma-separated lists or wildcards to query multiple data streams, indices,\nor aliases:\n\n[source,esql]\n----\nFROM employees-00001,employees-*\n----\n\n\n[[esql-where]]\n=== `WHERE`\n\nUse `WHERE` to produce a table that contains all the rows from the input table\nfor which the provided condition evaluates to `true`:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=where]\n----\n\nWhich, if `still_hired` is a boolean field, can be simplified to:\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereBoolean]\n----\n\n[discrete]\n==== Operators\n\nRefer to <<esql-operators>> for an overview of the supported operators.\n\n[discrete]\n==== Functions\n`WHERE` supports various functions for calculating values. Refer to\n<<esql-functions,Functions>> for more information.\n\n[source,esql]\n----\ninclude::{esql-specs}/docs.csv-spec[tag=whereFunction]\n----\n\n\nQuestion: Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\nHelpful Answer:"
  ]
}
[llm/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:ActionsClientLlm] [2.23s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] [2.23s] Exiting Chain run with output: {
  "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain > 7:chain:StuffDocumentsChain] [2.23s] Exiting Chain run with output: {
  "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[chain/end] [1:chain:AgentExecutor > 4:tool:ChainTool > 5:chain:RetrievalQAChain] [2.35s] Exiting Chain run with output: {
  "text": "FROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
[tool/end] [1:chain:AgentExecutor > 4:tool:ChainTool] [2.35s] Exiting Tool run with output: "FROM employees
| KEEP emp_no, hire_date
| EVAL month_year = DATE_FORMAT(hire_date, "MMMM YYYY")
| SORT hire_date
| LIMIT 5"
[chain/start] [1:chain:AgentExecutor > 10:chain:LLMChain] Entering Chain run with input: {
  "input": "\n\n\n\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \"September 2019\". Only show the query",
  "chat_history": [],
  "agent_scratchpad": [
    {
      "lc": 1,
      "type": "constructor",
      "id": [
        "langchain",
        "schema",
        "AIMessage"
      ],
      "kwargs": {
        "content": "```json\n{\n    \"action\": \"esql-language-knowledge-base\",\n    \"action_input\": \"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\"\n}\n```",
        "additional_kwargs": {}
      }
    },
    {
      "lc": 1,
      "type": "constructor",
      "id": [
        "langchain",
        "schema",
        "HumanMessage"
      ],
      "kwargs": {
        "content": "TOOL RESPONSE:\n---------------------\nFROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5\n\nUSER'S INPUT\n--------------------\n\nOkay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.",
        "additional_kwargs": {}
      }
    }
  ],
  "stop": [
    "Observation:"
  ]
}
[llm/start] [1:chain:AgentExecutor > 10:chain:LLMChain > 11:llm:ActionsClientLlm] Entering LLM run with input: {
  "prompts": [
    "[{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"SystemMessage\"],\"kwargs\":{\"content\":\"Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. However, above all else, all responses must adhere to the format of RESPONSE FORMAT INSTRUCTIONS.\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\nesql-language-knowledge-base: Call this for knowledge on how to build an ESQL query, or answer questions about the ES|QL query language.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nOutput a JSON markdown code snippet containing a valid JSON object in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": string, // The action to take. Must be one of [esql-language-knowledge-base]\\n    \\\"action_input\\\": string // The input to the action. May be a stringified object.\\n}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly and conversationally to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{\\n    \\\"action\\\": \\\"Final Answer\\\",\\n    \\\"action_input\\\": string // You should put what you want to return to use here and make sure to use valid json newline characters.\\n}\\n```\\n\\nFor both options, remember to always include the surrounding markdown code snippet delimiters (begin with \\\"```json\\\" and end with \\\"```\\\")!\\n\\n\\nUSER'S INPUT\\n--------------------\\nHere is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n\\n\\n\\n\\nFrom employees, I want to see the 5 earliest employees (hire_date), I want to display only the month and the year that they were hired in and their employee number (emp_no). Format the date as e.g. \\\"September 2019\\\". Only show the query\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"AIMessage\"],\"kwargs\":{\"content\":\"```json\\n{\\n    \\\"action\\\": \\\"esql-language-knowledge-base\\\",\\n    \\\"action_input\\\": \\\"Display the 'emp_no', month and year of the 5 earliest employees by 'hire_date'. Format the date as 'Month Year'.\\\"\\n}\\n```\",\"additional_kwargs\":{}}},{\"lc\":1,\"type\":\"constructor\",\"id\":[\"langchain\",\"schema\",\"HumanMessage\"],\"kwargs\":{\"content\":\"TOOL RESPONSE:\\n---------------------\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\\n\\nUSER'S INPUT\\n--------------------\\n\\nOkay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.\",\"additional_kwargs\":{}}}]"
  ]
}
[llm/end] [1:chain:AgentExecutor > 10:chain:LLMChain > 11:llm:ActionsClientLlm] [6.47s] Exiting LLM run with output: {
  "generations": [
    [
      {
        "text": "```json\n{\n    \"action\": \"Final Answer\",\n    \"action_input\": \"Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\\n\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\"\n}\n```"
      }
    ]
  ]
}
[chain/end] [1:chain:AgentExecutor > 10:chain:LLMChain] [6.47s] Exiting Chain run with output: {
  "text": "```json\n{\n    \"action\": \"Final Answer\",\n    \"action_input\": \"Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\\n\\nFROM employees\\n| KEEP emp_no, hire_date\\n| EVAL month_year = DATE_FORMAT(hire_date, \\\"MMMM YYYY\\\")\\n| SORT hire_date\\n| LIMIT 5\"\n}\n```"
}
[chain/end] [1:chain:AgentExecutor] [11.91s] Exiting Chain run with output: {
  "output": "Here is the query to get the employee number and the formatted hire date for the 5 earliest employees by hire_date:\n\nFROM employees\n| KEEP emp_no, hire_date\n| EVAL month_year = DATE_FORMAT(hire_date, \"MMMM YYYY\")\n| SORT hire_date\n| LIMIT 5"
}
```
2023-09-25 11:55:19 -06:00
Paul Tavares
3f03264dc0
[Security Solution][Endpoint] Refactor Cypress login task and ensure consistent use of users across ESS and Serverless tests (#166958)
## Summary

- Cypress `login` task refactored:
- `login(user?)` : logs use in using the default `user` or one of the
users supported by security solution and endpoint management tests
- `login.with(username, password)` : Logs a user in by using `username`
and `password`
- `login.withCustomRole(role)` : creates the provided `role`, creates a
user for it by the same role name and logs in with it
- The Cypress process for loading users into Kibana only applies to
non-serverless (at the moment). For serverless, it only validates that
the `username` being used is one of the approved user names that applies
to serverless
- FYI: the creation/availability of serverless roles/users for testing
is an ongoing effort by the kibana ops team
- New generic `RoleAndUserLoader` class. Is initialized with an map of
`Roles` and provide a standard interface for loading them.
- A sub-class (`EndpointSecurityTestRolesLoader`) was also created for
the endpoint security test users, which uses the existing set of role
definitions
- The `resolver_generator_script` was also updated to use the new
`EndpointSecurityTestRolesLoader` class for handling the `--rbacUser`
argument

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 13:46:35 -04:00
Kevin Delemme
5b24da796d
fix(slo): date range filter format (#166989) 2023-09-25 13:09:03 -04:00
Paulo Henrique
778dbf26b9
[Cloud Security] [Misconfigurations] Test coverage for the Alerts workflow (#166788) 2023-09-25 09:33:13 -07:00
Paulo Henrique
f9c35e4971
[Cloud Security] [Dashboard Navigation] Fix edit filter when navigating from dashboard (#166500) 2023-09-25 09:32:53 -07:00
Paulo Henrique
8c853b6ca7
[Cloud Security] [CSPM] Update cloud native deployment instructions (#166419) 2023-09-25 09:32:12 -07:00
Kevin Lacabane
88fdebdc81
[apm] allow retrieval of metric indices (#167041)
### Summary

Closes https://github.com/elastic/kibana/issues/166961

`/internal/apm/services/{serviceName}/infrastructure_attributes` route
was disabled in serverless as it relied on an infra API to function.
Since the infra plugin dependency was removed in
https://github.com/elastic/kibana/pull/164094 we can reenable the route

### Testing
I used a ccs cluster connected to edge-oblt and had to update the apm
indices to also search the remote_cluster
```
xpack.apm.indices.metric: remote_cluster:metrics-apm*,remote_cluster:apm*,metrics-apm*,apm*
xpack.apm.indices.transaction: remote_cluster:traces-apm*,remote_cluster:apm*,traces-apm*,apm*
xpack.apm.indices.span: remote_cluster:traces-apm*,remote_cluster:apm*,traces-apm*,apm*
xpack.apm.indices.error: remote_cluster:logs-apm*,remote_cluster:apm*,logs-apm*,apm*
```
- start serverless kibana
- navigate to Applications -> Services, we need to select a [service
linked to a
container](https://github.com/elastic/kibana/blob/main/x-pack/plugins/apm/server/routes/infrastructure/get_host_names.ts#L23)
to fully trigger the route logic (you can pick `quoteservice` if
connected to edge-oblt data)
- navigate to Logs tab
- call to `/infrastructure_attributes` is successful
2023-09-25 18:28:09 +02:00
Dima Arnautov
2bce7bbcbe
[ML] Shared service for elastic curated models (#167000)
## Summary

Adds a shared service for elastic curated models. The first use case is
to provide a default/recommended ELSER version based on the hardware of
the current cluster.

#### Why?
In 8.11 we'll provide a platform-specific version of the ELSER v2
alongside the portable one. At the moment several solutions refer to
ELSER for download/inference purposes with a `.elser_model_1` constant.
Starting 8.11 the model ID will vary, so using the `ElastcModels`
service allows retrieving the recommended version of ELSER for the
current cluster without any changes by solution teams in future
releases. It is still possible to request an older version of the model
if necessary.

#### Implementation 
- Adds a new Kibana API endpoint `/trained_models/model_downloads` that
provides a list of model definitions, with the `default` and
`recommended` flags.
- Adds a new Kibana API endpoint `/trained_models/elser_config` that
provides an ELSER configuration based on the cluster architecture.
- `getELSER` method is exposed from the plugin `setup` server-side as
part of our shared services and plugin `start` client-side.

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-09-25 09:15:35 -07:00
Aurélien FOUCRET
2f1b6ac896
[Guided onboarding] Updating search solution items. (#166953) 2023-09-25 18:02:33 +02:00
Lisa Cawley
e74ec69a0f
[DOCS] Move preconfigured Torq connector details (#166218) 2023-09-25 08:55:25 -07:00
Jonathan Budzenski
212bc53b8e skip failing test suite (#167076) 2023-09-25 10:42:29 -05:00
Sander Philipse
e4105331d3
[Search] Fix type errors (#167138)
## Summary

Fix type issues in the Search plugin.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 08:40:30 -07:00
Dario Gieselaar
13e5c076d5
[Observability AI Assistant] ES|QL query generation (#166041)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 08:39:34 -07:00
Quynh Nguyen (Quinn)
4957d87a66
[Transform] Improve loading behavior of Transform if stats takes too long or is not available (#166320)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 10:39:15 -05:00
Cauê Marcondes
2dcabd2b82
[Profiling] TS fixes (#167129)
I ran the ts check on all three projects owned by profiling (`profiling`
/ `profiling_data_access` / `kbn_profiling-utils`) and all passed now.

---------

Co-authored-by: Thomas Watson <watson@elastic.co>
2023-09-25 08:33:03 -07:00
Vadim Kibana
dfcf60ae33
Remove deprecated EUI components in kibana_react (#166225)
## Summary

Closes https://github.com/elastic/kibana/issues/161422

In this PR:

- [x] Removes `KibanaPageTemplate` utilities from `kibana_react`
- [x] Migrates remaining users of `KibanaPageTemplate` from
`kibana_react` to the one from `@kbn/shared-ux-page-kibana-template`
- [x]
`src/plugins/home/public/application/components/tutorial_directory.js`
- [x]
`src/plugins/home/public/application/components/tutorial/tutorial.js`
  - [x] `x-pack/plugins/osquery/public/components/empty_state.tsx`

## Visual Changes

Below are the main visual differences this PR introduces.

Sample data page before:

<img width="1419" alt="Screenshot 2023-09-13 at 12 12 11"
src="cbad5bd2-9c51-486c-a48d-ac44a80cdbf7">

Sample data page now:

<img width="1456" alt="Screenshot 2023-09-13 at 12 12 18"
src="4118db78-89e8-41ed-a3d3-51a221dde857">

Tutorials page before:

<img width="1402" alt="Screenshot 2023-09-13 at 13 09 58"
src="b51af548-211a-4a59-9143-b82f2ff55678">

Tutorials page now:

<img width="1439" alt="Screenshot 2023-09-13 at 13 10 04"
src="f71590a3-7c3d-4f60-a460-47aacdfcbf09">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 16:04:28 +02:00
Alexi Doak
a71242686a
[ResponseOps] [Alerting] Removing skip on x-pack/test/alerting_api_integration/security_and_spaces/group2/tests/telemetry/alerting_and_actions_telemetry·ts (#166985)
Resolves https://github.com/elastic/kibana/issues/140973

## Summary

Failure were caused by
```
Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read',
  response: undefined
}
```
Flaky test runner x 250
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3210
2023-09-25 07:02:23 -07:00
Maryam Saeidi
258e9848d1
Fix Errors rules link on observability alert page (#167027)
Fixes #167004

## Summary

This PR fixes the Errors stat link and improves the code using the Rules
locator.


![image](bb836360-3ee9-4b12-b079-7fa9ad26bbf7)

## 🧪 How to test
- Please check all the statuses (Disabled, Snoozed, Errors) to make sure
links work as expected.
2023-09-25 15:49:12 +02:00
Shahzad
d8b80901d7
[Uptime] Fixes monitor details overview (#166736) 2023-09-25 12:50:57 +02:00
Kibana Machine
e44362feae skip failing test suite (#151854) 2023-09-25 06:24:07 -04:00
Cauê Marcondes
9cbd597ebb
[Profiling] Fix set up process (#167067)
So clients reported that they got stuck in the set up screen In the
Universal Profling UI. And when the set up button was clicked an error
happened:

```
An integration policy with the name elastic-universal-profiling-collector already exists. Please rename it or choose a different name.
```

This happens because when we were checking if the Collector and
Symbolizer integrations were installed we weren't taking into
consideration that the Fleet API is paginated. So if neither integration
was available on the first page we just assumed the Profiling wasn't set
up.

This PR fixes it by adding a kuery filter in the Fleet API call to only
look for out integrations. So we don't need to worry about paginating.
2023-09-25 10:51:03 +01:00
Maxim Kholod
e6cbe509a0
[Cloud Security][Fleet] fix broken k8s manifest link (#167059)
## Summary

fixes:
- https://github.com/elastic/kibana/issues/166931

The issue was introduced in
https://github.com/elastic/kibana/pull/165127/files#diff-276f84c47e09954d668b83d633d87edc09406b69603dac7e63964b70e2342af1R120

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-09-25 11:17:10 +02:00
Dmitrii Shevchenko
38e6b76640
[Security Solution] Extract OpenAPI codegen to a package (#166269) 2023-09-25 10:51:40 +02:00
Maxim Kholod
c7f49c200c
[Cloud Security] fix ingest pipeline for benchmark scores index (#166966)
## Summary

during the [ILM fix](https://github.com/elastic/kibana/pull/165317) for
serverless the default pipeline for the scores index was also changed by
mistake. Reverting this change


### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
2023-09-25 09:46:51 +02:00
Ido Cohen
48d293f29a
[Cloud Security] Fix wrong fleet url in cloudFormation (#166130)
solves:
- https://github.com/elastic/security-team/issues/7482

choose the correct Fleet URL in case there is more than one fleet
server.

---------

Co-authored-by: Maxim Kholod <maxim.kholod@elastic.co>
2023-09-25 09:44:55 +02:00
Gerard Soldevila
b98b6d08cd
Fix flakiness on custom time range saved searches (#165454)
A bunch of tests on dashboards are customising some of the panels
settings and providing custom time ranges:

<img width="409" alt="image"
src="c869c1a3-f7db-4ccd-ad00-c5403f2b4201">

Currently, the logic is not waiting for the quick toggle animation to
complete, before proceeding to select a time range.
This can cause a flaky behavior if the logic tries to customize the
range before the button is actually available, as seen on [this failed
test](018a4c46-0e7a-4b69-9a3d-9c54c27165b0_4fcbc47e71644919129e320eea8bb3bc.html?response-content-type=text%2Fhtml&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LZWZ5UB5F%2F20230901%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230901T094837Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEB0aCXVzLWVhc3QtMSJGMEQCIGCyKcVLGPUawZubNzZdt5oZNb5v0saiIuPqXwI7rmwlAiAsOj%2Fiep94v%2BYZJtLY3Gw0m%2FmK5mJw2IcIBdNKFXgK%2BCr6Awjm%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAAaDDAzMjM3OTcwNTMwMyIMXOd1Hm6ks%2FNE37V0Ks4DgMUso7syv87hnPcC%2BB1soxvFFnj4JnNZc6ZgkLUe93z99iPFBUsqH%2BRbUTfSbjVOEJYBKGYuvp32xvSWsYNVPXKmcej18LC0yNi%2BBzoG2X%2Bj80g%2BbGMm6YfTncjPhOE0CHHqOWXts9nQ8WpDy8XOl0zfMtuiPjzOXHo9lvw2mgYDZIJIMV72FYB9JGg8FPbLQtD3rysLGNE0VDKgl5LCnYwhY1pwRCRHnVW41QfV0pwK%2FbjNf9HjdK31LQvMY%2FGPuB3M6O2CUZLsvLGfWBeGYHtkqb0hrL9ijO1Uo28ZSS1FytPftEdF0e1kAC9C5zD56HtYm55aktOWtaaC0XPWLdWWGUq%2FKQzhxSCiXK6ovATU3zI3yPNoZs92YBYmIPMOpEI40dCCpksjPwAMCiQd%2F9gMNKP5Qp5CbYd2Khy%2FeXaT8J7HOZCueN63O0j%2FtX1tbwfznhbr74lAcRQjueRYmwboZaGSDZUQ33lSSmyZk1V9WF9eJyt88oHvIx0q9bIjvOlW05DiNKfEFWYwfBywdGuvRU6eGMs1QcDNu33Lb%2BhymudM2JZmQKIjZOcb2l3Fzctp614owH4JcRlmF4%2BIa4xHeBdRlTMysS8bTIsgMK7axacGOqYBzIpC1wgZWJ1kZ0agLWCNaMIdUl%2B4xrr7w%2Fz0843WWMhRrvbJhDTHqk5UclF%2FSROAMe0FH2XEXiQ65ILyUPlrUMels5tfQ3Pp%2FJWPi9NsQJUQ1n9uLN%2BFPDOoMo8Uxg4%2FkG2O7yTkrIdArfA6pWN9I21gFMW%2BFZy9BMYltt5T65ZKOyYAIFGpLhgfBySIBCUMgwR1kusfDhf1%2FRTvtDKD2sJKN5a0IA%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=35fabe908aa7514e4a92de0ed12973af85ccfb439984fc3bdd7ef3bb8fe3419b).
(part of this [failed CI
build](https://buildkite.com/elastic/kibana-pull-request/builds/155285#018a4c46-0e7a-4b69-9a3d-9c54c27165b0))

The goal of this PR is to add a small waiting period, to make sure the
toggle animation has completed, and that the time range controls are
visible and clickable.

I used the opportunity to cleanup some "await delay millis" calls,
reusing existing logic instead.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
2023-09-25 09:32:57 +02:00
Kibana Machine
79e1508e7c skip failing test suite (#167094) 2023-09-22 19:52:26 -04:00
Kibana Machine
ae90171637 skip failing test suite (#167090) 2023-09-22 17:06:40 -04:00
Lola
1f9f572219
[Cloud Posture]remove cnvm from benchmarks (#162337)
## Summary

Remove CNVM  package policies from  the benchmarks rules page 
[Quick Win]()
<img width="1380" alt="Screen Shot 2023-07-20 at 11 17 04 AM"
src="e8d2d161-3c77-4fdb-ae0a-dcf7512e5a10">
2023-09-22 13:08:53 -07:00
Kibana Machine
180a823b7c skip failing test suite (#167081) 2023-09-22 15:53:37 -04:00
Maxim Palenov
5c7b57c88a
[Security Solution] Refactor Cypress tests navigation utilities (#166201)
**Relates to:** https://github.com/elastic/kibana/issues/153645

## Summary

This PR makes refactoring of navigation utilities and constants to make it transparent and be able to specify correct ownership later on.

## Motivation

Engineers need to add new tests and maintain the older ones. Base building blocks of this are constants and utility functions in particular URL constants and `visit()` and `visitWithoutDataRange()` functions. It turned out a simple `visit()` helper function also passes `timerange` as a query param while only some tests need it. Having it as a base utility function (as the name suggests) it also gets used in the new code while `visitWithoutDateRange()` should be preferred. On top of that URL constants are combined in one file without clear ownership and contain a mix of legacy and new urls with some parts looking outdated as navigating to the url causes redirecting to a specific page.

Having only relevant URL constants in a common file will help to reduce confusion. As the next step constants should be split into files/folders with clear ownership.

Also having `visit()` adding no extra params (besides common for ALL the tests) will make the intention clear. Whenever a time range is needed `visitWithTimeRange()` can be used (ideally accepting a time range defined in a test). And the same stays true for any other pages, e.g. rule details page can have a utility `visitRuleDetailsPage(id: string)` containing some general waiting logic so the following actions operate on a loaded page.

## Details

As a step towards clearness and transparent ownership this PR performs refactoring of `x-pack/test/security_solution_cypress/cypress/urls/navigation.ts` and `x-pack/test/security_solution_cypress/cypress/tasks/login.ts` files. The following has been done

- all url constants in `x-pack/test/security_solution_cypress/cypress/urls/navigation.ts` were checked and updated to remove duplications, avoid redirections and grouped
- legacy urls were moved to the only one test using them to test compatibility (`x-pack/test/security_solution_cypress/cypress/e2e/urls/compatibility.cy.ts`)
- `visit()` was renamed to `visitWithTimeRange()`
- `visitWithoutDateRange()` was renamed to `visit()`
- `visit()` was refactored to accept a query string

## Next steps

It's expected teams decompose `x-pack/test/security_solution_cypress/cypress/urls/navigation.ts` into feature specific file(s)/folder(s) with assigned owners.

There is no 100% chance a generic wait for a page to be loaded helper function meet requirements for each page. It makes sense to consider adding per feature `visitFeatureAPage()` helper function containing assertions for the page to be loaded. `visitRuleDetailsPage(id: string)` was added for this purpose while waiting for page to be loaded functionality is omitted in this PR to reduce a number of changes.
2023-09-22 12:01:00 -07:00
Kibana Machine
13b37aaa47 skip failing test suite (#167076) 2023-09-22 13:51:15 -04:00
Kibana Machine
d16abe3e44 skip failing test suite (#167073) 2023-09-22 12:42:10 -04:00
Gloria Hornero
1f74f23940
[Security Solution] Preparing Cypress to be able to execute on a real serverless environment (#166905)
Co-authored-by: YulNaumenko <jo.naumenko@gmail.com>
Co-authored-by: Sergi Massaneda <sergi.massaneda@gmail.com>
2023-09-22 18:26:21 +02:00
GitStart
101bd8d835
[Search Profiler] Migrate all usages of EuiPage*_Deprecated (#163131) 2023-09-22 11:55:34 -04:00
Yulia Čech
213ef5686b
[Index Management] Fix encoding issue on index details page (#166882)
## Summary
Fixes https://github.com/elastic/kibana/issues/166100

This PR adds a workaround fix for the new index details page when
opening for index names with special characters, for example
`test_index%`. Because of how encoding/decoding works, we can't use the
index name as a part of the url like `/indices/${indexName}` (see for
more details). Instead we have to pass the index name in a query
parameter like `/indices/index_details?indexName=${indexName}. The
downside of this workaround is that the url semantics doesn't reflect
that the index name is mandatory for the page to work. Once
https://github.com/elastic/kibana/issues/132600 is resolved, we should
revert this workaround and use the index name as a url segment again.

Note for reviewers: The jest tests for this fix are part of
https://github.com/elastic/kibana/pull/165705

### How to test
1. Add `xpack.index_management.dev.enableIndexDetailsPage: true` to the
file `config/kibana.dev.yml` to enable the new index details page
2. Navigate to Index Management and use the "create index" button 
3. Type a name with special characters, for example `test%`
4. Click the new index name in the list and check that the details page
and all tabs work
5. Also reload the page completely and check that the page still loads
correctly

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
2023-09-22 08:19:02 -07:00
Walter Rafelsberger
cdaa3d3851
[ML] Transforms: Enable SLO transforms installer on CI (#165470)
## Summary

This enables the installer for SLO transforms on CI.

- For the `many_fields_transform` performance journey, it will then look
for the "Create transform" on top of the table instead of the "Create
your first transform" button. Command to test the performance journey:
`node scripts/run_performance.js --journey-path
x-pack/performance/journeys/many_fields_transform.ts --skip-warmup`
- For the functional tests that assert the empty transform list, this
adds a command to delete all transforms before running those tests.
2023-09-22 17:13:38 +02:00
Maxim Palenov
f52ca02a3d
[Security Solution] Unskip related integrations Cypress tests in ESS (#166659)
**Related to: https://github.com/elastic/kibana/issues/165359**
**Resolves: https://github.com/elastic/kibana/issues/165504**

## Summary

This PR unskips skipped rule integrations ESS tests.

## Details

Besides just unskipping ESS tests this PR simplifies the test by fetching alert data from the ES directly instead of using multiple steps in UI to validate an expected field has an expected value.

## Flaky test runner

ESS [related_integrations.cy.ts 100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3199) 🟢
2023-09-22 06:42:41 -07:00
Alex Szabo
e2a7157071
[RAM] Correct renamed function after a race-condition merge (#167026)
## Summary
There was an accidental race-condition on a variable re-name and usage
between https://github.com/elastic/kibana/pull/166032 &
https://github.com/elastic/kibana/pull/166603.

This PR intends to correct that.
2023-09-22 14:21:01 +02:00
Yngrid Coello
2091870c5d
[Logs onboarding] Getting elastic-agent version from fleet (#166920)
Closes https://github.com/elastic/kibana/issues/165657.

In https://github.com/elastic/kibana/pull/166811 we exposed the value of
latest agent version available, with this PR we are aiming to use that
value as the `elastic-agent` version used in the installation script of
onboarding flow.

### How to test
1. Enter [System logs
onboarding](https://yngrdyn-deploy-kiban-pr166920.kb.us-west2.gcp.elastic-cloud.com/app/observabilityOnboarding/systemLogs)
2. Verify the elastic agent version in the installation script
<img width="870" alt="image"
src="c2d99244-6e5a-4d2e-a3fd-eea6041636d2">

3. Go to the
[console](https://yngrdyn-deploy-kiban-pr166920.kb.us-west2.gcp.elastic-cloud.com/app/dev_tools#/console)
4. Execute `GET /`
5. Verify Kibana version

As you can see the kibana version is an snapshot but the elastic agent
version proposed is the latest one released `8.10.1` allowing us to
construct a valid download url like
https://artifacts.elastic.co/downloads/beats/elastic-agent/8.10.1-linux-x86_64.tar.gz
2023-09-22 03:42:19 -07:00
Shahzad
f5ab4979a9
[Synthetics] Fix confusing labels for status alert toggle (#166918)
## Summary

 Fix confusing labels for status alert toggle

removed the Disabled wording from toggle 

<img width="1473" alt="image"
src="811cea3f-b3bd-4e65-bd48-317d259c8696">
2023-09-22 12:24:39 +02:00