As preparation for #106081, this PR adds the `size_in_bytes`
field to the enrich cache. This field is calculated by summing
the ByteReference sizes of all the search hits in the cache.
It's not a perfect representation of the size of the enrich cache
on the heap, but some experimentation showed that it's quite close.
* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
The GeoIP endpoint does not use the xpack http client. The GeoIP downloader uses the JDKs builtin cacerts.
If customer is using custom https endpoint they need to provide the cacert in the jdk, whether our jdk bundled in or their jdk. Otherwise they will see something like
```
...PKiX path building failed: sun.security.provier.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target...
```
* [DOCS] Categorize ingest processors on overview page, summarize use cases
* Add overview info, subheading, links
* Apply suggestions from review
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Insert space
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* added override flag for rename processer along with factory tests
* added yaml tests for rename processor using the override flag
* updated renameProcessor tests to include override flag as a parameter
* updated rename processor tests to incorporate override flag = true scenario
* updated rename processor asciidoc with override option
* updated rename processor asciidoc with override option
* removed unnecessary supresswarnings tag
* corrected formatting errors
* updated processor tests
* fixed yaml tests
* Prefer early throw style here
* Whitespace
* Move and rewrite this test
It's just a simple test of the primary behavior of the rename
processor, so put it first and simplify it.
* Rename this test
It doesn't actually exercise template snippets
* Tidy up this test
---------
Co-authored-by: Joe Gallo <joegallo@gmail.com>
This commit introduces a new _ingest/simulate API that runs any pipelines
on the given data that would be executed for a given index, but instead of
indexing the data into the index, returns the transformed documents and
the list of pipelines that were executed.
- Creates a new StackTemplateRegistry that uses the new names
- The new registry only respects stack.templates.enabled for index templates
- Renames the old registry to LegacyStackTemplateRegistry
- Component templates are not duplicated but registered under two different names
- Documents the new naming convention
- Index templates are not renamed, at least for now, as there are some challenges with it
See 7fd0423 for more details.
This PR extends the assumptions we make about database file availability to all database file
names instead of the default ones we host at Elastic. When creating a geo ip processor with
a database name that is not recognized we unilaterally convert the processor to one that
tags documents with a missing database message until the database file requested is
downloaded or provided via the manual configuration route. This allows a pipeline to be
created and for the download service to be started, potentially sourcing the needed files.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* When using a managed pipeline GeoIpDownloader is triggered only when an index exists for the pipeline.
* When using a managed pipeline GeoIpDownloader is triggered only when an index exists for the pipeline.
* Adding the geoip processor back
* Adding tags to the events mapping.
* Fix a forbidden API call into tests.
* lint
* Adding an integration tests for managed pipelines.
* lint
* Add a geoip_database_lazy_download param to pipelines and use it instead of managed.
* Fix a edge case: pipeline can be set after index is created.
* lint.
* Update docs/changelog/96624.yaml
* Update 96624.yaml
* Uses a processor setting (download_database_on_pipeline_creation) to decide database download strategy.
* Removing debug instruction.
* Improved documentation.
* Improved the way to check for referenced pipelines.
* Fixing an error in test.
* Improved integration tests.
* Lint.
* Fix failing tests.
* Fix failing tests (2).
* Adding javadoc.
* lint javadoc.
* Using a set instead of a list to store checked pipelines.
