# Backport
This will backport the following commits from `main` to `8.6`:
- [[Profiling] Add color for PHP JIT frames
(#147495)](https://github.com/elastic/kibana/pull/147495)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Florian
Lehner","email":"florianl@users.noreply.github.com"},"sourceCommit":{"committedDate":"2022-12-15T06:35:21Z","message":"[Profiling]
Add color for PHP JIT frames (#147495)\n\nSigned-off-by: Florian Lehner
<florian.lehner@elastic.cloud>\r\n\r\n## Summary\r\n\r\nAdd a color for
the PHP JIT frame type.\r\n\r\nSigned-off-by: Florian Lehner
<florian.lehner@elastic.cloud>","sha":"e7f858bdbe5539163d7e95ce6f41d973c841756e","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.7.0","v8.6.1"],"number":147495,"url":"https://github.com/elastic/kibana/pull/147495","mergeCommit":{"message":"[Profiling]
Add color for PHP JIT frames (#147495)\n\nSigned-off-by: Florian Lehner
<florian.lehner@elastic.cloud>\r\n\r\n## Summary\r\n\r\nAdd a color for
the PHP JIT frame type.\r\n\r\nSigned-off-by: Florian Lehner
<florian.lehner@elastic.cloud>","sha":"e7f858bdbe5539163d7e95ce6f41d973c841756e"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/147495","number":147495,"mergeCommit":{"message":"[Profiling]
Add color for PHP JIT frames (#147495)\n\nSigned-off-by: Florian Lehner
<florian.lehner@elastic.cloud>\r\n\r\n## Summary\r\n\r\nAdd a color for
the PHP JIT frame type.\r\n\r\nSigned-off-by: Florian Lehner
<florian.lehner@elastic.cloud>","sha":"e7f858bdbe5539163d7e95ce6f41d973c841756e"}},{"branch":"8.6","label":"v8.6.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>
# Backport
This will backport the following commits from `main` to `8.6`:
- [[Profiling] Remove link to 'Other' bucket
(#147523)](https://github.com/elastic/kibana/pull/147523)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Dario
Gieselaar","email":"dario.gieselaar@elastic.co"},"sourceCommit":{"committedDate":"2022-12-14T15:57:32Z","message":"[Profiling]
Remove link to 'Other' bucket (#147523)\n\nCloses
https://github.com/elastic/prodfiler/issues/2739","sha":"f2fcb1c5afc19f8439dfff4c4137e1b2253478dc","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v8.7.0","v8.6.1"],"number":147523,"url":"https://github.com/elastic/kibana/pull/147523","mergeCommit":{"message":"[Profiling]
Remove link to 'Other' bucket (#147523)\n\nCloses
https://github.com/elastic/prodfiler/issues/2739","sha":"f2fcb1c5afc19f8439dfff4c4137e1b2253478dc"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/147523","number":147523,"mergeCommit":{"message":"[Profiling]
Remove link to 'Other' bucket (#147523)\n\nCloses
https://github.com/elastic/prodfiler/issues/2739","sha":"f2fcb1c5afc19f8439dfff4c4137e1b2253478dc"}},{"branch":"8.6","label":"v8.6.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
# Backport
This will backport the following commits from `main` to `8.6`:
- [[profiling] Ensure constant colors in profiling bar chart
(#146696)](https://github.com/elastic/kibana/pull/146696)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Thomas
Dullien","email":"thomas.dullien@googlemail.com"},"sourceCommit":{"committedDate":"2022-12-01T11:11:49Z","message":"[profiling]
Ensure constant colors in profiling bar chart (#146696)\n\n##
Summary\r\n\r\nMake sure that identical objects in the bar chart always
get assigned\r\nthe same colors.\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Tim
Rühsen
<tim.ruhsen@elastic.co>","sha":"0a065f3c1d18b881209f510c574c7f463c401ffc","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.7.0","v8.6.1"],"number":146696,"url":"https://github.com/elastic/kibana/pull/146696","mergeCommit":{"message":"[profiling]
Ensure constant colors in profiling bar chart (#146696)\n\n##
Summary\r\n\r\nMake sure that identical objects in the bar chart always
get assigned\r\nthe same colors.\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Tim
Rühsen
<tim.ruhsen@elastic.co>","sha":"0a065f3c1d18b881209f510c574c7f463c401ffc"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/146696","number":146696,"mergeCommit":{"message":"[profiling]
Ensure constant colors in profiling bar chart (#146696)\n\n##
Summary\r\n\r\nMake sure that identical objects in the bar chart always
get assigned\r\nthe same colors.\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>\r\nCo-authored-by: Tim
Rühsen
<tim.ruhsen@elastic.co>","sha":"0a065f3c1d18b881209f510c574c7f463c401ffc"}},{"branch":"8.6","label":"v8.6.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Co-authored-by: Thomas Dullien <thomas.dullien@googlemail.com>
Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co>
# Backport
This will backport the following commits from `main` to `8.6`:
- [[Profiling] Add APIs to clear frame and executable caches
(#145499)](https://github.com/elastic/kibana/pull/145499)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Joseph
Crail","email":"joseph.crail@elastic.co"},"sourceCommit":{"committedDate":"2022-11-17T10:04:58Z","message":"[Profiling]
Add APIs to clear frame and executable caches (#145499)\n\nThis PR adds
two endpoints to clear each respective cache:\r\n\r\n* `DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/executables`\r\n* `DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/stackframes`\r\n\r\nRelated to
https://github.com/elastic/prodfiler/issues/2759\r\n\r\n#### Design
choices\r\n\r\n1. The `DELETE` method was chosen instead of `PUT` or
`POST` since the\r\ngiven semantics of `DELETE` matches the expected
behavior for the\r\nrelated issue.\r\n2. Each endpoint will remove all
items from the respective cache. A\r\nseparate API for each cache allows
us to selectively clear the necessary\r\ncache without the downsides of
a catch-all endpoint to clear all caches.\r\nThis gives us the
flexibility to add more endpoints if needed. Given the\r\ntradeoff
between complexity now or later, it was decided to implement\r\ngeneral
invalidation now with the option to invalidate specific
items\r\nlater.\r\n3. The RESTful design allows us to clear specific
items later (e.g.\r\n`DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/executables/{ID}`\r\ncould
clear only executable `ID` from the cache).\r\n4. Each endpoint returns
an empty payload on success. However, the\r\nKibana logs reflect the
actions taken and how many cache items were\r\naffected.\r\n5. The
stacktrace cache was ignored since it was not affected by
symbols\r\nwritten to Elasticsearch.\r\n6. Each endpoint is not directly
accessible from UI since it is expected\r\nthat the endpoints will be
called outside of Kibana. However, the\r\nendpoints can be called
manually from the browser's
console.","sha":"642ab74cfc1a4ef5c0a2dfdadf00bd3792f780a7","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.6.0","v8.7.0","v8.5.2"],"number":145499,"url":"https://github.com/elastic/kibana/pull/145499","mergeCommit":{"message":"[Profiling]
Add APIs to clear frame and executable caches (#145499)\n\nThis PR adds
two endpoints to clear each respective cache:\r\n\r\n* `DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/executables`\r\n* `DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/stackframes`\r\n\r\nRelated to
https://github.com/elastic/prodfiler/issues/2759\r\n\r\n#### Design
choices\r\n\r\n1. The `DELETE` method was chosen instead of `PUT` or
`POST` since the\r\ngiven semantics of `DELETE` matches the expected
behavior for the\r\nrelated issue.\r\n2. Each endpoint will remove all
items from the respective cache. A\r\nseparate API for each cache allows
us to selectively clear the necessary\r\ncache without the downsides of
a catch-all endpoint to clear all caches.\r\nThis gives us the
flexibility to add more endpoints if needed. Given the\r\ntradeoff
between complexity now or later, it was decided to implement\r\ngeneral
invalidation now with the option to invalidate specific
items\r\nlater.\r\n3. The RESTful design allows us to clear specific
items later (e.g.\r\n`DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/executables/{ID}`\r\ncould
clear only executable `ID` from the cache).\r\n4. Each endpoint returns
an empty payload on success. However, the\r\nKibana logs reflect the
actions taken and how many cache items were\r\naffected.\r\n5. The
stacktrace cache was ignored since it was not affected by
symbols\r\nwritten to Elasticsearch.\r\n6. Each endpoint is not directly
accessible from UI since it is expected\r\nthat the endpoints will be
called outside of Kibana. However, the\r\nendpoints can be called
manually from the browser's
console.","sha":"642ab74cfc1a4ef5c0a2dfdadf00bd3792f780a7"}},"sourceBranch":"main","suggestedTargetBranches":["8.6","8.5"],"targetPullRequestStates":[{"branch":"8.6","label":"v8.6.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/145499","number":145499,"mergeCommit":{"message":"[Profiling]
Add APIs to clear frame and executable caches (#145499)\n\nThis PR adds
two endpoints to clear each respective cache:\r\n\r\n* `DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/executables`\r\n* `DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/stackframes`\r\n\r\nRelated to
https://github.com/elastic/prodfiler/issues/2759\r\n\r\n#### Design
choices\r\n\r\n1. The `DELETE` method was chosen instead of `PUT` or
`POST` since the\r\ngiven semantics of `DELETE` matches the expected
behavior for the\r\nrelated issue.\r\n2. Each endpoint will remove all
items from the respective cache. A\r\nseparate API for each cache allows
us to selectively clear the necessary\r\ncache without the downsides of
a catch-all endpoint to clear all caches.\r\nThis gives us the
flexibility to add more endpoints if needed. Given the\r\ntradeoff
between complexity now or later, it was decided to implement\r\ngeneral
invalidation now with the option to invalidate specific
items\r\nlater.\r\n3. The RESTful design allows us to clear specific
items later (e.g.\r\n`DELETE
{BASE_KIBANA_PATH}/api/profiling/v1/cache/executables/{ID}`\r\ncould
clear only executable `ID` from the cache).\r\n4. Each endpoint returns
an empty payload on success. However, the\r\nKibana logs reflect the
actions taken and how many cache items were\r\naffected.\r\n5. The
stacktrace cache was ignored since it was not affected by
symbols\r\nwritten to Elasticsearch.\r\n6. Each endpoint is not directly
accessible from UI since it is expected\r\nthat the endpoints will be
called outside of Kibana. However, the\r\nendpoints can be called
manually from the browser's
console.","sha":"642ab74cfc1a4ef5c0a2dfdadf00bd3792f780a7"}},{"branch":"8.5","label":"v8.5.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Co-authored-by: Joseph Crail <joseph.crail@elastic.co>
## Summary
We are removing @timestamp from the backend soon.
And we do not currently use @timestamp in Kibana.
This PR just cleans up the unused field definitions for the Timestamps.
We store stack frame types with a run-length encoding in Elasticsearch
and decode the raw data in Kibana. The current implementation was
working fine when the frame type did not change often but failed for
long sequences of unique frame types. With this commit we make sure that
these longer sequences can also properly decoded.
Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co>
Improved the visualization of data in profiling by showing more details
and using more nicely formatted numbers.
Fixes https://github.com/elastic/prodfiler/issues/2772
Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co>
* Derive address and file ID from base64 encoding
This skips the intermediate deserialization step to a buffer object.
* Move run-length encoding methods
* Decode run-length directly from base64 encoding
This skips the intermediate deserialization step to a buffer object.
* Minor refactor
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Remove total and sampled traces from API
* Remove Samples array from flamegraph API
These values are redundant with CountInclusive so could be removed
without issue.
* Remove totalCount and eventsIndex
These values are no longer needed.
* Remove samples from callee tree
* Refactor columnar view model into separate file
* Add more lazy-loaded flamegraph calculations
* Fix spacing in frame label
* Remove frame information API
* Improve test coverage
* Fix type error
* Replace fnv-plus with custom 64-bit FNV1-a
* Add exceptions for linting errors
* Add workaround for frame type truncation bug
* Replace prior workaround for truncation bug
This fix supercedes the prior workaround and addresses the truncation at
its source.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Replace non-null assertion with nullish coalescing
* Remove createFrameGroup
* Remove callers
* Use adjacency list representation for tree
* Move frame type map outside function
* Inline frame group name
* Replace FrameGroupID with ID
* Create columnar view model in client
* Add instrumentation for flamegraph
I inlined the construction of the flamegraph into the respective route
so that we could add fine-grained instrumentation. We now use APM and
console logging to understand how long flamegraph construction takes.
* Remove unnecessary Set usage
* Remove superfluous clone
This was likely added when we needed to avoid infinite recursion when
serializing to JSON. This no longer has a useful function.
* Pass in pre-calculated frame group info
I noticed that we were creating frame group info multiple times so I
added it as a parameter for the intermediate node.
* Sort callees in one place
Callees should be sorted first by samples decreasing and then by frame
groups. Combining the two sorts makes the post-processing clearer to
future readers and/or maintainers.
* Capitalize fields in preparation of merging
* Align both node data structures
* Pass metadata instead of copying fields
* Refactor frame label method
* Use pre-calculated array length
* Use pre-allocated array
* Refactor intermediate node
* Remove intermediate node structure
* Move if statement out of for loop
* Fix comments
* Sort sibling nodes by frame group ID
* Calculate graph size during creation
* Add missing groupStackFrameMetadataByStackTrace
* Fix formatting
* Fix generated callee source
* Fix creation of frame group
* Fix test
* Remove filter for relevant traces
* Stop passing frame group
* Create root node inside createCallerCalleeGraph
* Fix timestamps
* Remove frame group comparator
* Add instrumentation for topN functions
* Allow for missing stacktraces
* Use Date.now instead
