Extend default log pattern on server-side to include error information (#219940)

## Release Notes
Kibana logging's pattern layout, used by default for the console
appender, will now use a new default pattern layout
`[%date][%level][%logger] %message %error`. This will include the error
name and stack trace if these were included in the log entry. To opt out
of this behavior users can omit the `%error` placeholder from their log
pattern config in kibana.yml e.g.:
```
logging:
  appenders:
    console:
      type: console
      layout:
        type: pattern
        pattern: "[%date][%level][%logger] %message"
```

## Summary

Previously, when we pass the error in meta, the information related to
stacktrace and error message was not available in console. This PR
changed the default pattern to also include error information if it is
provided in meta (similar to the way that the logging happens when error
is directly passed to logger.error).

New pattern: (added `%error` at the end)
```
[%date][%level][%logger] %message %error
```

Here you can see the difference:

Logger:

```
server.logger.error(
        `Unable to create Synthetics monitor ${monitorWithNamespace[ConfigKey.NAME]}`,
        { error: e }
      );
```

#### Before


![image](https://github.com/user-attachments/assets/4f3ff751-84d5-4b5b-b6a9-d49f868a9606)

#### After


![image](https://github.com/user-attachments/assets/e22b8e45-1b0a-4d8c-b51d-5dfb3938da4f)


### Alternative
We could also change the MetaConversion and include this information,
but we might have additional meta information which I am not sure if it
is OK to be logged by default. Let me know if you prefer changing
MetaConversion instead of adding a new error conversion.

<details>
<summary>Code changes for MetaConversion</summary>

```
function isError(x: any): x is Error {
  return x instanceof Error;
}

export const MetaConversion: Conversion = {
  pattern: /%meta/g,
  convert(record: LogRecord) {
    if (!record.meta) {
      return '';
    }
    const { error, ...rest } = record.meta;
    const metaString = Object.keys(rest).length !== 0 ? JSON.stringify(rest) : '';
    let errorString = '';

    if (isError(record.meta?.error)) {
      errorString = record.meta?.error.stack || '';
    }

    return [metaString, errorString].filter(Boolean).join(' ');
  },
};
```
</details>

Here is how adjusting meta will look like in this case:


![image](https://github.com/user-attachments/assets/d7dce9bc-7147-472d-b434-373322f41bbf)

docs/reference/configuration-reference/logging-settings.md

@@ -29,7 +29,7 @@ The following table serves as a quick reference for different logging configurat
 |     |     |
 | --- | --- |
 | `logging.appenders[].<appender-name>` | Unique appender identifier. |
-| `logging.appenders[].console:` | Appender to use for logging records to **stdout**. By default, uses the `[%date][%level][%logger] %message` **pattern*** layout. To use a ***json**, set the [layout type to `json`](docs-content://deploy-manage/monitor/logging-configuration/kibana-log-settings-examples.md#log-in-json-ecs-example). |
+| `logging.appenders[].console:` | Appender to use for logging records to **stdout**. By default, uses the `[%date][%level][%logger] %message %error` **pattern*** layout. To use a ***json**, set the [layout type to `json`](docs-content://deploy-manage/monitor/logging-configuration/kibana-log-settings-examples.md#log-in-json-ecs-example). |
 | `logging.appenders[].file:` | Allows you to specify a fileName to write log records to disk. To write [all log records to file](docs-content://deploy-manage/monitor/logging-configuration/kibana-log-settings-examples.md#log-to-file-example), add the file appender to `root.appenders`. If configured, you also need to specify [`logging.appenders.file.pathName`](docs-content://deploy-manage/monitor/logging-configuration/kibana-log-settings-examples.md#log-to-file-example). |
 | `logging.appenders[].rolling-file:` | Similar to [Log4j’s](https://logging.apache.org/log4j/2.x/) `RollingFileAppender`, this appender will log to a file and rotate if following a rolling strategy when the configured policy triggers. There are currently two policies supported: [`size-limit`](docs-content://deploy-manage/monitor/logging-configuration/kibana-logging.md#size-limit-triggering-policy) and [`time-interval`](docs-content://deploy-manage/monitor/logging-configuration/kibana-logging.md#time-interval-triggering-policy). |
 | `logging.appenders[].<appender-name>.type` | The appender type determines where the log messages are sent. Options are `console`, `file`, `rewrite`, `rolling-file`. Required. |

src/core/packages/logging/browser-internal/src/layouts/__snapshots__/pattern_layout.test.ts.snap

@@ -12,7 +12,7 @@ exports[`\`format()\` correctly formats record with custom pattern. 5`] = `"mock
 
 exports[`\`format()\` correctly formats record with custom pattern. 6`] = `"mock-message-6-context-6-message-6"`;
 
-exports[`\`format()\` correctly formats record with full pattern. 1`] = `"[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack"`;
+exports[`\`format()\` correctly formats record with full pattern. 1`] = `"[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack Meta error stack"`;
 
 exports[`\`format()\` correctly formats record with full pattern. 2`] = `"[2012-02-01T09:30:22.011-05:00][ERROR][context-2] message-2"`;
 

src/core/packages/logging/browser-internal/src/layouts/pattern_layout.test.ts

@@ -23,6 +23,8 @@ const stripAnsiSnapshotSerializer: jest.SnapshotSerializerPlugin = {
 };
 
 const timestamp = new Date(Date.UTC(2012, 1, 1, 14, 30, 22, 11));
+const error = new Error('Meta error');
+error.stack = 'Meta error stack';
 const records: LogRecord[] = [
   {
     context: 'context-1',
@@ -31,6 +33,9 @@ const records: LogRecord[] = [
       name: 'Some error name',
       stack: 'Some error stack',
     },
+    meta: {
+      error,
+    },
     level: LogLevel.Fatal,
     message: 'message-1',
     timestamp,

src/core/packages/logging/browser-internal/src/layouts/pattern_layout.ts

@@ -15,16 +15,16 @@ import {
   MetaConversion,
   MessageConversion,
   DateConversion,
+  ErrorConversion,
 } from '@kbn/core-logging-common-internal';
 
-const DEFAULT_PATTERN = `[%date][%level][%logger] %message`;
-
 const conversions: Conversion[] = [
   LoggerConversion,
   MessageConversion,
   LevelConversion,
   MetaConversion,
   DateConversion,
+  ErrorConversion,
 ];
 
 /**
@@ -33,7 +33,7 @@ const conversions: Conversion[] = [
  * @internal
  */
 export class PatternLayout extends BasePatternLayout {
-  constructor(pattern: string = DEFAULT_PATTERN) {
+  constructor(pattern?: string) {
     super({
       pattern,
       highlight: false,

src/core/packages/logging/common-internal/index.ts

@@ -14,6 +14,7 @@ export {
   MessageConversion,
   LevelConversion,
   MetaConversion,
+  ErrorConversion,
   type Conversion,
   AbstractLogger,
   type CreateLogRecordFn,

src/core/packages/logging/common-internal/src/index.ts

@@ -14,6 +14,7 @@ export {
   MessageConversion,
   LevelConversion,
   MetaConversion,
+  ErrorConversion,
   type Conversion,
 } from './layouts';
 export { AbstractLogger, type CreateLogRecordFn } from './logger';

src/core/packages/logging/common-internal/src/layouts/__snapshots__/pattern_layout.test.ts.snap

@@ -12,7 +12,7 @@ exports[`\`format()\` correctly formats record with custom pattern. 5`] = `"mock
 
 exports[`\`format()\` correctly formats record with custom pattern. 6`] = `"mock-message-6-context-6-message-6"`;
 
-exports[`\`format()\` correctly formats record with full pattern. 1`] = `"[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack"`;
+exports[`\`format()\` correctly formats record with full pattern. 1`] = `"[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack Meta error stack"`;
 
 exports[`\`format()\` correctly formats record with full pattern. 2`] = `"[2012-02-01T09:30:22.011-05:00][ERROR][context-2] message-2"`;
 

src/core/packages/logging/common-internal/src/layouts/conversions/error.ts

@@ -0,0 +1,27 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+// import { EcsError } from '@elastic/ecs';
+import { LogRecord } from '@kbn/logging';
+import { Conversion } from './types';
+
+function isError(x: any): x is Error {
+  return x instanceof Error;
+}
+
+export const ErrorConversion: Conversion = {
+  pattern: /%error/g,
+  convert(record: LogRecord) {
+    let error;
+    if (isError(record.meta?.error)) {
+      error = record.meta?.error.stack;
+    }
+    return error ? `${error}` : '';
+  },
+};

src/core/packages/logging/common-internal/src/layouts/conversions/index.ts

@@ -13,3 +13,4 @@ export { LevelConversion } from './level';
 export { MessageConversion } from './message';
 export { MetaConversion } from './meta';
 export { DateConversion } from './date';
+export { ErrorConversion } from './error';

src/core/packages/logging/common-internal/src/layouts/index.ts

@@ -14,5 +14,6 @@ export {
   MessageConversion,
   LevelConversion,
   MetaConversion,
+  ErrorConversion,
   type Conversion,
 } from './conversions';

src/core/packages/logging/common-internal/src/layouts/pattern_layout.test.ts

@@ -23,6 +23,8 @@ const stripAnsiSnapshotSerializer: jest.SnapshotSerializerPlugin = {
 };
 
 const timestamp = new Date(Date.UTC(2012, 1, 1, 14, 30, 22, 11));
+const error = new Error('Meta error');
+error.stack = 'Meta error stack';
 const records: LogRecord[] = [
   {
     context: 'context-1',
@@ -31,6 +33,9 @@ const records: LogRecord[] = [
       name: 'Some error name',
       stack: 'Some error stack',
     },
+    meta: {
+      error,
+    },
     level: LogLevel.Fatal,
     message: 'message-1',
     timestamp,

src/core/packages/logging/common-internal/src/layouts/pattern_layout.ts

@@ -15,12 +15,13 @@ import {
   MetaConversion,
   MessageConversion,
   DateConversion,
+  ErrorConversion,
 } from './conversions';
 
 /**
  * Default pattern used by PatternLayout if it's not overridden in the configuration.
  */
-const DEFAULT_PATTERN = `[%date][%level][%logger] %message`;
+const DEFAULT_PATTERN = `[%date][%level][%logger] %message %error`;
 
 const DEFAULT_CONVERSIONS: Conversion[] = [
   LoggerConversion,
@@ -28,6 +29,7 @@ const DEFAULT_CONVERSIONS: Conversion[] = [
   LevelConversion,
   MetaConversion,
   DateConversion,
+  ErrorConversion,
 ];
 
 export interface PatternLayoutOptions {
@@ -69,6 +71,6 @@ export class PatternLayout implements Layout {
       );
     }
 
-    return recordString;
+    return recordString.trim();
   }
 }

src/core/packages/logging/server-internal/src/layouts/__snapshots__/pattern_layout.test.ts.snap

@@ -12,7 +12,7 @@ exports[`\`format()\` correctly formats record with custom pattern. 5`] = `"mock
 
 exports[`\`format()\` correctly formats record with custom pattern. 6`] = `"mock-message-6-context-6-message-6"`;
 
-exports[`\`format()\` correctly formats record with full pattern. 1`] = `"[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack"`;
+exports[`\`format()\` correctly formats record with full pattern. 1`] = `"[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack Meta error stack"`;
 
 exports[`\`format()\` correctly formats record with full pattern. 2`] = `"[2012-02-01T09:30:22.011-05:00][ERROR][context-2] message-2"`;
 
@@ -24,7 +24,7 @@ exports[`\`format()\` correctly formats record with full pattern. 5`] = `"[2012-
 
 exports[`\`format()\` correctly formats record with full pattern. 6`] = `"[2012-02-01T09:30:22.011-05:00][TRACE][context-6] message-6"`;
 
-exports[`\`format()\` correctly formats record with highlighting. 1`] = `[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack`;
+exports[`\`format()\` correctly formats record with highlighting. 1`] = `[2012-02-01T09:30:22.011-05:00][FATAL][context-1] Some error stack Meta error stack`;
 
 exports[`\`format()\` correctly formats record with highlighting. 2`] = `[2012-02-01T09:30:22.011-05:00][ERROR][context-2] message-2`;
 

src/core/packages/logging/server-internal/src/layouts/conversions/index.ts

@@ -14,4 +14,5 @@ export {
   DateConversion,
   MessageConversion,
   MetaConversion,
+  ErrorConversion,
 } from '@kbn/core-logging-common-internal';

src/core/packages/logging/server-internal/src/layouts/pattern_layout.test.ts

@@ -23,6 +23,8 @@ const stripAnsiSnapshotSerializer: jest.SnapshotSerializerPlugin = {
 };
 
 const timestamp = new Date(Date.UTC(2012, 1, 1, 14, 30, 22, 11));
+const error = new Error('Meta error');
+error.stack = 'Meta error stack';
 const records: LogRecord[] = [
   {
     context: 'context-1',
@@ -31,6 +33,9 @@ const records: LogRecord[] = [
       name: 'Some error name',
       stack: 'Some error stack',
     },
+    meta: {
+      error,
+    },
     level: LogLevel.Fatal,
     message: 'message-1',
     timestamp,

src/core/packages/logging/server-internal/src/layouts/pattern_layout.ts

@@ -19,10 +19,9 @@ import {
   MessageConversion,
   PidConversion,
   DateConversion,
+  ErrorConversion,
 } from './conversions';
 
-const DEFAULT_PATTERN = `[%date][%level][%logger] %message`;
-
 export const patternSchema = schema.string({
   maxLength: 1000,
   validate: (string) => {
@@ -43,6 +42,7 @@ const conversions: Conversion[] = [
   MetaConversion,
   PidConversion,
   DateConversion,
+  ErrorConversion,
 ];
 
 /**
@@ -53,7 +53,7 @@ const conversions: Conversion[] = [
 export class PatternLayout extends BasePatternLayout {
   public static configSchema = patternLayoutSchema;
 
-  constructor(pattern: string = DEFAULT_PATTERN, highlight: boolean = false) {
+  constructor(pattern?: string, highlight: boolean = false) {
     super({
       pattern,
       highlight,

src/core/server/docs/kib_core_logging.mdx

@@ -129,7 +129,7 @@ There are two types of layout supported at the moment: `pattern` and `json`.
 ### Pattern layout
 With `pattern` layout it's possible to define a string pattern with special placeholders `%conversion_pattern` (see the table below) that
 will be replaced with data from the actual log message. By default the following pattern is used:
-`[%date][%level][%logger] %message`. Also `highlight` option can be enabled for `pattern` layout so that
+`[%date][%level][%logger] %message %error`. Also `highlight` option can be enabled for `pattern` layout so that
 some parts of the log message are highlighted with different colors that may be quite handy if log messages are forwarded
 to the terminal with color support.
 `pattern` layout uses a sub-set of [log4j2 pattern syntax](https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout)