tiennm99/tjahzi
1
0 Fork 0
mirror of https://github.com/tiennm99/tjahzi.git synced 2026-10-14 16:18:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #41 from tkowalcz/36-logback-appender

Browse Source 
36 logback appender
This commit is contained in:
Tomasz Kowalczewski authored and GitHub committed 2021-03-13 17:05:02 +01:00
2 parents a45da4e702 7fd1a21c14
commit 63dfff3735
45 files changed
+1922 -68
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+8 -44
View File
@@ -9,64 +9,28 @@ Latest releases:
[![Maven Central](https://img.shields.io/maven-central/v/pl.tkowalcz.tjahzi/core.svg?label=Core&style=for-the-badge)](https://search.maven.org/search?q=g:pl.tkowalcz.tjahzi)
[![Maven Central](https://img.shields.io/maven-central/v/pl.tkowalcz.tjahzi/log4j2-appender.svg?label=Log4j2%20Appender&style=for-the-badge)](https://search.maven.org/search?q=g:pl.tkowalcz.tjahzi)
[![Maven Central](https://img.shields.io/maven-central/v/pl.tkowalcz.tjahzi/logback-appender.svg?label=Logback%20Appender&style=for-the-badge)](https://search.maven.org/search?q=g:pl.tkowalcz.tjahzi)
If you find the project useful (or not useful for some reason) please let me know to encourage further development. ⭐ Stars are also welcome ⭐ ;).
## Features
Tjahzi allows pushing log data to Loki. It uses protobuf format to push log lines with timestamps and labels to Loki. This
is implemented in the `core` component. On top of that we built the `log4j2 appender`. It's features include:
is implemented in the `core` component. On top of that we built appenders for `log4j2` and `Logback`. These feature:
1. Logging does not allocate objects nor take any locks.
1. Sending data to Loki in batches allocates as little as possible.
1. We also provide a no-dependency version of log4j2 appender.
1. Includes in-house implementation of protobuf wire format for Loki to reduce dependencies and improve performance
1. We also provide a no-dependency versions of these appenders.
1. Includes in-house implementation of protobuf wire format for Loki to reduce dependencies and improve performance.
It is currently used (that I know of) to ship logs from tens of servers at about 10GB of logs per day.
Log4j2 appender is currently used to ship logs from tens of servers at about 10GB of logs per day.
## Getting started
1. Grab the no-dependency version of the appender:
```xml
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>log4j2-appender-nodep</artifactId>
<version>0.9.4</version>
</dependency>
```
If you already use a compatible version of Netty in your project then to reduce the size of your dependencies include
regular appender distribution:
```xml
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>log4j2-appender</artifactId>
<version>0.9.4</version>
</dependency>
```
1. Add `packages="pl.tkowalcz.tjahzi.log4j2"` attribute to your log4j2 configuration xml.
1. Include minimal appender configuration
For `log4j2` appender quick start guide and detailed discussions see this [README.md](log4j2-appender/README.md).
For `Logback` appender quick start guide and detailed discussions see this [README.md](logback-appender/README.md).
```xml
<Loki name="loki-appender">
<host>${sys:loki.host}</host>
<port>${sys:loki.port}</port>
<PatternLayout>
<Pattern>%X{tid} [%t] %d{MM-dd HH:mm:ss.SSS} %5p %c{1} - %m%n%exception{full}</Pattern>
</PatternLayout>
<Label name="server" value="${sys:hostname}"/>
</Loki>
```
Detailed discussion of how to configure the appender with examples and description of all possible tags check
[Log4j2 appender docs](log4j2-appender/README.md).
## Design principles
## Core design principles
Logging should be lightweight and not interfere with main business tasks of threads that happen to log a message.
Asking the logging subsystem to log a message should be as CPU efficient as possible.
log4j2-appender/README.md
+49 -2
View File
@@ -3,14 +3,61 @@
On top of a `Core` component with rather simplistic API we intend to build several layers that make it truly useful. Log4j2
appender seemed like a good first.
## Example Log4j2 configuration
## Quick start guide
1. Grab the no-dependency version of the appender:
```xml
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>log4j2-appender-nodep</artifactId>
<version>0.9.4</version>
</dependency>
```
If you already use a compatible version of Netty in your project then to reduce the size of your dependencies include
regular appender distribution:
```xml
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>log4j2-appender</artifactId>
<version>0.9.4</version>
</dependency>
```
1. Add `packages="pl.tkowalcz.tjahzi.log4j2"` attribute to your existing log4j2 configuration file.
1. Include minimal appender configuration:
```xml
<Loki name="loki-appender">
<host>${sys:loki.host}</host>
<port>${sys:loki.port}</port>
<PatternLayout>
<Pattern>%X{tid} [%t] %d{MM-dd HH:mm:ss.SSS} %5p %c{1} - %m%n%exception{full}</Pattern>
</PatternLayout>
<Label name="server" value="${sys:hostname}"/>
</Loki>
```
1. Reference the appender from inside one of your logger definitions:
```xml
<Root level="INFO">
<AppenderRef ref="Loki"/>
</Root>
```
## Advanced configuration
This example sets up a root logger with a Loki appender. Note that `pl.tkowalcz.tjahzi.log4j2` is added to `packages` attribute
of configuration so that the appender can be found.
```xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
log4j2-appender/src/main/java/pl/tkowalcz/tjahzi/log4j2/LabelFactory.java
+5 -3
View File
@@ -52,7 +52,7 @@ public class LabelFactory {
if (!duplicatedLabels.isEmpty()) {
LOGGER.error(
"There are duplicated labels which is not allowed by Loki. " +
"These labels will be deduplicated undeterministically: {}",
"These labels will be deduplicated non-deterministically: {}",
duplicatedLabels
);
}
@@ -67,7 +67,8 @@ public class LabelFactory {
return Stream.of(label);
}
LOGGER.error("Label '{}' contains invalid characters - ignoring it. {}",
LOGGER.error(
"Ignoring label '{}' - contains invalid characters. {}",
label.getName(),
GitHubDocs.LABEL_NAMING.getLogMessage()
);
@@ -85,7 +86,8 @@ public class LabelFactory {
String logLevelLabel
) {
if (!Label.hasValidName(logLevelLabel)) {
LOGGER.error("Log level label '{}' contains invalid characters - ignoring it. {}",
LOGGER.error(
"Ignoring log level label '{}' - contains invalid characters. {}",
logLevelLabel,
GitHubDocs.LABEL_NAMING.getLogMessage()
);
log4j2-appender/src/main/java/pl/tkowalcz/tjahzi/log4j2/LokiAppenderBuilder.java
+1 -1
View File
@@ -101,7 +101,7 @@ public class LokiAppenderBuilder<B extends LokiAppenderBuilder<B>> extends Abstr
);
int bufferSizeBytes = getBufferSizeMegabytes() * BYTES_IN_MEGABYTE;
if (TjahziInitializer.isCorrectSize(bufferSizeBytes)) {
if (!TjahziInitializer.isCorrectSize(bufferSizeBytes)) {
LOGGER.warn("Invalid log buffer size {} - using nearest power of two greater than provided value, no less than 1MB. {}",
bufferSizeBytes,
GitHubDocs.LOG_BUFFER_SIZING.getLogMessage()
log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/AllocationTest.java
+1 -1
View File
@@ -57,7 +57,7 @@ public class AllocationTest {
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-log4j2-configuration.xml")
.getResource("basic-appender-test-configuration.xml")
.toURI();
((org.apache.logging.log4j.core.LoggerContext) LogManager.getContext(false))
log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/LogLevelLabelConfigurationTest.java
+1 -1
View File
@@ -37,7 +37,7 @@ class LogLevelLabelConfigurationTest {
.withExposedPorts(3100);
@Test
void shouldWorkWIthNoLogLevelConfigured() throws URISyntaxException {
void shouldWorkWIthNoLogLevelConfigured() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/LokiAppenderLargeBatchesTestTest.java → log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/LokiAppenderLargeBatchesTest.java
+3 -4
View File
@@ -17,7 +17,6 @@ import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.List;
import java.util.concurrent.TimeUnit;
@@ -26,7 +25,7 @@ import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.CoreMatchers.hasItems;
@Testcontainers
class LokiAppenderLargeBatchesTestTest {
class LokiAppenderLargeBatchesTest {
@Container
public GenericContainer loki = new GenericContainer("grafana/loki:latest")
@@ -41,7 +40,7 @@ class LokiAppenderLargeBatchesTestTest {
.withExposedPorts(3100);
@Test
void shouldSendData() throws URISyntaxException {
void shouldSendData() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
@@ -65,7 +64,7 @@ class LokiAppenderLargeBatchesTestTest {
"Souffle cake muffin liquorice tart souffle pie sesame snaps.";
long expectedTimestamp = System.currentTimeMillis();
Logger logger = LogManager.getLogger(LokiAppenderLargeBatchesTestTest.class);
Logger logger = LogManager.getLogger(LokiAppenderLargeBatchesTest.class);
// When
for (int i = 0; i < 1000; i++) {
log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/LokiAppenderMonitoringTest.java
+4 -1
View File
@@ -21,9 +21,12 @@ class LokiAppenderMonitoringTest {
@Test
void shouldInjectMonitoringAndUseIt() throws URISyntaxException {
// Given
System.setProperty("loki.host", "somewhere");
System.setProperty("loki.port", "42");
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-log4j2-configuration.xml")
.getResource("basic-appender-test-configuration.xml")
.toURI();
LoggerContext context = (LoggerContext) LogManager.getContext(false);
log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/LokiAppenderShutdownTest.java
+1 -1
View File
@@ -40,7 +40,7 @@ class LokiAppenderShutdownTest {
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-log4j2-configuration.xml")
.getResource("basic-appender-test-configuration.xml")
.toURI();
((org.apache.logging.log4j.core.LoggerContext) LogManager.getContext(false))
log4j2-appender/src/test/java/pl/tkowalcz/tjahzi/log4j2/LokiAppenderTest.java
+2 -3
View File
@@ -17,7 +17,6 @@ import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.List;
import java.util.concurrent.TimeUnit;
@@ -41,14 +40,14 @@ class LokiAppenderTest {
.withExposedPorts(3100);
@Test
void shouldSendData() throws URISyntaxException {
void shouldSendData() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-log4j2-configuration.xml")
.getResource("basic-appender-test-configuration.xml")
.toURI();
((org.apache.logging.log4j.core.LoggerContext) LogManager.getContext(false))
log4j2-appender/src/test/resources/appender-test-large-batches.xml
+4 -2
View File
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
@@ -7,7 +7,9 @@
</Loggers>
<appenders>
<Loki name="Loki" bufferSizeMegabytes="2" batchWait="10000">
<Loki name="Loki" bufferSizeMegabytes="2" batchWait="10">
<batchSize>1048576</batchSize>
<host>${sys:loki.host}</host>
<port>${sys:loki.port}</port>
log4j2-appender/src/test/resources/appender-test-with-buffer-size-megabytes-set-to-64.xml
+1 -1
View File
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
log4j2-appender/src/test/resources/appender-test-with-buffer-size-megabytes-unset.xml
+1 -1
View File
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
log4j2-appender/src/test/resources/appender-test-with-log-label-set.xml
+1 -1
View File
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
log4j2-appender/src/test/resources/appender-test-with-log-label-unset.xml
+1 -1
View File
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
log4j2-appender/src/test/resources/basic-appender-test-log4j2-configuration.xml → log4j2-appender/src/test/resources/basic-appender-test-configuration.xml
+1 -1
View File
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="OFF" shutdownHook="disable" packages="pl.tkowalcz.tjahzi.log4j2">
<configuration packages="pl.tkowalcz.tjahzi.log4j2">
<Loggers>
<Root level="INFO">
<AppenderRef ref="Loki"/>
logback-appender-nodep/.gitignore
+2
View File
@@ -0,0 +1,2 @@
src/**
dependency-reduced-pom.xml
logback-appender-nodep/README.md
+3
View File
@@ -0,0 +1,3 @@
# Logback Appender - nodep
This repository contains only `pom.xml` to generate distribution with no external dependencies (other than logback-classic).
logback-appender-nodep/pom.xml
+97
View File
@@ -0,0 +1,97 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>tjahzi-parent</artifactId>
<version>0.9.5-SNAPSHOT</version>
</parent>
<name>logback-appender-nodep</name>
<artifactId>logback-appender-nodep</artifactId>
<packaging>jar</packaging>
<properties>
<!--add comma separated artifact names from dependencies, to be included in unpacking-->
<project.build.unpack_includes>logback-appender</project.build.unpack_includes>
</properties>
<dependencies>
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>logback-appender</artifactId>
<version>0.1.0-SNAPSHOT</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>unpack-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>unpack-dependencies</goal>
</goals>
</execution>
</executions>
<configuration>
<classifier>sources</classifier>
<type>jar</type>
<outputDirectory>${project.build.sourceDirectory}</outputDirectory>
<includeArtifactIds>${project.build.unpack_includes}</includeArtifactIds>
<includes>**\/*.java</includes>
<overWriteSnapshots>true</overWriteSnapshots>
</configuration>
</plugin>
<plugin>
<artifactId>maven-javadoc-plugin</artifactId>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>ch.qos.logback.logback-classic</exclude>
</excludes>
</artifactSet>
<relocations>
<relocation>
<pattern>org.agrona</pattern>
<shadedPattern>pl.tkowalcz.tjahzi.org.agrona</shadedPattern>
</relocation>
<relocation>
<pattern>io.netty</pattern>
<shadedPattern>pl.tkowalcz.tjahzi.io.netty</shadedPattern>
</relocation>
</relocations>
<promoteTransitiveDependencies>true</promoteTransitiveDependencies>
<minimizeJar>true</minimizeJar>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
logback-appender/README.md
+112
View File
@@ -0,0 +1,112 @@
# Logback Appender
On top of the `Core` component that sends logs to Loki this project gives you Logback appender.
## Quick start guide
1. Grab the no-dependency version of the appender:
```xml
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>logback-appender-nodep</artifactId>
<version>0.1.0</version>
</dependency>
```
If you already use a compatible version of Netty in your project then to reduce the size of your dependencies include
regular appender distribution:
```xml
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>logback-appender</artifactId>
<version>0.1.0</version>
</dependency>
```
1. Include minimal appender configuration in your existing logback configuration:
```xml
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<label>
<name>server</name>
<value>${HOSTNAME}</value>
</label>
</appender>
```
1. Reference the appender from inside one of your logger definitions:
```xml
<root level="debug">
<appender-ref ref="Loki"/>
</root>
```
## Advanced configuration
This example sets up a root logger with Loki appender. The appender definition must point to class `pl.tkowalcz.tjahzi.logback.LokiAppender`.
```xml
<configuration>
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Codewise</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
<logLevelLabel>
log_level
</logLevelLabel>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
```
### Lookups / variable substitution
Contents of the properties are automatically interpolated by Logback (see [here](http://logback.qos.ch/manual/configuration.html#variableSubstitution)).
All environment, system etc. variable references will be replaced by their values during initialization of the appender.
## Details
Let's go through the example config above and analyze configuration options (**Note: Tags are case-insensitive**).
| Tag | Required | Default value | Description |
|-----|----------|---------------|-------------|
| Host | Y | | Network host address of Loki instance. Either IP address or host name. It will be passed to Netty and end up being resolved by call to `InetSocketAddress.createUnresolved`. |
| Port | Y | | Self-explanatory :) |
| Header | N | (empty list) | This tag can be used multiple times to specify additional headers that are passed to Loki instance. One example is to pass a `X-Scope-OrgID` header when [running Loki in multi-tenant mode](https://grafana.com/docs/loki/latest/operations/authentication/). |
| Label | N | (empty list) | Specify additional labels attached to each log line sent via this appender instance. See also note about [label naming](https://github.com/tkowalcz/tjahzi/wiki/Label-naming). |
| LogLevelLabel | N | null | If defined then log level label of configured name will be added to each line sent to Loki. It will contain Log4j log level e.g. `INFO`, `WARN` etc. See also note about [label naming](https://github.com/tkowalcz/tjahzi/wiki/Label-naming). |
| BufferSizeMegabytes | N | 32 MB | Size of the `log buffer`. Must be power of two between 1MB and 1GB. See [log buffer sizing](https://github.com/tkowalcz/tjahzi/wiki/Log-buffer-sizing) for more explanations. |
| MaxRetries | N | 3 | Maximum number of retries to perform when delivering log message to Loki. Log buffer data is delivered in order, one batch after the other, so too much retries will block delivery of subsequent log batches (on the other hand if we need to retry many times then next batches will probably fail too). |
| ConnectTimeoutMillis | N | 5s | This configures socket connect timeout when connecting to Loki. After unsuccessful connection attempt it will continue to retry indefinitely employing exponential backoff (initial backoff = 250ms, maximum backoff = 30s, multiplier = 3). |
| ReadTimeoutMillis | N | 60s | Sets socket read timeout on Loki connection. |
| UseOffHeapBuffer | N | true | Whether Tjahzi should allocate native buffer for `Log buffer` component. We can go into a rabbit hole of divagations what are the implications of this. Most important in our view is that having 10s or 100s of MB of space taken out of heap is not very friendly to garbage collector which might have to occasionally copy it around. |
| BatchSize | N | 100 KB | Like in [promtail configuration](https://grafana.com/docs/loki/latest/clients/promtail/configuration/) `maximum batch size (in bytes) of logs to accumulate before sending the batch to Loki`.|
| BatchWait | N | 5s | Like in [promtail configuration](https://grafana.com/docs/loki/latest/clients/promtail/configuration/) `maximum amount of time to wait before sending a batch, even if that batch isn't full`.|
logback-appender/pom.xml
+94
View File
@@ -0,0 +1,94 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>tjahzi-parent</artifactId>
<version>0.9.5-SNAPSHOT</version>
</parent>
<name>logback-appender</name>
<artifactId>logback-appender</artifactId>
<packaging>jar</packaging>
<version>0.1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>pl.tkowalcz.tjahzi</groupId>
<artifactId>core</artifactId>
<version>0.9.5-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>4.1.17</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.code.java-allocation-instrumenter</groupId>
<artifactId>java-allocation-instrumenter</artifactId>
<version>3.3.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.rest-assured</groupId>
<artifactId>rest-assured</artifactId>
<version>4.3.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.0.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.15.0</version>
<scope>test</scope>
</dependency>
<!-- Testcontainers -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>1.14.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>1.14.3</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
logback-appender/src/main/java/pl/tkowalcz/tjahzi/logback/Header.java
+23
View File
@@ -0,0 +1,23 @@
package pl.tkowalcz.tjahzi.logback;
public class Header {
private String name;
private String value;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
}
logback-appender/src/main/java/pl/tkowalcz/tjahzi/logback/Label.java
+50
View File
@@ -0,0 +1,50 @@
package pl.tkowalcz.tjahzi.logback;
import java.util.Map;
import java.util.regex.Pattern;
public class Label {
private static final Pattern LABEL_NAME_PATTER = Pattern.compile("[a-zA-Z_][a-zA-Z0-9_]*");
private String name;
private String value;
// For Logback this is just a convenience method for testing
public static Label createLabel(String name, String value) {
Label result = new Label();
result.setName(name);
result.setValue(value);
return result;
}
public boolean hasValidName() {
return hasValidName(getName());
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
// @VisibleForTests
public Map.Entry<String, String> asEntry() {
return Map.entry(getName(), getValue());
}
public static boolean hasValidName(String label) {
return LABEL_NAME_PATTER.matcher(label).matches();
}
}
logback-appender/src/main/java/pl/tkowalcz/tjahzi/logback/LabelFactory.java
+119
View File
@@ -0,0 +1,119 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.core.spi.ContextAware;
import pl.tkowalcz.tjahzi.github.GitHubDocs;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import static java.util.Arrays.stream;
import static java.util.stream.Collectors.counting;
public class LabelFactory {
private final ContextAware internalLogger;
private final String logLevelLabel;
private final Label[] labels;
public LabelFactory(
ContextAware internalLogger,
String logLevelLabel,
Label... labels
) {
this.internalLogger = internalLogger;
this.logLevelLabel = logLevelLabel;
this.labels = labels;
}
public HashMap<String, String> convertLabelsDroppingInvalid() {
detectAndLogDuplicateLabels();
return convertAndLogViolations();
}
public String validateLogLevelLabel(HashMap<String, String> existingLabels) {
if (logLevelLabel != null) {
return validateLogLevelLabelAgainst(
existingLabels,
logLevelLabel
);
}
return null;
}
private void detectAndLogDuplicateLabels() {
List<String> duplicatedLabels = stream(labels)
.collect(Collectors.groupingBy(Label::getName, counting()))
.entrySet()
.stream().filter(entry -> entry.getValue() > 1)
.map(Map.Entry::getKey)
.collect(Collectors.toList());
if (!duplicatedLabels.isEmpty()) {
internalLogger.addWarn(
String.format(
"There are duplicated labels which is not allowed by Loki. " +
"These labels will be deduplicated non-deterministically: %s\n",
duplicatedLabels
)
);
}
}
private HashMap<String, String> convertAndLogViolations() {
HashMap<String, String> lokiLabels = new HashMap<>();
stream(labels)
.flatMap(label -> {
if (label.hasValidName()) {
return Stream.of(label);
}
internalLogger.addWarn(
String.format(
"Ignoring label '%s' - contains invalid characters. %s\n",
label.getName(),
GitHubDocs.LABEL_NAMING.getLogMessage()
)
);
return Stream.of();
}
)
.forEach(__ -> lokiLabels.put(__.getName(), __.getValue()));
return lokiLabels;
}
private String validateLogLevelLabelAgainst(
Map<String, String> existingLabels,
String logLevelLabel
) {
if (!Label.hasValidName(logLevelLabel)) {
internalLogger.addWarn(
String.format(
"Ignoring log level label '%s' - contains invalid characters. %s\n",
logLevelLabel,
GitHubDocs.LABEL_NAMING.getLogMessage()
)
);
return null;
}
if (existingLabels.remove(logLevelLabel) != null) {
internalLogger.addWarn(
String.format(
"Ignoring log level label '%s' - conflicts with label defined in configuration.\n",
logLevelLabel
)
);
}
return logLevelLabel;
}
}
logback-appender/src/main/java/pl/tkowalcz/tjahzi/logback/LokiAppender.java
+86
View File
@@ -0,0 +1,86 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.encoder.PatternLayoutEncoder;
import ch.qos.logback.classic.spi.ILoggingEvent;
import pl.tkowalcz.tjahzi.LoggingSystem;
import pl.tkowalcz.tjahzi.TjahziLogger;
import pl.tkowalcz.tjahzi.stats.MonitoringModule;
import pl.tkowalcz.tjahzi.stats.MutableMonitoringModuleWrapper;
import java.nio.ByteBuffer;
import java.util.Collections;
public class LokiAppender extends LokiAppenderConfigurator {
private PatternLayoutEncoder encoder;
private LoggingSystem loggingSystem;
private TjahziLogger logger;
private String logLevelLabel;
private MutableMonitoringModuleWrapper monitoringModuleWrapper;
public PatternLayoutEncoder getEncoder() {
return encoder;
}
public void setEncoder(PatternLayoutEncoder encoder) {
this.encoder = encoder;
}
/**
* This is an entry point to set monitoring (statistics) hooks for this appender. This
* API is in beta and is subject to change (and probably will).
*/
public void setMonitoringModule(MonitoringModule monitoringModule) {
monitoringModuleWrapper.setMonitoringModule(monitoringModule);
}
// @VisibleForTesting
public LoggingSystem getLoggingSystem() {
return loggingSystem;
}
@Override
protected void append(ILoggingEvent event) {
String logLevel = event.getLevel().toString();
byte[] logLine = encoder.encode(event);
logger.log(
event.getTimeStamp(),
Collections.emptyMap(),
logLevelLabel,
logLevel,
ByteBuffer.wrap(logLine)
);
}
@Override
public void start() {
if (encoder == null) {
addError("No encoder set for the appender named [" + name + "].");
return;
}
encoder.start();
LokiAppenderFactory lokiAppenderFactory = new LokiAppenderFactory(this);
loggingSystem = lokiAppenderFactory.createAppender();
logLevelLabel = lokiAppenderFactory.getLogLevelLabel();
monitoringModuleWrapper = lokiAppenderFactory.getMonitoringModuleWrapper();
logger = loggingSystem.createLogger();
loggingSystem.start();
super.start();
}
@Override
public void stop() {
loggingSystem.close(
1000,
thread -> addError("Loki appender was unable to stop thread on shutdown: " + thread)
);
super.stop();
}
}
logback-appender/src/main/java/pl/tkowalcz/tjahzi/logback/LokiAppenderConfigurator.java
+136
View File
@@ -0,0 +1,136 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.UnsynchronizedAppenderBase;
import java.util.ArrayList;
import java.util.List;
public abstract class LokiAppenderConfigurator extends UnsynchronizedAppenderBase<ILoggingEvent> {
static final int BYTES_IN_MEGABYTE = 1024 * 1024;
private String host;
private int port;
private int connectTimeoutMillis = 5000;
private int readTimeoutMillis = 60_000;
private int maxRetries = 3;
private int bufferSizeMegabytes = 32;
private boolean useOffHeapBuffer = true;
private String logLevelLabel;
private long batchSize = 10_2400;
private long batchWait = 5;
private int maxRequestsInFlight = 100;
private final List<Header> headers = new ArrayList<>();
private final List<Label> labels = new ArrayList<>();
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public int getPort() {
return port;
}
public void setPort(int port) {
this.port = port;
}
public int getConnectTimeoutMillis() {
return connectTimeoutMillis;
}
public void setConnectTimeoutMillis(int connectTimeoutMillis) {
this.connectTimeoutMillis = connectTimeoutMillis;
}
public int getReadTimeoutMillis() {
return readTimeoutMillis;
}
public void setReadTimeoutMillis(int readTimeoutMillis) {
this.readTimeoutMillis = readTimeoutMillis;
}
public int getMaxRetries() {
return maxRetries;
}
public void setMaxRetries(int maxRetries) {
this.maxRetries = maxRetries;
}
public int getBufferSizeMegabytes() {
return bufferSizeMegabytes;
}
public void setBufferSizeMegabytes(int bufferSizeMegabytes) {
this.bufferSizeMegabytes = bufferSizeMegabytes;
}
public boolean isUseOffHeapBuffer() {
return useOffHeapBuffer;
}
public void setUseOffHeapBuffer(boolean useOffHeapBuffer) {
this.useOffHeapBuffer = useOffHeapBuffer;
}
public String getLogLevelLabel() {
return logLevelLabel;
}
public void setLogLevelLabel(String logLevelLabel) {
this.logLevelLabel = logLevelLabel;
}
public void setBatchSize(long batchSize) {
this.batchSize = batchSize;
}
public void setBatchWait(long batchWait) {
this.batchWait = batchWait;
}
public long getBatchSize() {
return batchSize;
}
public long getBatchWait() {
return batchWait;
}
public int getMaxRequestsInFlight() {
return maxRequestsInFlight;
}
public void setMaxRequestsInFlight(int maxRequestsInFlight) {
this.maxRequestsInFlight = maxRequestsInFlight;
}
public void addHeader(Header header) {
headers.add(header);
}
public List<Header> getHeaders() {
return headers;
}
public void addLabel(Label label) {
labels.add(label);
}
public List<Label> getLabels() {
return labels;
}
}
logback-appender/src/main/java/pl/tkowalcz/tjahzi/logback/LokiAppenderFactory.java
+91
View File
@@ -0,0 +1,91 @@
package pl.tkowalcz.tjahzi.logback;
import pl.tkowalcz.tjahzi.LoggingSystem;
import pl.tkowalcz.tjahzi.TjahziInitializer;
import pl.tkowalcz.tjahzi.github.GitHubDocs;
import pl.tkowalcz.tjahzi.http.ClientConfiguration;
import pl.tkowalcz.tjahzi.http.HttpClientFactory;
import pl.tkowalcz.tjahzi.http.NettyHttpClient;
import pl.tkowalcz.tjahzi.stats.MutableMonitoringModuleWrapper;
import pl.tkowalcz.tjahzi.stats.StandardMonitoringModule;
import java.util.HashMap;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
public class LokiAppenderFactory {
private final LokiAppenderConfigurator configurator;
private final HashMap<String, String> lokiLabels;
private final String logLevelLabel;
private MutableMonitoringModuleWrapper monitoringModuleWrapper;
public LokiAppenderFactory(LokiAppenderConfigurator configurator) {
this.configurator = configurator;
LabelFactory labelFactory = new LabelFactory(
configurator,
configurator.getLogLevelLabel(),
configurator.getLabels().toArray(new Label[0])
);
lokiLabels = labelFactory.convertLabelsDroppingInvalid();
logLevelLabel = labelFactory.validateLogLevelLabel(lokiLabels);
}
public LoggingSystem createAppender() {
ClientConfiguration configurationBuilder = ClientConfiguration.builder()
.withHost(configurator.getHost())
.withConnectionTimeoutMillis(configurator.getConnectTimeoutMillis())
.withPort(configurator.getPort())
.withMaxRetries(configurator.getMaxRetries())
.withRequestTimeoutMillis(configurator.getReadTimeoutMillis())
.withMaxRequestsInFlight(configurator.getMaxRequestsInFlight())
.build();
String[] additionalHeaders = configurator.getHeaders().stream()
.flatMap(header -> Stream.of(header.getName(), header.getValue()))
.toArray(String[]::new);
monitoringModuleWrapper = new MutableMonitoringModuleWrapper();
monitoringModuleWrapper.setMonitoringModule(new StandardMonitoringModule());
NettyHttpClient httpClient = HttpClientFactory
.defaultFactory()
.getHttpClient(
configurationBuilder,
monitoringModuleWrapper,
additionalHeaders
);
int bufferSizeBytes = configurator.getBufferSizeMegabytes() * LokiAppenderConfigurator.BYTES_IN_MEGABYTE;
if (!TjahziInitializer.isCorrectSize(bufferSizeBytes)) {
configurator.addWarn(
String.format(
"Invalid log buffer size %d - using nearest power of two greater than provided value, no less than 1MB. %s\n",
bufferSizeBytes,
GitHubDocs.LOG_BUFFER_SIZING.getLogMessage()
)
);
}
return new TjahziInitializer().createLoggingSystem(
httpClient,
monitoringModuleWrapper,
lokiLabels,
configurator.getBatchSize(),
TimeUnit.SECONDS.toMillis(configurator.getBatchWait()),
bufferSizeBytes,
configurator.isUseOffHeapBuffer()
);
}
public String getLogLevelLabel() {
return logLevelLabel;
}
public MutableMonitoringModuleWrapper getMonitoringModuleWrapper() {
return monitoringModuleWrapper;
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/BufferSizeConfigurationTest.java
+58
View File
@@ -0,0 +1,58 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import pl.tkowalcz.tjahzi.logback.util.TestUtil;
import java.net.URI;
import static org.assertj.core.api.Assertions.assertThat;
class BufferSizeConfigurationTest {
@Test
void shouldUseDefaultWhenLogBufferSizeIsNotSetViaConfiguration() throws Exception {
// Given
URI uri = getClass()
.getClassLoader()
.getResource("appender-test-with-buffer-size-megabytes-unset.xml")
.toURI();
// When
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
// Then
LokiAppender loki = TestUtil.getLokiAppender(context);
assertThat(loki.getLoggingSystem().getLogBufferSize()).isEqualTo(32 * 1024 * 1024);
}
@Test
void shouldSetLogBufferSizeViaConfiguration() throws Exception {
// Given
URI uri = getClass()
.getClassLoader()
.getResource("appender-test-with-buffer-size-megabytes-set-to-64.xml")
.toURI();
// When
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
// Then
LokiAppender loki = TestUtil.getLokiAppender(context);
assertThat(loki.getLoggingSystem().getLogBufferSize()).isEqualTo(64 * 1024 * 1024);
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LabelFactoryTest.java
+145
View File
@@ -0,0 +1,145 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.core.ConsoleAppender;
import org.junit.jupiter.api.Test;
import java.util.HashMap;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
class LabelFactoryTest {
@Test
void shouldRemoveDuplicatedLabels() {
// Given
Label thisIsADuplicateAndShouldBeDropped = Label.createLabel("ip", "127.0.0.1");
Label thisShouldStay = Label.createLabel("ip", "10.0.2.34");
Label thisTooShouldStay = Label.createLabel("hostname", "hostname");
LabelFactory labelFactory = new LabelFactory(
new ConsoleAppender<>(),
"log_level",
thisIsADuplicateAndShouldBeDropped,
thisShouldStay,
thisTooShouldStay
);
// When
HashMap<String, String> actual = labelFactory.convertLabelsDroppingInvalid();
// Then
assertThat(actual).containsOnly(
thisShouldStay.asEntry(),
thisTooShouldStay.asEntry()
);
}
@Test
void shouldSkipLabelsWithInvalidName() {
// Given
Label thisShouldStay = Label.createLabel("ip", "10.0.2.34");
Label thisShouldStayToo = Label.createLabel("region", "coruscant-west-2");
Label invalidNameShouldBeDropped = Label.createLabel("host-name", "hostname");
LabelFactory labelFactory = new LabelFactory(
new ConsoleAppender<>(),
"log_level",
thisShouldStayToo,
thisShouldStay,
invalidNameShouldBeDropped
);
// When
HashMap<String, String> actual = labelFactory.convertLabelsDroppingInvalid();
// Then
assertThat(actual).containsOnly(
thisShouldStay.asEntry(),
thisShouldStayToo.asEntry()
);
}
@Test
void shouldAcceptNullLogLevel() {
// Given
Label label1 = Label.createLabel("ip", "10.0.2.34");
Label label2 = Label.createLabel("region", "coruscant-west-2");
LabelFactory labelFactory = new LabelFactory(
new ConsoleAppender<>(),
null,
label2,
label1
);
// When
HashMap<String, String> actual = labelFactory.convertLabelsDroppingInvalid();
// Then
assertThat(actual).containsOnly(
label1.asEntry(),
label2.asEntry()
);
}
@Test
void logLevelLabelShouldOverrideConflictingLabel() {
// Given
String logLevelLabel = "log_level";
Label thisShouldStay = Label.createLabel("ip", "10.0.2.34");
Label thisShouldBeRemovedDueToConflict = Label.createLabel(logLevelLabel, "coruscant-west-2");
LabelFactory labelFactory = new LabelFactory(
new ConsoleAppender<>(),
logLevelLabel,
thisShouldBeRemovedDueToConflict,
thisShouldStay
);
HashMap<String, String> labels = new HashMap<>(
Map.ofEntries(
thisShouldStay.asEntry(),
thisShouldBeRemovedDueToConflict.asEntry()
)
);
// When
String actual = labelFactory.validateLogLevelLabel(labels);
// Then
assertThat(labels).containsOnly(
thisShouldStay.asEntry()
);
assertThat(actual).isEqualTo(logLevelLabel);
}
@Test
void shouldDisableLogLevelLabelIfItHasInvalidName() {
// Given
String logLevelLabel = "log-level";
LabelFactory labelFactory = new LabelFactory(new ConsoleAppender<>(), logLevelLabel);
// When
String actual = labelFactory.validateLogLevelLabel(new HashMap<>());
// Then
assertThat(actual).isNull();
}
@Test
void shouldHandleDisabledLogLevelLabel() {
// Given
String logLevelLabel = null;
LabelFactory labelFactory = new LabelFactory(new ConsoleAppender<>(), logLevelLabel);
// When
String actual = labelFactory.validateLogLevelLabel(new HashMap<>());
// Then
assertThat(actual).isNull();
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LabelTest.java
+21
View File
@@ -0,0 +1,21 @@
package pl.tkowalcz.tjahzi.logback;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
class LabelTest {
@Test
void shouldValidateNamePattern() {
// Given
Label correctLabel = Label.createLabel("aaa_bbb_ccc", "fobar");
Label tooShortLabel = Label.createLabel("", "foobar");
Label labelWithInvalidChars = Label.createLabel("log-level", "foobar");
// When & Then
assertThat(correctLabel.hasValidName()).isTrue();
assertThat(tooShortLabel.hasValidName()).isFalse();
assertThat(labelWithInvalidChars.hasValidName()).isFalse();
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LogLevelLabelConfigurationTest.java
+141
View File
@@ -0,0 +1,141 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import io.restassured.RestAssured;
import io.restassured.http.ContentType;
import io.restassured.parsing.Parser;
import org.awaitility.Awaitility;
import org.awaitility.Durations;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.BindMode;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.net.URI;
import static io.restassured.RestAssured.given;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.CoreMatchers.nullValue;
@Testcontainers
class LogLevelLabelConfigurationTest {
@Container
public GenericContainer loki = new GenericContainer("grafana/loki:latest")
.withCommand("-config.file=/etc/loki-config.yaml")
.withClasspathResourceMapping("loki-config.yaml",
"/etc/loki-config.yaml",
BindMode.READ_ONLY)
.waitingFor(
Wait.forHttp("/ready")
.forPort(3100)
)
.withExposedPorts(3100);
@Test
void shouldWorkWIthNoLogLevelConfigured() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
URI uri = getClass()
.getClassLoader()
.getResource("appender-test-with-log-label-unset.xml")
.toURI();
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
// When
Logger logger = context.getLogger(LokiAppenderLargeBatchesTest.class);
logger.info("Test");
// Then
RestAssured.port = loki.getFirstMappedPort();
RestAssured.baseURI = "http://" + loki.getHost();
RestAssured.registerParser("text/plain", Parser.JSON);
Awaitility
.await()
.atMost(Durations.TEN_SECONDS)
.pollInterval(Durations.ONE_SECOND)
.ignoreExceptions()
.untilAsserted(() -> {
given()
.contentType(ContentType.URLENC)
.urlEncodingEnabled(false)
.formParam("query=%7Bserver%3D%22127.0.0.1%22%7D")
.when()
.get("/loki/api/v1/query_range")
.then()
.log()
.all()
.statusCode(200)
.body("status", equalTo("success"))
.body("data.result.size()", equalTo(1))
.body("data.result[0].stream.server", equalTo("127.0.0.1"))
.body("data.result[0].stream.log_level", nullValue());
});
}
@Test
void shouldSendLogLevelAsConfigured() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
URI uri = getClass()
.getClassLoader()
.getResource("appender-test-with-log-label-set.xml")
.toURI();
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
// When
Logger logger = context.getLogger(LokiAppenderLargeBatchesTest.class);
logger.info("Test");
// Then
RestAssured.port = loki.getFirstMappedPort();
RestAssured.baseURI = "http://" + loki.getHost();
RestAssured.registerParser("text/plain", Parser.JSON);
Awaitility
.await()
.atMost(Durations.TEN_SECONDS)
.pollInterval(Durations.ONE_SECOND)
.ignoreExceptions()
.untilAsserted(() -> {
given()
.contentType(ContentType.URLENC)
.urlEncodingEnabled(false)
.formParam("query=%7Bserver%3D%22127.0.0.1%22%7D")
.when()
.get("/loki/api/v1/query_range")
.then()
.log()
.all()
.statusCode(200)
.body("status", equalTo("success"))
.body("data.result.size()", equalTo(1))
.body("data.result[0].stream.server", equalTo("127.0.0.1"))
.body("data.result[0].stream.log_level", equalTo("INFO"));
});
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LokiAppenderLargeBatchesTest.java
+130
View File
@@ -0,0 +1,130 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import io.restassured.RestAssured;
import io.restassured.http.ContentType;
import io.restassured.parsing.Parser;
import org.awaitility.Awaitility;
import org.awaitility.Durations;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.BindMode;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.net.URI;
import java.util.List;
import java.util.concurrent.TimeUnit;
import static io.restassured.RestAssured.given;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.CoreMatchers.hasItems;
@Testcontainers
class LokiAppenderLargeBatchesTest {
@Container
public GenericContainer loki = new GenericContainer("grafana/loki:latest")
.withCommand("-config.file=/etc/loki-config.yaml")
.withClasspathResourceMapping("loki-config.yaml",
"/etc/loki-config.yaml",
BindMode.READ_ONLY)
.waitingFor(
Wait.forHttp("/ready")
.forPort(3100)
)
.withExposedPorts(3100);
@Test
void shouldSendData() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
URI uri = getClass()
.getClassLoader()
.getResource("appender-test-large-batches.xml")
.toURI();
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
String expectedLogLine = "Cupcake ipsum dolor sit amet cake wafer. " +
"Souffle jelly beans biscuit topping. " +
"Danish bonbon gummies powder caramels. " +
"Danish jelly beans sweet roll topping jelly beans oat cake toffee. " +
"Chocolate cake sesame snaps brownie biscuit cheesecake. " +
"Ice cream dessert sweet donut marshmallow. " +
"Muffin bear claw cookie jelly-o sugar plum jelly beans apple pie fruitcake cookie. " +
"Tootsie roll carrot cake pastry jujubes jelly beans chupa chups. " +
"Souffle cake muffin liquorice tart souffle pie sesame snaps.";
long expectedTimestamp = System.currentTimeMillis();
Logger logger = context.getLogger(LokiAppenderLargeBatchesTest.class);
// When
for (int i = 0; i < 1000; i++) {
logger.info(i + " " + expectedLogLine);
}
// Then
RestAssured.port = loki.getFirstMappedPort();
RestAssured.baseURI = "http://" + loki.getHost();
RestAssured.registerParser("text/plain", Parser.JSON);
Awaitility
.await()
.atMost(Durations.ONE_MINUTE)
.pollInterval(Durations.ONE_SECOND)
.ignoreExceptions()
.untilAsserted(() -> {
given()
.contentType(ContentType.URLENC)
.urlEncodingEnabled(false)
.formParam("&start=" + expectedTimestamp + "&limit=1000&query=%7Bserver%3D%22127.0.0.1%22%7D")
.when()
.get("/loki/api/v1/query_range")
.then()
.log()
.all()
.statusCode(200)
.body("status", equalTo("success"))
.body("data.result.size()", equalTo(1))
.body("data.result[0].stream.server", equalTo("127.0.0.1"))
.body("data.result[0].values.size()", equalTo(1000))
.body("data.result[0].values", hasItems(new BaseMatcher<>() {
@Override
public boolean matches(Object o) {
List<Object> list = (List<Object>) o;
if (list.size() != 2) {
return false;
}
long actualTimestamp = Long.parseLong(list.get(0).toString());
String actualLogLine = list.get(1).toString();
return actualLogLine.contains(expectedLogLine)
&& (expectedTimestamp - actualTimestamp) < TimeUnit.MINUTES.toMillis(1);
}
@Override
public void describeTo(Description description) {
}
}
));
});
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LokiAppenderMonitoringTest.java
+68
View File
@@ -0,0 +1,68 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import com.codahale.metrics.Counter;
import com.codahale.metrics.MetricRegistry;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import org.testcontainers.junit.jupiter.Testcontainers;
import pl.tkowalcz.tjahzi.logback.util.TestUtil;
import pl.tkowalcz.tjahzi.stats.DropwizardMonitoringModule;
import java.net.URI;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
@Testcontainers
class LokiAppenderMonitoringTest {
@Test
void shouldInjectMonitoringAndUseIt() throws Exception {
// Given
System.setProperty("loki.host", "somewhere");
System.setProperty("loki.port", "42");
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-configuration.xml")
.toURI();
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
LokiAppender loki = TestUtil.getLokiAppender(context);
MetricRegistry metricRegistry = new MetricRegistry();
loki.setMonitoringModule(
new DropwizardMonitoringModule(
metricRegistry,
"appender.loki"
)
);
// When
Logger logger = context.getLogger(LokiAppenderLargeBatchesTest.class);
logger.info("Test test test");
// Then
assertThat(metricRegistry.getMetrics()).isNotEmpty();
Counter connectAttemptsCounter = metricRegistry
.getCounters()
.get("appender.loki.httpConnectAttempts");
assertThat(connectAttemptsCounter).isNotNull();
await().untilAsserted(() -> {
assertThat(connectAttemptsCounter.getCount()).isEqualTo(1);
}
);
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LokiAppenderShutdownTest.java
+84
View File
@@ -0,0 +1,84 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import org.awaitility.Awaitility;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.BindMode;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.net.URI;
import static org.assertj.core.api.Assertions.assertThat;
@Testcontainers
class LokiAppenderShutdownTest {
@Container
public GenericContainer loki = new GenericContainer("grafana/loki:latest")
.withCommand("-config.file=/etc/loki-config.yaml")
.withClasspathResourceMapping("loki-config.yaml",
"/etc/loki-config.yaml",
BindMode.READ_ONLY)
.waitingFor(
Wait.forHttp("/ready")
.forPort(3100)
)
.withExposedPorts(3100);
@Test
void shouldSendData() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-configuration.xml")
.toURI();
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
// Verify our assumptions that we can find threads started by Tjahzi
Awaitility.await().untilAsserted(() -> {
ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = threadMXBean.dumpAllThreads(false, false);
assertThat(threadInfos)
.extracting(ThreadInfo::getThreadName)
.contains(
"ReadingLogBufferAndSendingHttp",
"tjahzi-worker"
);
});
// When
context.stop();
// Then
Awaitility.await().untilAsserted(() -> {
ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = threadMXBean.dumpAllThreads(false, false);
assertThat(threadInfos)
.extracting(ThreadInfo::getThreadName)
.doesNotContain(
"ReadingLogBufferAndSendingHttp",
"tjahzi-worker"
);
});
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/LokiAppenderTest.java
+118
View File
@@ -0,0 +1,118 @@
package pl.tkowalcz.tjahzi.logback;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import io.restassured.RestAssured;
import io.restassured.http.ContentType;
import io.restassured.parsing.Parser;
import org.awaitility.Awaitility;
import org.awaitility.Durations;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.BindMode;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import java.net.URI;
import java.util.List;
import java.util.concurrent.TimeUnit;
import static io.restassured.RestAssured.given;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.CoreMatchers.hasItems;
@Testcontainers
class LokiAppenderTest {
@Container
public GenericContainer loki = new GenericContainer("grafana/loki:latest")
.withCommand("-config.file=/etc/loki-config.yaml")
.withClasspathResourceMapping("loki-config.yaml",
"/etc/loki-config.yaml",
BindMode.READ_ONLY)
.waitingFor(
Wait.forHttp("/ready")
.forPort(3100)
)
.withExposedPorts(3100);
@Test
void shouldSendData() throws Exception {
// Given
System.setProperty("loki.host", loki.getHost());
System.setProperty("loki.port", loki.getFirstMappedPort().toString());
URI uri = getClass()
.getClassLoader()
.getResource("basic-appender-test-configuration.xml")
.toURI();
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
context.reset();
configurator.doConfigure(uri.toURL());
String expectedLogLine = "Test";
long expectedTimestamp = System.currentTimeMillis();
// When
Logger logger = context.getLogger(LokiAppenderTest.class);
logger.info(expectedLogLine);
// Then
RestAssured.port = loki.getFirstMappedPort();
RestAssured.baseURI = "http://" + loki.getHost();
RestAssured.registerParser("text/plain", Parser.JSON);
Awaitility
.await()
.atMost(Durations.TEN_SECONDS)
.pollInterval(Durations.ONE_SECOND)
.ignoreExceptions()
.untilAsserted(() -> {
given()
.contentType(ContentType.URLENC)
.urlEncodingEnabled(false)
.formParam("query=%7Bserver%3D%22127.0.0.1%22%7D")
.when()
.get("/loki/api/v1/query_range")
.then()
.log()
.all()
.statusCode(200)
.body("status", equalTo("success"))
.body("data.result.size()", equalTo(1))
.body("data.result[0].stream.server", equalTo("127.0.0.1"))
.body("data.result[0].values", hasItems(new BaseMatcher<>() {
@Override
public boolean matches(Object o) {
List<Object> list = (List<Object>) o;
if (list.size() != 2) {
return false;
}
long actualTimestamp = Long.parseLong(list.get(0).toString());
String actualLogLine = list.get(1).toString();
return actualLogLine.contains(expectedLogLine)
&& (expectedTimestamp - actualTimestamp) < TimeUnit.MINUTES.toMillis(1);
}
@Override
public void describeTo(Description description) {
}
}
));
});
}
}
logback-appender/src/test/java/pl/tkowalcz/tjahzi/logback/util/TestUtil.java
+26
View File
@@ -0,0 +1,26 @@
package pl.tkowalcz.tjahzi.logback.util;
import ch.qos.logback.classic.LoggerContext;
import pl.tkowalcz.tjahzi.logback.LokiAppender;
import java.util.Spliterators;
import java.util.stream.StreamSupport;
public class TestUtil {
public static LokiAppender getLokiAppender(LoggerContext context) {
return (LokiAppender) context
.getLoggerList()
.stream()
.flatMap(
logger ->
StreamSupport.stream(
Spliterators.spliteratorUnknownSize(logger.iteratorForAppenders(), 0),
false
)
)
.filter(appender -> appender instanceof LokiAppender)
.findAny()
.orElseThrow(() -> new AssertionError("Expected to find Loki appender"));
}
}
logback-appender/src/test/resources/appender-test-large-batches.xml
+29
View File
@@ -0,0 +1,29 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<bufferSizeMegabytes>2</bufferSizeMegabytes>
<batchWait>10</batchWait>
<batchSize>1048576</batchSize>
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Circus</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/appender-test-with-buffer-size-megabytes-set-to-64.xml
+27
View File
@@ -0,0 +1,27 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<bufferSizeMegabytes>64</bufferSizeMegabytes>
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Circus</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/appender-test-with-buffer-size-megabytes-unset.xml
+24
View File
@@ -0,0 +1,24 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Circus</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/appender-test-with-log-label-set.xml
+28
View File
@@ -0,0 +1,28 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Circus</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
<logLevelLabel>
log_level
</logLevelLabel>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/appender-test-with-log-label-unset.xml
+24
View File
@@ -0,0 +1,24 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Circus</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/basic-appender-test-configuration.xml
+24
View File
@@ -0,0 +1,24 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Circus</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/logback-configuration.xml
+28
View File
@@ -0,0 +1,28 @@
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
<header>
<name>X-Org-Id</name>
<value>Codewise</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
<logLevelLabel>
log_level
</logLevelLabel>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>
logback-appender/src/test/resources/loki-config.yaml
+48
View File
@@ -0,0 +1,48 @@
auth_enabled: false
server:
http_listen_port: 3100
log_level: debug
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
min_ready_duration: 0s
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0
schema_config:
configs:
- from: 2018-04-15
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
storage_config:
boltdb:
directory: /loki/index
filesystem:
directory: /loki/chunks
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_entries_limit_per_query: 10000
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
pom.xml
+2
View File
@@ -15,6 +15,8 @@
<module>loki-protobuf</module>
<module>log4j2-appender</module>
<module>log4j2-appender-nodep</module>
<module>logback-appender</module>
<module>logback-appender-nodep</module>
<module>core</module>
</modules>