Table of Contents

1. Behavior changes
2. D4 forward

d4-goclient v0.4 has been released with several improvements and the addition of one major feature: the ability to forward data natively between d4 servers with the go client without using an analyzer to query the source d4 server.

We discuss here some behavior changes that appear in the release CHANGELOG.

Behavior changes

• chg: [main] no exit unless specified
• chg: [main] rate limiter when reaching EOF

With this release, d4-goclient will behave differently: it won’t stop on EOF by default but instead continue to poll the source for new element to ship to the destination. This change comes with the ability to limit the rate at which the source will be polled. This is controlled by the new rate limiter flag -rl set by default at 200ms. To regain the previous behavior, one should append the flag -rt set to 0 when launching the program.

• chg: [main] no output on stdout unless specified
• chg: [log] proper log file

d4-goclient had the silly behavior to write some information that should go into a log file into stdout. This was not an issue while having a network destination but posed unnecessary parsing issues when sending to stdout. This is now fixed and all logging lines go into a d4-goclient.log file.

D4 forward

• add: [input] d4 forward from d4 server redis

Here comes the new feature that will considerably ease communications between D4 servers and other tools that can push into a redis queue. Consider this blog post where we describe how to forward traffic from a server to another:

$ ./d4-stdout.py -t 1 -u 84723644-0841-4580-97e9-23e98682739c -f | zcat | tcprewrite --pnat=10.1.0.0/16:192.168.0.0/16 -i - -o - | d4-goclient -c ~/conf.sample 

In this particular situation we use tcprewrite to remove some private IP addresses. But in a case where this is not necessary we would be left with using an analyzer to write to stdout, only to pipe stdin again in a d4 client:

$ ./d4-stdout.py -t 1 -u 84723644-0841-4580-97e9-23e98682739c -f | d4-goclient -c ~/conf.sample 

This was an unnecessary step and this is now fixed with the new release of the go client:

$ d4-goclient -c ~/conf.sample 

This will actually do the same job if in conf.sample:

• redis_d4 is a file that points the location of the source redis server/database. For instance:

127.0.0.1:6380/2

• redis_queue is a file that points the queue to pop. For instance:

analyzer:3:d42967c1-f7ad-464e-bbc7-4464c653d7a6

What’s up with analyzer:3, and specifying the right database? Well, the client does not treat d4 server’s redis differently that any other redis. This has the nice implication that the client can be used to pop any redis queue, not only d4. We will make use of this property in our next blog post in which we will show how to use Apache NiFi with D4.

Stay tuned.