[BTrace: update from trenches] - Unattended execution

A bit of history

BTrace origins go back more than 10 years and it shows that the modus operandi at that time was to have one JVM at a time and do all the experiments and debugging on that single JVM.
The standard workflow for BTrace dynamic attach was decided to be:

1. Identify the JVM to attach to
2. Attach to that JVM and deploy the probe(s)
3. Stay connected until you are satisfied with the results

While probably quite ok for the one JVM situation it becomes quite a hindrance when trying to operate in ad-hoc mode for bunch of JVMs running on several hosts at once (yes, k8s, talking about you). The lack of unattended execution basically makes BTrace unusable in modern environments - if dynamic attach is required.

Time to fix

So, after a period of procrastination I finally decided to add unattended mode of execution to BTrace. It was both easy and hard task at the same time - the binary I/O protocol BTrace is using is very easy to extend but the the underlying management of client 'sessions' had to be refactored slightly to allow disconnecting and reconnecting to a session without killing it. But nothing that could not have been done during a particularly grey and rainy COVID lockdown weekend.

So, here we go with a few improvements to the BTrace client which should make it much easier to use BTrace in fire&forget mode - which, I think, will become more and more popular with the JFR support when one can easily define a dynamic JFR event type and deploy a probe to generate that event and leave the probe running, turning on and of the JFR recording as necessary.

• 'Detach client' client command
  In addition to 'Exit' option in the BTrace CLI it is now possible to detach from the running session.
  Upon detaching a unique probe ID is displayed which can be used to later reconnect to that probe.
  
  

• 'List probes' client command
  This command will list any probes which were deployed and the clients left them detached.
  
  
  

• List probes from command line
  Use btrace -lp <pid> to list the probes in detached mode in a particular JVM
  
  
  
  
  
  
  
  
• Reconnect to a detached probe
  Use btrace -r <probe id> <pid> to reconnect to a detached probe and start receiving probe data.
  The detached probes are maintaining a circular buffer for the latest data so you can get a bit of history after reconnecting as well.
• Attach a probe and disconnect immediately
  Useful shortcut for scripting BTrace deployments when the probe is deployed and the client disconnects immediately.
  Use btrace <btrace options> -x <pid> <probe file> to run in this mode. 
  Upon disconnecting the probe ID is printed out so it can be eg. processed and stored.

New possibilities

Having implemented the support for listing the detached probes and reconnecting to them as a form of command line switches opened doors to easy scripting when one can write a quick one-liner to attach to a named probe:
./bin/btrace -r $(./bin/btrace -lp anagrams.jar | fgrep AllMethods1 | cut -f2 -d' ') anagrams.jar

The unattended execution support was checked in and a development build binaries are available at https://github.com/btraceio/btrace/actions/runs/394037357

[BTrace: update from trenches] - Experimental support for emitting JFR events, take two

In the previous post I have introduced the prototype of JFR support in BTrace.

The first attempt, however, was riddled with serious shortcomings - the events had to be defined externally and then added to boot classpath.
In addition to that the BTrace verifier had to be made more permeable to allow calls to JFR APIs - this caused the verifier complexity to increase significantly, opening potential holes to be exploited.

Fortunately, there is a very cool API directly in JDK which allows creating JFR event types dynamically, therefore removing the requirement to have the events defined beforehand and added to boot classpath. As an added benefit the refactoring allowed the use of the standard BTraceUtils accessor class for operating on JFR events, thus removing all the custom 'holes' in the BTrace verifier.

The implementation is available on GitHub in jfr_events branch. It is fairly complete (at least for the use cases I was able to come up with) but as usual a user input is more than welcome.

The code example showing the intended usage follows.

Fig.1: Code Example

[BTrace: update from trenches] - Experimental support for emitting JFR events

Java Flight Recorder (JFR) is an amazing piece of technology allowing collection of a huge amount of very detailed data points from the running application and (also) Java runtime.
It has been widely available since Java 9 and recently it has been backported to JDK 8 update 265 - thus covering all Java version currently available (intentionally disregarding JDK 7 which I really hope will gracefully fade away very soon).

Having a standardized, low impact way to collect data from a running Java application is something BTrace can hugely benefit from.
Among other things this will allow seamless integration with tools already using JFR as their native format (JDK Mission Control or, recently, also VisualVM) creating synergy between 'free-form' instrumentation and well established perf analysis tools.

Although the implementation might seem quite trivial at the first look it quickly becomes more involved because the BTrace safety guards needs to be modified and extended to allow easy cooperation with JFR events while not compromising the security guarantees. Also, it is imperative that working JFR events does not need introducing and learning any new concepts - everything should be expressible via annotations and plain Java code. 

After pondering all the requirements for a while I came up with the idea to split periodic and non-periodic event usage. This allows a neat registration of periodic event handlers while keeping the simple event handling code, well, simple.

Here are the proposed annotations:

• @JfrPeriodicEventHandler
• used to define the JFR related code which will be run at the beginning/end of a recording chunk or at a given time interval
  
• the periodic event is passed in the handler as a method parameter by BTrace (the handler method must have exactly one argument which must be a jdk.jfr.Event subclass)
• all safety rules known from eg. @OnMethod handlers still applies except of operations on the event instance
• @JfrBlock
• delimits the code block which allows creating new event instances and executing operation on those instances
• the annotation may specify the list of event types which will be registered by BTrace (events may also be auto-registered so the usage depends on the actual event types)

Fig.1: Code Example

The custom events are to be developed uisng the standard JFR APIs and annotations and the resulting classes are to be packed in a jar file which will then be added to the bootClassPath BTrace agent argument. The events need to be added to the application bootstrap classpath in order for all possible instrumentations to have access to them (eg. instrumented java.util classes etc.). 

This is still an experimental prototype and I am looking for early testers to validate my assumptions in the wild. You can get the binaries at bintray or build BTrace from source using jfr_events branch.

Looking for feedback about the proposed notation and the expected usability.

Enjoy!

Performance impact of JFR on JDK8u

Recently JFR was integrated into JDK8u repository meaning that JFR will be available in the next public Java 8 update which happens to be JDK8u262 in July 2020 - yay!

Before the final integration some concerns were voiced regarding a performance degradation in JFR enabled builds - even when a JFR recording was not started.

This really didn't correspond to my experience so I decided to give it a quick spin with a standard benchmark suite. For the licensing terms and the ease of getting such a suite I picked SPECjvm2008 - even though rather outdated it still can provide meaningful numbers.

Setup

• dedicated c5.2xlarge AWS instance running Ubuntu 18.04
• jdk8 builds (with and without JFR) downloaded from https://builds.shipilev.net/
• SPECjvm2008 downloaded from https://www.spec.org/jvm2008/

Run

The benchmarks were run with '-i 7' argument to force 7 iterations for each particular case. This should reduce the test jitter but unfortunately it seems to be making these results 'non-compliant' but that should be fine for this quick check.

The runs (with and without JFR) were done in sequence to remove all interference and the host was fully dedicated to benchmarks.

Results

Results are, well, unsurprising. There is no statistically significant difference between the same JDK8 build with and without JFR and no JFR recording started. The overall composite scores are for all purposes equal.

In the following table you can find a more detailed breakdown of the benchmark runs - the 'diff' column shows the performance difference between the run with and without JFR where regression is indicated by a negative number.

Benchmark name	base (ops/m)	jfr (ops/m)	diff (ops/m)
compiler	619.37	633.77	14.4
compress	339.33	340.85	1.52
crypto	586.48	595.06	8.58
derby	713.85	732.22	18.37
mpegaudio	206.23	207.7	1.47
scimark.large	107.43	109.81	2.38
scimark.small	492.93	482.03	-10.9
serial	238.14	246.33	8.19
startup	50.73	51.19	0.46
sunflow	131.18	132.29	1.11
xml	811.26	814.39	3.13
composite	290.33	293.76	3.43

Addendum

In parallel to this quick SPECjvm2008 run we at DataDog ran also a bunch of more exhaustive benchmarks which happen to be internal and therefore irreproducible in public. But they all confirm the initial hunch that there would be no performance regression whatsoever for JFR enabled JDK8u - meaning that you can go and enjoy JFR on JDK8u!

[BTrace Bits: Part 1] Templated variables in BTrace

Foreword

There have been a fair number of additions to BTrace in the last year or two which, I am afraid, might have slipped the user attention as they were not given a lot of publicity at the time they were introduced.

The followup series of mini blog posts covering the potentially obscure but cool functionality available in BTrace - whimsically named 'BTrace Bits' - intends to fix it.


Ok. Let's start with something simple.

---

[1] Templated Variables

Since version 1.3.11 it has been possible to use template variables in the probe declarations. This allows delayed configuration at deployment time and makes the probe definition much more flexible.

A template variable has a value defined at deployment time and can be referenced using ant-like format - eg. ${refresh}. The variable value will be resolved as a plain string and will be the subject of any conversions necessary for the corresponding place of use.

Example: Templated uptime probe

The following probe will run the uptime check each uptime_period_ms milliseconds. 

@BTrace
public class Uptime {
    @OnTimer(from  = "${uptime_period_ms}")
    public static void f() {
        println("uptime: " + Sys.VM.vmUptime());
    }
}

The uptime check period value must be provided to BTrace agent - either as a javaagent argument or an argument to btrace dynamic attach launcher.

Attach on launch:

java -jar application.jar -javaagent:btrace-agent.jar=script=Uptime.class,uptime_period_ms=3000

Dynamic attach:

btrace <pid> Uptime.java uptime_period_ms=3000

Elements supporting templated variables

• @OnMethod
  Templating similar to what was shown in the previous example is available for all @OnMethod attributes - clazz, method, type and location whereas again the templated variables can be used in its clazz, method and type attributes in turn.
• @OnTimer
  The timer period can be defined via templated variable and is to be provided in the annotation's from attribute. The usage is shown in the previous example.

BTrace update for JPMS

BTrace 2.0.0

JPMS (Java Platform Module System) support

After long-long time BTrace finally got updated to work with JPMS. 

What does this mean? Well, now you can use BTrace for Java with JPMS (8, 9, 10, 11 and theoretically anything newer than 11, although no version after 11 was explicitly tested).

As for testing - the functional test suite is now executed for Java 8, 9 and 11 to make sure BTrace keeps on working with modules as well as with a number of internal APIs removed or made inaccessible. Java 8 testing ensures the ongoing backward compatibility without resorting to multiple agent binaries. You can see all the test rounds in the Travis CI.

This brings a slight complication to build process. BTrace now requires three environment variables defined - JAVA_8_HOME, JAVA_9_HOME and JAVA_11_HOME. They need to point to valid Java installations in order for the tests to work correctly. To make the setup easier there is `config_build.sh` script which tries automating the part of location and downloading JDK versions using SDKMAN!.

Preparation for move to OpenJDK GitHub (Skara)

In anticipation of moving the development to OpenJDK GitHub environment (project Skara, and yes, BTrace is an OpenJDK project) the sources were re-packaged not to use obsolete `com.sun.btrace` and go to `org.openjdk.btrace` instead.

For new probes you will have to use new package names but previously compiled probes should keep on working - there is a compatibility layer in place taking care of translating the package names.

Project restructuring

This item was long due. The project project was re-structured to have proper Gradle submodules and contain less 'after-build-magic'. As a bonus it will not build the DTrace lib when on supported OS (Solaris) and use a pre-built binary to include in the distribution image otherwise. Since the DTrace lib is barely changing this should be fine for most of the users.

---

Getting to BTrace 2.0.0 took some time. The release contain quite extensive changes and while it is passing all tests just fine and brief manual testing didn't show any problems either, please, keep in mind that there might be some wrinkles to iron and if you encounter any problems feel free to report them.

Flight Recorder for OpenJDK 8 is one step closer

Java Flight Recorder in OpenJDK 8 (soon?)

For last several months a lot of work was being put into back-porting fully functional Java Flight Recorder (or JFR in short) to OpenJDK 8 to bring all its goodies even to the ones not wishing to move to the latest and greatest Java.

Finally, all the changes are ready for review  and hopefully it will not take long before they are integrated to the main OpenJDK 8u tree.

Meanwhile, you can test the proposed changes using the incubator project which is fully buildable. Just make sure to run configuration script with --enable-jfr option to actually have JFR built in.

AFAIK, the --enable-jfr configuration option will be required even when the changes are merged to the main OpenJDK 8u tree - I hope all the providers of binary OpenJDK 8 distribution will include this option as default.