[BTrace: update from trenches] - Unattended execution

A bit of history

BTrace origins go back more than 10 years and it shows that the modus operandi at that time was to have one JVM at a time and do all the experiments and debugging on that single JVM.
The standard workflow for BTrace dynamic attach was decided to be:

1. Identify the JVM to attach to
2. Attach to that JVM and deploy the probe(s)
3. Stay connected until you are satisfied with the results

While probably quite ok for the one JVM situation it becomes quite a hindrance when trying to operate in ad-hoc mode for bunch of JVMs running on several hosts at once (yes, k8s, talking about you). The lack of unattended execution basically makes BTrace unusable in modern environments - if dynamic attach is required.

Time to fix

So, after a period of procrastination I finally decided to add unattended mode of execution to BTrace. It was both easy and hard task at the same time - the binary I/O protocol BTrace is using is very easy to extend but the the underlying management of client 'sessions' had to be refactored slightly to allow disconnecting and reconnecting to a session without killing it. But nothing that could not have been done during a particularly grey and rainy COVID lockdown weekend.

So, here we go with a few improvements to the BTrace client which should make it much easier to use BTrace in fire&forget mode - which, I think, will become more and more popular with the JFR support when one can easily define a dynamic JFR event type and deploy a probe to generate that event and leave the probe running, turning on and of the JFR recording as necessary.

• 'Detach client' client command
  In addition to 'Exit' option in the BTrace CLI it is now possible to detach from the running session.
  Upon detaching a unique probe ID is displayed which can be used to later reconnect to that probe.
  
  

• 'List probes' client command
  This command will list any probes which were deployed and the clients left them detached.
  
  
  

• List probes from command line
  Use btrace -lp <pid> to list the probes in detached mode in a particular JVM
  
  
  
  
  
  
  
  
• Reconnect to a detached probe
  Use btrace -r <probe id> <pid> to reconnect to a detached probe and start receiving probe data.
  The detached probes are maintaining a circular buffer for the latest data so you can get a bit of history after reconnecting as well.
• Attach a probe and disconnect immediately
  Useful shortcut for scripting BTrace deployments when the probe is deployed and the client disconnects immediately.
  Use btrace <btrace options> -x <pid> <probe file> to run in this mode. 
  Upon disconnecting the probe ID is printed out so it can be eg. processed and stored.

New possibilities

Having implemented the support for listing the detached probes and reconnecting to them as a form of command line switches opened doors to easy scripting when one can write a quick one-liner to attach to a named probe:
./bin/btrace -r $(./bin/btrace -lp anagrams.jar | fgrep AllMethods1 | cut -f2 -d' ') anagrams.jar

The unattended execution support was checked in and a development build binaries are available at https://github.com/btraceio/btrace/actions/runs/394037357

[BTrace: update from trenches] - Experimental support for emitting JFR events, take two

In the previous post I have introduced the prototype of JFR support in BTrace.

The first attempt, however, was riddled with serious shortcomings - the events had to be defined externally and then added to boot classpath.
In addition to that the BTrace verifier had to be made more permeable to allow calls to JFR APIs - this caused the verifier complexity to increase significantly, opening potential holes to be exploited.

Fortunately, there is a very cool API directly in JDK which allows creating JFR event types dynamically, therefore removing the requirement to have the events defined beforehand and added to boot classpath. As an added benefit the refactoring allowed the use of the standard BTraceUtils accessor class for operating on JFR events, thus removing all the custom 'holes' in the BTrace verifier.

The implementation is available on GitHub in jfr_events branch. It is fairly complete (at least for the use cases I was able to come up with) but as usual a user input is more than welcome.

The code example showing the intended usage follows.

Fig.1: Code Example

[BTrace: update from trenches] - Experimental support for emitting JFR events

Java Flight Recorder (JFR) is an amazing piece of technology allowing collection of a huge amount of very detailed data points from the running application and (also) Java runtime.
It has been widely available since Java 9 and recently it has been backported to JDK 8 update 265 - thus covering all Java version currently available (intentionally disregarding JDK 7 which I really hope will gracefully fade away very soon).

Having a standardized, low impact way to collect data from a running Java application is something BTrace can hugely benefit from.
Among other things this will allow seamless integration with tools already using JFR as their native format (JDK Mission Control or, recently, also VisualVM) creating synergy between 'free-form' instrumentation and well established perf analysis tools.

Although the implementation might seem quite trivial at the first look it quickly becomes more involved because the BTrace safety guards needs to be modified and extended to allow easy cooperation with JFR events while not compromising the security guarantees. Also, it is imperative that working JFR events does not need introducing and learning any new concepts - everything should be expressible via annotations and plain Java code. 

After pondering all the requirements for a while I came up with the idea to split periodic and non-periodic event usage. This allows a neat registration of periodic event handlers while keeping the simple event handling code, well, simple.

Here are the proposed annotations:

• @JfrPeriodicEventHandler
• used to define the JFR related code which will be run at the beginning/end of a recording chunk or at a given time interval
  
• the periodic event is passed in the handler as a method parameter by BTrace (the handler method must have exactly one argument which must be a jdk.jfr.Event subclass)
• all safety rules known from eg. @OnMethod handlers still applies except of operations on the event instance
• @JfrBlock
• delimits the code block which allows creating new event instances and executing operation on those instances
• the annotation may specify the list of event types which will be registered by BTrace (events may also be auto-registered so the usage depends on the actual event types)

Fig.1: Code Example

The custom events are to be developed uisng the standard JFR APIs and annotations and the resulting classes are to be packed in a jar file which will then be added to the bootClassPath BTrace agent argument. The events need to be added to the application bootstrap classpath in order for all possible instrumentations to have access to them (eg. instrumented java.util classes etc.). 

This is still an experimental prototype and I am looking for early testers to validate my assumptions in the wild. You can get the binaries at bintray or build BTrace from source using jfr_events branch.

Looking for feedback about the proposed notation and the expected usability.

Enjoy!