Recently, I visited the Vaadin offices in Turku to co-present a webinar with Vaadin developer advocate Matti Tahvonen, just before hopping on the Vaadin Cruise from Turku to Jfokus. The latter was a lot of fun, involving presentations from Sven Ruppert, Henri Muurimaa, Holly Cummins and myself.
Anyway, this blog post is going to provide an overview of a really interesting discussion I had with Matti on our recent webinar. The Vaadin office has a super-cool setup for webinars by the way, with a green screen behind the presenters, although I was hoping for a more exciting overlay than an office (could have been a nice beach somewhere with cocktails!). You can see our setup here:
What’s it all about?
So, the Webinar was essentially themed around finding and fixing bugs as early as you can. Sounds easy? Well, they’re not your everyday bugs. Imagine a cockroach, that’s part of a gang, he always mixed with the wrong bugs, served a bit of time inside… Yeh, not the kind of bug you wanna go for a beer with. Oh, back to coding! We’re talking issues you’d normally find much later in the cycle, that can affect your ability to scale or run in a performant manner. These would normally be picked up by the more typical heavy weight profilers you’ll likely be aware of today, much later in the cycle, which results in taking more time and effort to fix (remember these bugs would have been introduced days/weeks/months/maybe even years before, if they were found by traditional profilers at prod/staging time).
So what’s different?
The magic we’re using in the session is XRebel, a lightweight Java profiler which is so lightweight and non-intrusive that it runs all the time on the developer machine. So now, these bad bugs can be picked up during the developer cycle, and critically, fixed in the same cycle. This means that these issues won’t even enter your code repo as technical debt that would require removing (at least you hope they get caught) later in your staging or even as late as production investigations. If you’ve heard of an n+1 error or have ever used hibernate, you know the kind of bugs we’re talking about (if you use hibernate and don’t know of these bugs, they’re likely slowing down your app in production right now!).
Here’s the video in it’s entirety. I walk through some of the pieces of the webinar which were particularly interesting beneath the video in text.
Our setup included a Vaadin application, deployed and running on a WildFly server, managed by NetBeans. Our server has XRebel enabled as a javaagent, and that’s it. No browser plugin, no IDE plugin, nothing – see, told you it was lightweight!
Our first application is a simple phonebook application which is rendered using Vaadin. We saw that our first request on the phonebook application reveals 64 SQL statements for a simple table. Clicking through the XRebel UI revealed a number of SQL statements which the XRebel UI kindly grouped for us and showed up two n+1 issues. Note in the image below our first SQL query returns 30 rows (n+1 – Our initial SQL stmt is the ‘1’, the 30 rows returned is the ‘n’). Next we could see a further two batches of 30 SQL queries on two tables (n, in this case is 30, we run it over 2 tables, so 60 stmts total). Hence 2n+1 here caters for 61 of the 64 statements run.
This can be fixed by creating a join query for the many to many relationships between the data in the tables that can be achieved using the Hibernate APIs or even manually creating the SQL (jOOQ users are all shouting at the screen now :)). Matti updated the code (directed by XRebel) to change the data loading from
EAGER, i.e. loading all at once, to
LAZY, i.e. only load when you need the information. This change however created an exception, which our XRebel toolbar revealed.
After some diagnosis and pointers from the exception shown to us by XRebel, Matti noticed that our entities are being populated outside of a transaction. We changed the code to lazily load the data using a more granular Hibernate call within a method running under a JTA transaction. Once fixed and rerun, we can see just 6 SQL queries executed taking just 3.1 ms. A vast improvement and code I’d much rather allow 1000 concurrent users to play with :)
The second Java EE app was a GPS waypoint tracking application. Clicking the list view, showed over 51k waypoints in a list, which although being just 1 SQL statement, takes over a second to run, just in SQL execution time. Not a great start when looking at a single user! Digging down into the SQL, shows the JPA statement return 51455 rows of results. The session data is also over 30MB in size which is not scalable and totally unacceptable!
From the info XRebel provides us with, we see that the
ListViewStrategy is being used, which is a loading strategy that loads the entire dataset into memory. During the session we make changes to our strategy, and see how it affects our runtime. We switch through several variations before opting to use the
ServiceWiringStrategy strategy, which is a lazy query fetching only the entities we require based on what’s in view in the UI at the time. It was really nice seeing the quick feedback from XRebel to see how our changes affect the runtime so quickly.
Finally after running with the
ServiceWiringStrategy, we looked through the XRebel UI to find we only had a couple of SQL queries, but one takes over 40ms whereas the other is much smaller. We copied and pasted the query which XRebel gave us into NetBeans which we can run directly on the postgres DB, but this time with the
explain keyword preceding the SQL statement. The suggested improvement we received was to index the table based on the
timestamp key. Once we added this to the backend database, and reran our test, we saw an updated time being reported by XRebel of 4.5ms – much nicer.
Ultimately, XRebel provides the ability to understand how your code is going to fare in production, while you’re developing it. This provides you with numbers that allow you to fix these bugs and tweak your codebase. Meaning, at the time you commit your code into your repo, you’re confident these bugs no longer exist, eliminating the expensive find and fix process for these types of bugs.
Want a Cool XRebel T-shirt? TRY XRebel for Free Now!