Name: Password:

December 8, 2013

Quick and easy source code metrics with unix scripting

Recently I was looking at a project and wanted to capture a number of code metrics to see how the project developed over time.

A few metrics that I was interested in were

  • source code size (lines of code and files)
  • test code size (lines of code and files)
  • number of tests

What I came up with is number of bash snippets that collected the information form me.

Bash Metrics

Given I wanted to scan the current directory . for source files matching *.java.

Number of Files

find . -type f -name "*.java" | wc -l

Lines of Code (LOC)

find . -type f -name "*.java" -exec cat {} \; | sed '/^\s*$/d' | wc -l

Excluding empty lines (sed '/^\s*$/d').

Number of Tests

find . -type f -name "*.java" -exec cat {} \; | grep @Test | wc -l

Scans for JUnit @Test annotations.

Over Time Metrics

I combined the above snippets in a small script that checks out a number of commits to collect the metrics of a project over time.

It logs commit_hash, src_files, test_files, src_loc, test_loc and num_tests as CSV into a specified file.

To use it, collect all commits you would like to analyse via git rev-list first_commit..last_commit first.

On my developer machine for around 90k lines of code project, the scanning takes aproximately five seconds per commit.

October 29, 2013

Accessing Proxy Targets in Spring

In a recent dive into spring framework proxies I was searching for a way to access real objects behind a proxy. That was required to fix field injection in jerseys jersey-spring3 extension.

It turns out that this is easily possible through the Advised interface that gets added to spring bean proxies as a management interface.

The interface provides the method getTargetSource() that allows you to access the current proxy target source.

if (bean instanceof Advised) {
  Object realTarget = ((Advised) bean).getTargetSource().getTarget();

  inject(realTarget);
}

The above code does work for static proxy targets, only. It breaks if hot swapping or pooling is used because the target obtained may be different on each access.

Your application code should not use those internal interfaces, anyway. Your libraries may.

October 21, 2013

Glassfish 4.0 / JEE 6 Encoding Recipies

Working on an internationalized web application I realized that Glassfish 4.0 is still using ISO 8859-1 encoding to serve web resources. Too bad, as UTF-8 is the de facto standard for encoding on the web.

This short post shows my recipe to make glassfish ready for non latin1 languages.

TLDR;

Add a filter with the following lines of code

request.setCharacterEncoding("UTF-8");
response.setCharacterEncoding("UTF-8");

as the first filter in your web application filter chain.

Add -Dfile.encoding=UTF8 to JVM properties in case you load resources from disk.

Fix Request Encoding

Explicitly set the encoding to UTF-8 before request body and or parameters are accessed:

request.setCharacterEncoding("UTF-8");

Alternatively, add the following line to the WEB-INF/glassfish-web.xml of a web application:

<glassfish-web-app>
  <parameter-encoding default-charset="UTF-8" />
</glassfish-web-app>

Fix response encoding

Response encoding cannot be externally configured. Instead it is always derived from current locale programmatically configured character encoding.

Explicitly set encoding before response writer is accessed:

response.setCharacterEncoding("UTF-8");

Otherwise, Glassfish will try to determine encoding from current locale which will always result in inappropriate ASCII derivates.

Fix loading of web resources

In case you are loading files from disk (i.e. from exploded web application archives), make sure to force Glassfish to load these files as UTF-8 by specifying the JVM property

-Dfile.encoding=UTF8

Read more

July 26, 2013

ngDefine: AngularJS support for RequireJS/AMD modules

Developing web applications using AngularJS is fun. Modularizing your application using RequireJS/AMD modules is nice, too unless you enjoy it to shuffle around <script /> tags everytime you include another javascript file into the application.

This post introduces ngDefine, a tiny library I have created that makes both technologies get to know each other.

Integration in a Nutshell

ngDefine integrates AngularJS and RequireJS by wrapping define and adding a little bit of AngularJS flavour to it.

Defining a module using ngDefine may look as follows:

ngDefine('app', [
  'jquery',
  'module:app.controllers:',
  'module:app.services:',
  'module:app.directives',
  './local'
], function(module, $) {
  
  // module = { name: 'app', .. }
  
  module.config(function($routeProvider) {
    $routeProvider.when( ... );
  });
})

It calls the function ngDefine with the name of the AngularJS module, a list of dependencies and a factory function. Dependencies may contain references to AngularJS modules, denoted by module:{moduleName}. Those get translated to actual file resources before the whole list of dependencies is resolved via RequireJS. ngDefine wires together the Angular module and passes it into the factory function along with all resolved AMD dependencies.

Features

The ngDefine integration offers a number of benefits over plain AngularJS applications:

  • Improved modularization: AngularJS modules may be packaged into reusable AMD modules.
  • Integration of external libraries: Non-AngularJS dependencies may easily be declared via AMD dependencies.
  • Minimal RequireJS boilerplate: The amount of RequireJS boilerplate in the application code is kept at a minimum (only the ngDefine wrapper is needed).
  • Minification: The integration can be minified. After minification module definitions are stripped down to plain AngularJS/RequireJS code after minification.

In addition it helps to avoid common pitfalls in AngularJS applications such as unintended module re-definition.

Additional Resources

If you would like to learn more, check out the project website or its home on GitHub.

Feel free to give feedback on the project and report bugs on the issue tracker.

June 13, 2013

Contributing to GitHub hosted Open Source Projects through Pull Requests: A Checklist

So there is an open source project on GitHub. And you forked it already. And there is this bug XYZ you would like to fix, throught a pull request. Here is a short checklist how you can do it.

Use Feature Branches

Get ready to work with feature branches. Those isolate the work you do for a particicular pull request from all the other things you want to contribute.

Create a branch for the issue XYZ where you will implement the bug fix.

git checkout master        # starting point is your master
git checkout -b XYZ        # create and checkout branch XYZ
                           # you are going to work on

Do non-stop coding. All day all night. Mind coding styles of the project you are working on. Use the right tools.

At some point you are done.

git log                    # check what you have done

Cleanup using git rebase

You may have produced several commits. And there may be typos in your commits. Or you commited try to fix bug (1..n). Squash (i.e. combine) all of your commits and give it a nice commit message. This makes it easier for reviewers to see the actual changes and to eventually merge the pull request.

Your tool for the job is interactive rebase. It allows you to reword, remove, combine and alter commits.

git rebase -i HEAD~n	     # n=number of commits from HEAD
                             # you created

This pops up an interactive console

pick 906eb32 yea, nearly done
pick 8a45bb9 ok, late night trying again
pick b86deaf got ya

Combine all of your commits into one by assigning s or squash as the rebase operation.

pick 906eb32 yea, nearly done
s 8a45bb9 ok, late night trying again
s b86deaf got ya

Finish the job, i.e. press [ESC], [:wq] to write the changes and close the editor.

You will be asked to write a commit message that describes everything you did. Make it a good one (check your projects contribution guidelines, too).

Again, finish the job with [ESC], [:wq].

You are done and your changes are ready to be published!

Push, Pull Request, restart

Push the stuff to your remote:

git push origin XYZ

If you have pushed parts of the work before, you may need to override these changes by applying force (-f):

git push -f origin XYZ      # overrides history on the server
                            # branch so take care what you do

Create a pull request from your origin/XYZ branch to the OSS projects development branch.

Switch back to your master branch

git checkout master

And create a new feature branch for this AABB issue you wanted to fix for such a long time already.

git checkout -b AABB

There is more

There exist many more resources worth checking out on the topic:

March 24, 2013

Camunda BPMN 2.0 Modeler Introduction

Do you want to create and refactor BPMN 2.0 without pain?

Check out the introduction on the Camunda Modeler, a tool I have mainly been working on in the last month. The modeler is open source as part of our camunda BPM platform.

Can be downloaded, too.

March 24, 2013

Using STFU powers to deal with Ruby version upgrades

... or how to spend unneccessary time on infrastructure problems.

Ruby is a great language and offers many tools to quickly develop (web) applications and awesome libraries. But sometimes it sucks. And that is the case when migrating applications to new Ruby versions.

The last weekends I worked on switching my Ruby on Rails 2.x applications from Ruby 1.8 to Ruby 1.9. One thing I learned is that Ruby 1.9 uses UTF-8 for file encoding now. Rails 2.3 applications and libraries (e.g. the mysql gem) have a number of issues with that, different ones depending on the platform you are running ruby on. This is a non-complete list of things to care about when updating.

Encoding update

To start with, Ruby uses the LANG environment variable to determine file encoding. So make sure you have it set to a UTF-8 locale on your system.

 irb(main):001:0> ENV['LANG']
 => "en_US.UTF-8"

Next, make sure that you do not use any special NON-ASCII characters (åößè are good candidates) in mail templates because the action-mailer gem for Rails 2.3 does not quite seem to understand UTF-8 yet.

Also, make sure your source files are UTF-8 encoded (adding the # encoding: utf-8 header may be required, too). And then, instruct your rails application to use UTF-8 as the default encoding:

# add to top of config/environment.rb
Encoding.default_external = Encoding.default_internal = Encoding::UTF_8

If you use regular expressions, do not wonder why they don't match anymore. They just don't because you may not have solved all encoding problems yet.

After encoding

You are done with encoding problems now and want to use all the latest UTF-8 compatible gems now? Reconsider. Then make sure to stick to the 1.x version of rubygems. Because rails 2.3 does not like the 2.0 versions of the gem.

Check your database connectivity library (I used the mysql gem for Ruby 1.8). The mysql gem has encoding issues on Linux systems, so switching to the mysql2 gem is inevitable. Only versions < 0.3 work though. Just because. If you are on windows stick to the mysql gem for now. Because mysql2 gives Bad file descriptor errors.

Remove deprecation warnings

Use Object#tap instead of #returning. Looks awful in code but prevents your server log from getting spammed.

Think about monkey patching your Rails 2 app for some more UTF-8 compatibility or switch directly to Rails 3.

Finally

Drink a coffee, eat a muffin and read about others going through the same kind of stuff, some even spending month to do it.

September 24, 2012

Master thesis finished

Finished my master thesis entitled Exploring Run-time Behavior in Reversible Experiments - An Application of Worlds for Debugging. It is about painting walls and refrigerators. From the abstract:

Debuggers are often used to inspect running software systems in order to support the understanding of source code. They differ substantially from most tools that visualize run-time data as they make it possible to keep track of a running program as it executes. Thereby they encourage learning through experience. At the same time, conventional debuggers do not properly support incremental understanding. The reason is that they offer only limited options to safely re-explore a particular run-time behavior without restarting the program under observation.

Against that background, this work examines Worlds [44]—a language extension that allows for scoping of side effects in imperative programming languages. The thesis evaluates the concept of Worlds to isolate side effects caused during debugging. By doing so, the effects of prior explorations can be discarded and the examination of run-time behavior gets safe to be repeated until it yields the desired knowledge gain.

In the course of this work, we present the principle application of Worlds to enable debugging in reversible experiments. Furthermore, we propose a general purpose Worlds mechanism for Squeak/Smalltalk, which is necessary to evaluate the practical application of the concept for Smalltalk. In the course of the evaluation, we present a debugger that employs the mechanism to realize the safe re-examination of behavior inside a debugging session.

September 5, 2011

A jQuery plugin for bootstrap modal boxes

Bootstrap is a CSS toolkit made by two guys at Twitter which kick-starts website development.

Along other gimmicks it offers modal boxes for which I have created a jQuery plugin, Dialog Two.

Check it out.

Fork me on GitHub

July 5, 2011

Spring might say uuuh if you use JPA and throw Exceptions in nested @Transactional contexts

In my Jersey + Spring + JPA stack I recently came across an interesting issue: Throwing an exception inside a @Transactional annotated Spring bean method would cause a org.springframework.transaction.UnexpectedRollbackException in the outer container. The result was that the request ended screwed up and the following stack trace appearing on my application server logs:

org.springframework.transaction.UnexpectedRollbackException: JPA transaction unexpectedly rolled back (maybe marked rollback-only after a failed operation); nested exception is javax.persistence.RollbackException: Error while commiting the transaction
Caused by: 
  javax.persistence.RollbackException: Error while commiting the transaction
  ...

After doing some research, it got apparent that, while there were many varieties, many people had this issue. Some said this was due to limitation in JPA's exception hierarchy, others just claimed it is a Spring or Hibernate or whatever bug.

Wondering, why everyone pointed on the issue, no solved it I created a small test case to investigate what is actually happening.

Test Setup

At first, I got the NewsManager which does some database action when asked to #aggregateNews. If it finds an error, it will throw an exception which itself makes the changes rolled back (pretty clear so far).

@Component
public NewsManager {

  @Transactional
  public News aggregateNews(Date date) {
    // Does some random action here
    // e.g. updating stuff in the database
    
    // May throw a (unchecked) operation failed 
    // exception on error, rolling back the transaction
    if (error) {
      throw new OperationFailedException(
                    "Could not aggregate due to error");
    }
  }
}

The root controller of my Jersey application (a Spring bean, too) will ask the NewsManager to aggregate news and will catch the expected exception in case one is thrown. The reason for choosing this two-layer approach is that roll-backs can cleanly be done without propagating an exception to the presentational layer.

@Path("/")
@Component
public RootController {

  @Inject
  private NewsManager newsManager; // Spring bean
   
  @GET
  @Transactional
  public Object news() {
    try {
      // Call Spring bean
      return newsManager.aggregateNews(new Date()); 
    } catch (OperationFailedException e) {
      return null;
    }
  }
}

What happens when a OperationFailedException is thrown is that this exception will be wrapped nicely with a javax.persistence.RollbackException and then with a org.springframework.transaction.UnexpectedRollbackException and so forth. Your application is not be able to handle it and you read the above mentioned stack trace in your server logs.

Problem deduction

The actual cause of the problem is not the badness of the exception thrown, but the fact that the previous example used nested @Transactional contexts to handle the business logic. Both, the controller and the news bean had @Transactional annotations specified on their business methods. Thus they were instrumented by Spring to use transactional logic.

@Path("/")
@Component
public RootController {
  
  @GET
  public Object news() {
    try {
      // Call Spring bean
      return newsManager.aggregateNews(new Date()); 
    } catch (OperationFailedException e) {
      return null;
    }
  }
}

If you simply remove the outer @Transactional annotation (which might be unnecessary anyway) everything works fine.

Conclusion

I am not convinced, that the above fix may help everyone posting about UnexpectedRollbackException and Spring + JPA. Neither am I hundred percent sure why this happens. What I got is that deliberately throwing exceptions inside nested transactional contexts may not be good for your Spring + JPA applications. Too many miracles happen there, eventually screwing things up and stealing one day of your time to figure out what and why.

About

I'm Nico Rehwaldt, a software engineer from Berlin (Germany). On this website I talk about issues of the web, open source and my own projects.

Archive

2010

2011

2012

2013