Creating Web Integration Tests for Spring Boot REST Controller

I’ve been working on a simple REST API for a CRUD backend application using Spring Boot. Nothing revolutionary, but it’s simply a new way of writing less code that I haven’t tried before.

Spring Boot is part of the Spring web application framework. The beauty of the Spring Boot project lies in using convention over configuration (similar to Ruby on Rails) to speed up development of web applications. It’s fast to run, you don’t need to worry about containers since the application has Tomcat embedded, and you can write a REST API in just a few lines of code with minimal configuration.

In this article, I will not go in depth into how to configure a Spring Boot application from the beginning. I will focus on creating a Controller class that will expose a REST API and how to write web integration tests to make sure it’s working as intended. I’ll be using Spring Boot version 2.0.1.

Controller Class

In my simple application, I’m just making a CRUD interface for a Fabric object. I want to expose five methods to start: listing all fabrics, creating a new fabric, retrieving a specific fabric, updating a specific fabric and deleting a specific fabric.

In Spring Boot you can create a REST Controller class by using the @RestController annotation. I also added the @RequestMapping annotation to give a prefix of “api/v1/” to all actions which I’m mapping on each method.

The next thing to do is to create our interface to a JpaRepository and use the @AutoWired annotation to inject our dependency via the constructor. 

The repository will be responsible for allowing access to the data source. To keep this post short I’ll not show how to create a data source but you can read more about it here.

Now that we can access data the first action to map is a way to list all fabrics in the database. We can do this by simply calling the findAll() method on the repository. Notice the @RequestMapping annotation to specify the request method and the last part of the URL. We’ll follow this pattern for all methods.

We can create a new Fabric object by doing a POST to the same endpoint. The method will return the object.

To get a specific object we access it by ID.

To update a specific object we retrieve it and then copy the properties of the passed in object to the retrieved one.

Finally, we also want to be able to delete a specific object. 

This concludes our Controller class and our five methods. As you can see we can implement a very simple CRUD API in just over 30 lines of code.

Web Integration Test Class

Now I want to make sure that the API is working as intended. In order to do that I can write some web integration tests.

The first thing to do is to create my class.

We use the @RunWith annotation to specify that we’ll be running this class with the Spring JUnit class runner. The @SpringBootTest explicitly sets this a test class for our application. Finally, the @DirtiesContext forces our tests to run with a clean database each time so that we don’t have to clean it ourselves.

As seen before, we can use the @Autowired annotation to inject objects. In this case, we can inject a TestRestTemplate which will allow us to make calls to our API. 

We’re also setting a few private variables that will be used in all test methods when creating a Fabric object.

Our first test is to make sure that, on an empty database, we retrieve an empty list of objects. Notice the @Test annotation to mark this method as a runnable test. We can call the getForEntity method on our TestRestTemplate object and then inspect the ResponseEntity object to make sure the Status Code is set to 200 and that the list is indeed empty.

In order to test if we can create a new object, we’ll follow a similar pattern. We create the object, pass it in our request, which this time is using the POST method, and then inspect the response to see if all our parameters are the same. Remember, our API methods always retrieve an object.

To test retrieving a specific object we’ll start by creating one and then finding it. We could have used mocking in this test but I wanted to keep these tests mock free so that the real API was always used.

Now we want to test if we can update an existing object. We’ll have to create it first, before calling our update method.

Finally, we’ll test if we can delete an existing object. 

So there you have. We’ve implemented and tested a simple CRUD interface using Spring Boot. 

United Colours of File Systems

The Problem

This weekend I needed to access some external hard drive folders on a Windows laptop. We follow a united colours approach to our systems at home, which, just like in the real world comes with a big set of advantages and occasional challenges. One of the downsides is the fact that each operating system uses different file systems.

This hard drive, in particular, had folders that had been created in an Ubuntu Linux laptop and some that have been created in a macOS laptop. They needed to be read on a Windows 10 laptop. Should this be easy? The honest answer in my experience? Most of the times, but not always…which can be abbreviated as sometimes!

Now, I’m not going to comment on Apple’s reluctance in accepting different colours and flavours of file systems and tools than their own but this is usually the biggest challenge. Over the years I found that Windows and Ubuntu play nicely with each other than macOS. Not this time though.

I mounted my hard drive on Windows and could see all my folders. but when accessing them I was confronted with two different types of issues.

Some folders were complaining that I didn’t have permission to access them. I matched all of these as folders that were created in macOS. This was a non-issue as I just had to give myself permission to access them by confirming the popup dialogue, just like in the image below.

Permission to continue for macOS file system folders
Make sure you just give yourself permission to access the folder

With the rest of the folders, I had a different issue though.

Location not available for Linux file system folders
Location not available

My folder was not available. The suggestions weren’t very helpful either. I searched online for a solution but most of them were pretty heavy-handed and involved formatting my file system, which could lead to losing my data.

The Solution

I did some online research on differences between HFS, NTFS and ext4. I already knew some things, others didn’t and yet I couldn’t find a solution that matched my case. And then, just as a lightbulb going up (or any other analogy for a Eureka moment) I remembered an almost insignificant and superficial detail.

Some of my folders were accessible after giving my user permission. Not only these folders were created on macOS but they also didn’t have one particular character in their names, “:“. Windows systems don’t allow the use of colon characters due to them serve as a drive separator, such as in “C:”. In macOS, because it’s a Unix based system you can technically use a colon in filenames but not through Finder. Finder doesn’t allow you to do this due to historical reasons.  I know most people use Finder to create folders since it’s the easiest way to drag and drop files. These folders didn’t have any colon characters because we can’t save them with them! The solution was apparent.

I plugged in my hard drive in my Linux laptop, promptly got rid of the colon characters and lo and behold it now didn’t complain at all in Windows!

I felt this was one of those moments where the solution was staring you in the face and yet you couldn’t see it for it was so simple that it just couldn’t be that. 

Hopefully, this will help anyone with the same corner-case that I faced!

Creating a municipality map of Portugal with Raphael.js

Last year I found out about a rule in the Portuguese tax code that allows local councils to give back a small percentage of the taxes to be collected back to their residents. This was seen as a measure to incentivize people to move or stay in poorer regions, which probably suffer from a shortage of jobs and services. This was a common practice in medieval times, with incentives as big as a more lax treatment in possible crimes among them1. Of course back then the risk of an Eastern invasion by a Spanish monarch was very much in the day-to-day thinking of our rulers. One can speculate that if in 2018 all the Portuguese people live by the coast is the East still Portugal? I suppose if enough Irish speakers moved there it could be claimed as part of Ireland! But I diverge.

Coming from one of the poorest regions of Portugal myself, I wondered if the “right” places were the ones indeed taking advantage of this measure and trying to attract people to live there. After a bit of research, I found that my local town didn’t use this measure but some neighbouring places did. I searched for a map so that I could visualize where this was happening and by how much (municipalities can give back up to 5% of the tax to the be collected). I didn’t find any so I thought I would create one.

GitHub project
Step 1 – Creating a municipality map in svg

This was more of a search operation than a creative one. I needed an SVG map that I could manipulate in something like JavaScript in order to load some data dynamically, such as the tax information for each council. I found one map on Wikipedia under a Creative Commons license that I could change for my purpose since it contained some errors and missed some information such as three councils. This map is not perfect because it’s missing the 30 councils that make up the archipelagos of Madeira and Açores and instead just represents the archipelago as a whole. For the purpose of this exercise, it will suffice.

Step 2 – Creating a Javascript map with Raphael.js

In order to create an interactive map, I used Raphael, a JavaScript library that allows using vector maps as JavaScript objects. Armed with the SVG data and with a way to create objects from it I started by following an online tutorial that taught me with a quick way to create a map. The hardest part was actually mapping (no pun intended) the vector data to the correct councils in an array. The SVG map didn’t have the right names, sometimes, and also didn’t have all of the councils. This left me with the time-consuming option of manually copying the data into the array.

After some time I had a map that correctly represented all of the councils with names and real borders. In order to use this in JavaScript I created a file, concelhosSVG.js , and I created the following:

The MAP_WIDTH  and MAP_HEIGHT  variables allow us to define a size to work with. The mapContainer  variable will be used in the map div in our HTML file to display the actual map. The map  variable represents the Raphael object on which we will make our manipulations. 

Finally, we define concelhosSVG which will hold an index with the name of the council and the path to append to our map variable. 

In Raphael we can use the SVG path syntax which basically allows us to start with an M  (move to) and add coordinates until we issue a Z  for ending the path.

Step 3 – Adding our map to html

Now that we have a map we can create a simple HTML page to include the map. 

I’ve gone ahead and added a simple style to our CSS file

And here’s the result

Portugal Council Map using Raphael.js
Step 4 – Adding our tax information

Now that we have a map, albeit a very ugly one, let’s add our tax information. This information is public and you can get it from the Portuguese Revenue website

We’ll create a new JavaScript file called concelhosIRS.js  and we’ll create an array inside it with all the information needed.

Step 5 – Styling our map with the tax information

Finally, we just need to syle our map based on this information. In order to do that we’ll create a new JavaScript file named main.js  and we’ll add the following code to it.

We’ll start by adding our initial style, our style when hovering the mouse over a council and the style for each of the tax brackets. I chose to create six tax brackets between 0% and 5%. The councils that don’t offer any tax benefit will not have a colour and will just show as grey.

Next, we add a function to change the styling based on the tax benefit that the council provides.

And we wrap everything together by adding our main code that initialises the map colours and adds our event listeners for when we hover our mouse over a council and move out.

Final result and notes

After a few updates to our HTML file, we end up with the following map.

Final map with colours
Final map with colours

Much nicer right?

You can find all the source code on GitHub if you want to take a look or try it out for yourself.

I’ll throw some additional notes just so you’re aware of them:

  • I have not used councils for Azores and Madeira as I couldn’t find any usable SVG for them. I also suspect they would be tiny to select, as some councils in the continent already are. The tax data for them are mapped in concelhosIRS.js though.
  • I have used global variables for the tax data and the Raphael map. I did this because it was a learning exercise for me around Raphael and SVG and not programming. Global variables are fine to use in very small projects such as this tutorial but you shouldn’t use them in any production code or larger collaborative projects. Start here and do your own research.
  • A shout out to Andy Fitch who also has a nice tutorial on this subject.

 

 

Compound interest in your career: how use the power of the environment to accelerate your professional growth

Over the last week I’ve been reading Judith Rich Harris’ The Nurture Assumption, where the author exposes the idea that children are not so much influenced by their parents as they are by their peers. Most examples Harris gives us during the book show that the environment in which children grow up is a lot more powerful in socializing them than their parents. Despite best intentions for getting children to behave in certain ways at home, the world outside and other children are the key to derive their social behaviors which might or might not impact their adult lives.

This got me thinking about a very simple observation from the workplace. Put mediocre engineers in a team of great engineers and watch them grow. Seems obvious once it’s stated like this doesn’t it? And yet how many people forget about this simple principle when applying and choosing a job?

Let me elaborate a little bit on this last rhetorical question. Imagine you’re looking for a new job and after some arduous code tests and technical interviews you finally have a couple of offers in your hand. As a software engineer you want to work with a hopefully recent and great technology (whatever that means for your particular industry and interests), get a nice salary and work on challenging products. These are the most common requirements when thinking about a job offer.

  • What technology do you guys use?
  • How much am I going to get paid?
  • What will I be working on?

These are the typical questions I hear from candidates in interviews. A surprising small number of candidates ask the question “What does your team look like and how do you work together?”.

The reason why that is surprising to me is that working in a team of great engineers who put the focus on sharing knowledge, helping others and educating the team has the same power as essentially compound interest for your career. In other words, it’s a snowball effect disguised in plain sight. Tools of Titans’ author Tim Ferriss already said that you are the average of the five people that you spend more time with. I’ll propose here that you should be using that when it’s time to choose a job.

Let’s think about this for a moment by examining each one of these questions in more detail.

If you take the job with the most recent technology does that guarantee you’ll grow your career, become smarter and get more money in the long run? Maybe. That’s the honest answer. Technologies come and go and in a software engineer’s lifetime you’ll probably end up working with dozens of programming languages, frameworks and techniques. Do you want to bet your career in Angular’s success? What about Spring? Should you be the .NET MVC guy? I’ll grant you specializing in a particular technology could allow you to surf a wave of clients and projects for as much as a decade with great return on investment, especially if you work as a consultant directly for clients, taking out agencies and other services middlemen. Bur surely you can’t be thinking in realistic terms to be using this 15 or 20 years from now and counting on it to pay your bills?

Speaking about bills, what about money? Surely you can’t be saying that I shouldn’t take the highest paying job, all other things being equal. Again, my answer will be maybe. It will depend a lot on which phase of your career you are, on your own personal goals and just responsibilities. Maybe you have a family that depends on you. Maybe you want to make a big investment in a couple of years and you need to save some cash for that. The point is, this will be a contentious answer no matter which way I put it. The reason why I’m bringing it up then is to give you an alternative view to the common position, which is just taking the highest paid offer. What about taking the offer where you’ll be working on more difficult challenges? What about taking the offer where you’ll be working with a great team? If you apply some long term thinking to this, and taking some numbers to illustrate this example, you might be refusing company’s A 50k offer to go work with company B on a 40k offer that allows you to grow at 4x the pace you would in your highest offer. Presumably you will be adding much more value in a year or two at company B that the managers will have no choice but to promote you. The worst case scenario on company A is you being tied down to a 50k+ job in 2 years time without any real options to grow from there. The worst case scenario on company B has you not being promoted by the managers, still making less money than on company A but with confidence and skills to command higher offers from other companies, should you decide to leave. Of course this is dependent on job markets, but just for the numbers themselves. Value is value, no matter what part of the globe you are and if you add value people will jump at giving you opportunities (and money to go with them).

The last of our three questions has to do with what will you be working on. This one is probably one of the most relevant for me. A good product can help you learn a lot about software architecture, design and patterns. Working with legacy code bases, while not pleasant for the most part, could provide an opportunity for relentless refactoring, assuming quality checkpoints such as automated regression testing are in place. Again, the answer here about your career growth being tied directly to the projects you work on is maybe. The project can go horribly wrong and yet you learned so much from it that you have a newfound confidence about tackling other challenges in the next project. The project can also be cruising along well and you can be working on maintenance with very little emphasis on actually improving the product. And this will have me jump straight into my last question and the one that I consider to be the most important one to ask.

Who do you think will have the better chance of becoming a great software professional in the long run, the developer who ended up working with his favorite technology, in a challenging project by himself (and making more money in the meantime) or the developer who had to learn something new from a great team that put time in to teach him and offered so much advice on how to write code, design practices and architecture that he now can teach it himself?

I truly believe it’s the second. Great teams make great software. And great teams are hard to come by, but that doesn’t mean they don’t exist and surely it doesn’t mean you can’t ask about them in a job interview. The environment around us plays a huge role in shaping everything about us. A smart coder turned loose in a team that plays along well with anybody will transform him, one can say 10x him, to use the famous 10x metaphor. Humans are a social species and as such, it’s only natural that we learn from others. I can definitely see great developers ending up in crappy (or non existent) teams, working in recent technology stacks and what it just brings them unhappiness. Unhappy developers are not productive developers. And developers that are not productive don’t learn as much, which directly translates into career stagnation.

Conclusion

Stretching to learn by yourself is a great strategy for growth but it doesn’t beat working 8 hours per day in a team that throws so much feedback and knowledge at you that by the time you realize it you are the equivalent of a veteran in your lone wolf friend’s company. Bringing it back to the idea of compound interest, your learning, your drive to excel, that is the principle. We can assume it will grow linearly throughout time. What you learn via your team, that is the interest that you get and this will grow exponentially. The key to accelerate your career growth is then, counter-intuitively, to give it to others around you and not focus on it yourself too much. Think about it the next time you’re interviewing for a job and who knows, you might just 10x yourself in a couple of years.

How to create a blog using GitHub Pages and JBake

Recently I was looking into creating a blog in order to write down some thoughts. In looking for simple ways to create one, but being a bit more technical than the average user, I started by discarding Blogger, WordPress, Medium and the like. I wanted to keep content under my control (let’s pretend for a second that GitHub is under my control) and still have a workflow that allows me to write, save and publish with simplicity and flexibility.

As a software engineer, a very simple solution that I found is using Git, which I already use on a daily basis for development work.

So without further ado let’s get started with the details of what we need to get writing and publishing.

GitHub Pages setup

The first step is creating a regular GitHub account. There is nothing fancy here and I will not go into any details on how to do this, although it should be pretty straightforward.

Once you have your repository created you should install Git locally in your machine. It’s worth mentioning that the repository name must obey the rule username.github.io where username is your username on GitHub.

After you have Git locally set up you can clone the repository by copying the clone URL of your repository GitHub page and using it on your terminal.

$ git clone https://github.com/jdmartinho/jdmartinho.github.io.git

For user repositories GitHub Pages requires that you put your site in the master branch. For project repositories, you should use the gh-pages branch. For the remainder of this article, we’ll assume we’re interested in creating a page for the user and so we’ll use the master branch.

GitHub Pages will pick up whatever content you have in the master branch and it will serve it in the **http://username.github.io** URL. This means that if you just add a simple HTML document to your repository, you instantly get a page on that URL. Despite not being extremely interesting it’s a start.

Domain setup

If you have a domain name bought and paid for you can use it to point to your new page with some more configuration steps that I’ll explain now.

You’ll need access to your hosting provider (e.g. GoDaddy) typically to your control panel like cPanel or a custom interface. Here you need to add a CNAME or A record, depending on your goal.

The first step is creating a file called CNAME and adding it to the root of our repository. On this file, the only thing we need to add is the domain name we’ll be using to access this site, for instance jdmartinho.com.

After having this file in place in the repository, we go to our hosting provider control panel and we look for the option to zone records or alias. Here we’ll add a CNAME record with jdmartinho.github.io pointing to the subdomain that we want to use like www.jdmartinho.com or blog.jdmartinho.com. If we want to use the actual top level domain we’ll need to add an A or ALIAS record for jdmartinho.com to 192.30.252.153 and another A record also for jdmartinho.com to 192.30.252.154. These IP addresses belong to GitHub and are subject to change (they have changed in the past). Typically GitHub help will contain articles on the latest IP addresses that you should use so check here for the latest news on this.

You can quickly check that the domain is correctly set up by running the dig command online here. You should see something like this:

JBake setup

JBake is a great project that takes the Jekyll static site generator and brings it to the JVM world via Java. JBake allows us to edit pages using HTML, Markdown or AsciiDoc, process these pages with templates defined using Freemarker, Groovy or Thymeleaf and style them using CSS frameworks such as Bootstrap or Foundation.

Jekyll is what GitHub Pages actually uses behind the scenes to serve the content that you are uploading to the repository. If you want to use Jekyll locally you’ll have to install Ruby and RubyGems. In order to use JBake you’ll just need Java 6+ and since I’m quite lazy and I happen to use a Java/JVM environment on a daily basis I decided to give it a try.

We’ll start by going to the JBake website and downloading the latest release. Unpack the zip to a location of your choice and add that location to your Path environment variable so that we can launch JBake from any location.

Now let’s take a look at the structure of the site with JBake. I’m borrowing the structure from the official documentation to exemplify:

On the assets directory, we have the typical static files that we’ll need to load the pages, such as CSS, JavaScript and image files. Usually, we’ll edit this folder once at the beginning and then let it be unless we’re adding a new JavaScript library or editing our CSS in order to get a new look on the site.

On the content directory is where we will keep our data that will be used by JBake to craft the site. This is where we put our HTML, Markdown or AsciiDocs, which means this is where we’ll spend most of the time after our initial setup. After all, the whole point of this is getting us to write more and spend less time fiddling with the process of publishing.

On the templates directory, we keep the template files to edit with the language of our choice. Below I have a simple example in Freemarker for the post page that comes with JBake.

We can easily create an example site by issuing the command

$jbake -i

It’s very useful to run this command the first time as it will provide us with the structure that we can then adapt to our needs. Finally when we have some content and we are ready to generate the site we can do

$jbake -b . blog

This will generate the site from the root of the repository and put it into the blog directory. Bear in mind that the blog directory will be overwritten by this. An easy way to bypass having to issue this command to generate the site with the source and output directories is to configure the default in jbake.properties such as

destination.folder=blog

Having configured this, we can just do $jbake -b from our root directory.

Workflow

Now that we have the site structure in place a typical workflow for me goes like this:

  • Pull the latest code available because I’ll use multiple laptops to work on. I’ll do this with $git pull origin master or any other branch name that you might be working on.
  • Navigate to my content directory and create a new file for a new blog post. This usually takes the form of year-month-day-post-name and I’ll usually write in Markdown which is starting to become ubiquitous.
  • Save the work on and commit it with git commit -m "My message.".
  • If it’s a work in progress then we have two choices: either I’ll push the commit to a branch different than master and later when it’s finished I’ll merge that branch or I’ll push it to master now but I’ll keep the header options of the post as status=draft. In the latter, I’ll then change the status to status=published when I’m done with the blog post so that JBake will pick it and publish it.
  • Run the $jbake -b command, pushing the content to the remote repository and see it published.

Here are the links for the official documentation. Go explore!

JBake Documentation – http://jbake.org/docs/

Freemarker Manual – http://freemarker.org/docs/index.html