image of the blog
ShadowThink Logo

Makefile with Automated Test

How to write Makefile with test suites?


This is one of my develop note for CRISPR-X. CRISPRs (clustered regularly interspaced palindromic repeats) are DNA sequences that many bacteria and archaea use to defend themselves. It has crazed wept across scientific community in 2013. CRISPR-X is our iGEM project, designed to find and evaluate potential sgRNA. Most importantly, CRISPR-X follows the standards of synthetic biology based on standard parts.

Basic Makefile

Considering a lot of old stuff about make online, I just want to highlight the syntax and philosophy of make.

Syntax: target: dependencies

[tab] commands

make is a rule, it has default targets. Some are file targets, and others are command targets. The idea of automated test using make is treating every test case as a command target to execute. Integrating other tools, we can do lots of awesome things, such as using diff to check program validity, using gcov to get C/C++ program test coverage and so on.

Example based on Travis CI, Coveralls


cc = g++
main = ../main
obj = main.o score.o cJSON.o region.o gene.o localresult.o util.o
x = -I ../mysql-connector/include
y = -L ../mysql-connector/lib -Wl,-dn -lmysqlclient -Wl,-dy -lm -lz -lcrypt -lpthread -ldl -lrt
util = util.h main.h cJSON/cJSON.h
CF = -c -fprofile-arcs -ftest-coverage
testcases = basic exception

all: $(obj)
	$(cc) -o $(main) $(obj) $(y) -fprofile-arcs -ftest-coverage -lgcov

	$(cc) -c cJSON/cJSON.c $(x)

util.o: util.cpp $(util)
	$(cc) $(CF) util.cpp $(x)

main.o: main.cpp $(util)
	$(cc) $(CF) main.cpp $(x)

score.o: score.cpp $(util)
	$(cc) $(CF) score.cpp $(x)

region.o: region.cpp $(util)
	$(cc) $(CF) region.cpp $(x)

gene.o: gene.cpp $(util)
	$(cc) $(CF) gene.cpp $(x)

localresult.o: localresult.cpp $(util)
	$(cc) $(CF) localresult.cpp $(x)

.PHONY: clean cleanall test $(testcases)

	rm $(obj)

	rm $(obj) ../main

test: $(testcases)

	echo "--------------test case 1 start-----------------"
	echo "basic test case, illegal call"
	../main '{"specie":"Saccharomyces-cerevisiae","location":"NC_001134-chromosome2:200..2873","pam":"NGG","rfc":"100010"}'
	echo "---------------test case 1 end------------------"

	# exception test, no NC_001134-chromosome3 record in Table_sgRNA
	echo "--------------test case 2 start-----------------"
	echo "exception test, call no-exist location"
	../main '{"specie":"Saccharomyces-cerevisiae","location":"NC_001134-chromosome3:200..2873","pam":"NGG","rfc":"100010"}'
	echo "---------------test case 2 end------------------"

In the above makefile, I use some variables to control the compile process. Here, two things need attention. Firstly, I try to use gcov, the GNU coverage evaluation tool. To make gcov works, the source file which you want to track must be compiled with flags -fprofile-arcs and -ftest-coverage. These flags tell gcc set counter in the compiled file to get the run times of every code lines. Most surprisingly, gcov returns a *.gcov file for every tracked source file which recording the details. Secondly, in makefile, we can use .PHONY the build-in target to mark those command targets. It’s useful when you have some file targets with the same name as your command targets. .PHONY avoids their name conflict.


language: cpp
services: mysql
  - gcc
  # install cpp-coveralls help submit test coverage to coveralls
  - sudo pip install cpp-coveralls
  # setup CasDB
  - mysql -e "CREATE database CasDB;" -u root
  - cd server/CasDB
  - tar -zxvf CasDB.tar.gz
  - mysql -u root CasDB < CasDB_part1.sql
  - mysql -u root CasDB < CasDB_part2.sql
  - mysql -u root CasDB < CasDB_part3.sql
  - mysql -u root CasDB < CasDB_part4.sql
  - mysql -u root CasDB < views.sql
  # setup MySQL driver
  - cd ../
  - wget
  - tar -zxvf mysql-connector-c-6.1.5-linux-glibc2.5-x86_64.tar.gz
  - mv mysql-connector-c-6.1.5-linux-glibc2.5-x86_64 mysql-connector
  # redirect to target file
  - make
  # automated test
  - make test
  - coveralls --exclude cJSON --gcov-options '../main'
  # show .gcov file
  - cat *.gcov
 - linux

You might have guessed it as the travis config is so easy. I use Travis CI mysql service, install C++ coveralls tool, setup my database and get the DB driver. It works successfully, and I got two badges now, the build passing and coverage percentage.