Finatra 2 – From Nothing to Any

DevEnglishTutorials

Hello there, today we’re going to learn how to use Twitter Finatra 2 .
This is a fully step by step tutorial (duration : 30-45 minutes), we start from having Nothing and we will end with a workable REST API behavior that can do almost Anything…
So let’s go.

Edit 2016/01/02 :
– Added github repository, https://github.com/jrevault/finatra2-startup
– Changed package names

Installation

First you’ll have to have (or install) theses software (I bet most of you already have them) :

  • A computer and a keyboard (mouse is optional but useful though)
  • JDK at least 1.8, you’ll find it there : http://www.oracle.com/technetwork/java/javase/downloads/index.html
  • SCALA: elegant functional programming language (a must learn when your ‘;‘ keyboard key is broken)
  • SBT : Scala build tools
  • A text editor, but I would recommend Jetbrain Intellij idea IDE, a nice fit for your scala projects
  • Curl command line utility… or a browser, or Postman, or anything that can call and receive HTTP requests

Once everything is installed, make sure your sbt, java and scala command line are operational:

D:\DEV>java -version
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

D:\DEV>scala -version
Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

D:\DEV>sbt sbtVersion
[info] Set current project to dev (in build file:/D:/DEV/)
[info] 0.13.8

As there are NO differences between a Linux and a windows setup, I’ll deal with a windows one, as Linux guys will catch up easily (as the opposite is not always true).

Command line setup

So let’s assume we work on D:\DEV folder

mkdir finatra2-startup
cd finatra2-startup

Once we are in the folder, let’s create the sbt built file, open your favorite editor and create a file named build.sbt, inside you’ll add explicit params (adapt with your scala version) :

name := "my-api"

version := "1.0"

scalaVersion := "2.11.7"

Quite explicit…

Now we add some code, first create the source structure, recommended default structure is explained here
Scala sources and resource will be there :

mkdir src\main\scala
mkdir src\main\resources

Finatra ‘ping / pong’

Finatra is a project based on Twitter Finagle so we need to add dependencies inside build.sbt

name := "my-api"

version := "1.0"

scalaVersion := "2.11.7"

resolvers ++= Seq(
  Resolver.sonatypeRepo("releases"),
  "Twitter Maven" at "https://maven.twttr.com"
)

lazy val versions = new {
  val finatra = "2.1.2"
}

libraryDependencies ++= Seq(
  "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra,
  "com.twitter.finatra" % "finatra-slf4j_2.11" % versions.finatra,
  "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra
)

Here we define 3 new things :

  • resolvers : repositories where sbt will find and download dependencies
  • lazy val versions : just a value that contains versions we will use (for instance only finatra but we’ll add other later
  • libraryDependencies : Dependencies we need within this project (want to know more about these % meanings ?)

Let’s create a package structure and add 2 scala sources file to have a wonderful server. First file is the server, second file is a controller that will handle http requests.
create the package structure :

mkdir src\main\scala\fr\quidquid\my_api\controllers

Inside src\main\scala\fr\quidquid\my_api, you create a file name MyAPIServer.scala and put this code inside :

package fr.quidquid.my_api

import com.twitter.finatra.http.HttpServer
import com.twitter.finatra.http.routing.HttpRouter
import fr.quidquid.my_api.controllers._

object MyAPIMain extends MyAPIServer

class MyAPIServer extends HttpServer {

  override def configureHttp(router: HttpRouter): Unit = {
    router.add[TestsController]  // We register the controllers we use
  }
}

If you want to add other controllers, syntax would be :

  override def configureHttp(router: HttpRouter): Unit = {
    router
      .add[AuthController]
      .add[PublicController]
      .add[TestsController]
  }

Then inside the controller package src\main\scala\fr\quidquid\my_api\controllers, you create a file name TestsController.scala and put this code inside :

package fr.quidquid.my_api.controllers

import javax.inject.Inject

import com.twitter.finagle.http.Request
import com.twitter.finatra.http.Controller

class TestsController extends Controller {

  get("/ping") { request: Request =>
    "pong"
  }

}

Code is quite explicit, when server receive a GET request on /ping, it returns pong.

Now we can compile the application from your project’s root folder (where the build.sbt file is), with sbt compile.
The first time you may have to download lots of dependencies, go and get a cup of tea…

D:\DEV\finatra2-startup>sbt compile
[info] Loading project definition from D:\DEV\finatra2-startup\project
[info] Set current project to my-api (in build file:/D:/DEV/finatra2-startup/)
[info] Compiling 2 Scala sources to D:\DEV\finatra2-startup\target\scala-2.11\classes...
[success] Total time: 4 s, completed 02 janv. 2016 14:40:12

Then if you have a success just run your app with sbt run (yeah this one was though!)

D:\DEV\finatra2-startup>sbt run
... BLA ..
... BLABLA ..
... BLABLABLA ..
I 0102 13:52:44.870 THREAD46: Serving admin http on 0.0.0.0/0.0.0.0:8889
I 0102 13:52:44.895 THREAD46: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135
I 0102 13:52:46.137 THREAD46: Tracer: com.twitter.finagle.zipkin.thrift.SamplingTracer

Everything seems ok.

If you have an error saying the server failed to bind the address because it’s already in use (this is my case), you can override the default port by adding command line parameters (via flags, we’ll see that later):
sbt "run -http.port=:8888 -admin.port=:8889"

Then you can test your ping pong test controller (http://127.0.0.1:8888/ping)with curl or your favorite (or hated one) navigator, or via Postman, or WTF :

D:\DEV\finatra2-startup>curl http://127.0.0.1:8888/ping
pong

Testing your API

Ok I’m gonna show you how to write some tests, create the test package inside a test folder structure :

mkdir src\test\scala
mkdir src\test\resources

Create the base package of our application:

mkdir src\test\scala\fr\quidquid\my_api

Inside src\test\scala\fr\quidquid\my_api, you create a file name TestsControllerFeatureTest.scala and put this code inside :

package fr.quidquid.my_api

import com.twitter.finagle.http.Status
import com.twitter.finatra.http.test.EmbeddedHttpServer
import com.twitter.inject.server.FeatureTest

class TestsControllerFeatureTest extends FeatureTest {
  override val server: EmbeddedHttpServer = new EmbeddedHttpServer( twitterServer = new MyAPIServer)

  "Tests - ping" in {
    server.httpGet(
      path = "/ping",
      andExpect = Status.Ok,
      withBody = "pong"
    )
  }
}

Notice that the package is the same as the server MyAPIServer.scala (otherwise you would have to simply import it)
This tests starts an embedded server, and then calls the /ping with an http get request, and expects a response status 200 (ok) with ‘pong’ as an answser..

Before testing, we need to add dependencies for the embedded server so open your build.sbt and add new version values and dependencies inside your libraryDependencies sequence :

lazy val versions = new {
  val finatra = "2.1.2"
  val logback = "1.1.3"
  val guice = "4.0"
}

libraryDependencies ++= Seq(
  "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra,
  "com.twitter.finatra" % "finatra-slf4j_2.11" % versions.finatra,
  "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra,
  "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra % "test",
  "com.twitter.inject" % "inject-server_2.11" % versions.finatra % "test",
  "com.twitter.inject" % "inject-app_2.11" % versions.finatra % "test",
  "com.twitter.inject" % "inject-core_2.11" % versions.finatra % "test",
  "com.twitter.inject" %% "inject-modules" % versions.finatra % "test",
  "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test",
  "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra % "test",
  "com.twitter.finatra" % "finatra-http_2.11" % versions.finatra % "test" classifier "tests",
  "com.twitter.inject" % "inject-server_2.11" % versions.finatra % "test" classifier "tests",
  "com.twitter.inject" % "inject-app_2.11" % versions.finatra % "test" classifier "tests",
  "com.twitter.inject" % "inject-core_2.11" % versions.finatra % "test" classifier "tests",
  "com.twitter.inject" % "inject-modules_2.11" % versions.finatra % "test" classifier "tests",
  "com.google.inject.extensions" % "guice-testlib" % versions.guice % "test" classifier "tests",
  "com.twitter.finatra" % "finatra-jackson_2.11" % versions.finatra % "test" classifier "tests",
  "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test",
  "org.specs2" %% "specs2" % "2.3.12" % "test"
)

D:\DEV\finatra2-startup>sbt test
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 13 s, completed 02 janv. 2016 15:21:55

Everything is OK 🙂

Now you can add more tests or go on with any existing test library, framework, application you’re used to, that’s your problem now…

Database connection

Well, what would be an application without a database (yeah obviously it can be a lot of things, but for the sake of this tutorial let’s assume nothing could be done without a database) ?

There is no problem when you have 0 or 1 options, problems only exists when you have several options, as problems is to choose a solution!

Ok so at this point several possibilities, ORM or no ORM, SQL or NOSql…. AHHHHHHH!
This time I’ll answer for you, let’s pick (plouf plouf) a MySQL DB along with jooq and HikariCP.
– HikariCP is “A solid high-performance JDBC connection pool”
– Jooq is an almost ORM framework

For this we are going to do 3 things :
– add the database initialization as a Module (this way you’ll know how to add a module)
– database configuration will use twitter flags,
– we’ll add a WarmupHandler to init database before the server starts allowing http requests
isn’t is wonderful!

Twitter-util flags is a nice way to deal with configuration, as we add some command line arguments to the program (like -admin.port we saw earlier)

We start by adding dependencies to our build.sbt file inside libraryDependencies sequence :

libraryDependencies ++= Seq(
 ... previous dependencies ... , // do not copy this line but last line before has to end with a comma ','
  // ***************************************
  // **** DB Stuffs
  "org.jooq" % "jooq" % "3.7.1",
  "org.jooq" % "jooq-meta" % "3.7.1",
  "org.jooq" % "jooq-codegen" % "3.7.1",
  "org.jooq" % "jooq-scala" % "3.7.1",
  "com.zaxxer" % "HikariCP" % "2.4.3",
  // **** SQL Drivers
  "com.h2database" % "h2" % "1.4.190",
  "mysql" % "mysql-connector-java" % "5.1.38",
  "org.postgresql" % "postgresql" % "9.4-1206-jdbc42",
  "net.sourceforge.jtds" % "jtds" % "1.3.1",
  "org.apache.derby" % "derby" % "10.9.1.0"
  // ***************************************

I added some typical jdbc drivers if you want to use another DB system.
Before creating the module and the warmup I suggest you a simplier approach in order to connect to your DB easely, and we will complexify right after

So create a database named quidquid_db

CREATE DATABASE quidquid_db;

create a simple (really simple) user table, enough for our purpose :

CREATE TABLE quidquid_db.users (
  id INT NOT NULL AUTO_INCREMENT,
  email VARCHAR(255) NOT NULL,
  password CHAR(42) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE INDEX (email)
);

and add some data :

INSERT INTO quidquid_db.users (email, password) VALUES ( "arthur@dent.net", "FourTy2");
INSERT INTO quidquid_db.users (email, password) VALUES ( "tricia@millan.mac", "eArTh");

Oh my god passwords are in clear!
Now directly inside the MyAPIServer we’ll add the connection, this is bad but we will remove it as soon as possible!

Add the connection params (Type of variables are preceised even if you don’t need to ex :
val rs: ResultSet = ds.getConnection.createStatement().executeQuery(sql)
could be written :
val rs = ds.getConnection.createStatement().executeQuery(sql)

  val db_props: Properties = new Properties
  db_props.setProperty("dataSourceClassName", "com.mysql.jdbc.jdbc2.optional.MysqlDataSource")
  db_props.setProperty("dataSource.serverName", "127.0.0.1")
  db_props.setProperty("dataSource.port", "3306")
  db_props.setProperty("dataSource.databaseName", "quidquid_db")
  db_props.setProperty("dataSource.user", "my_user")
  db_props.setProperty("dataSource.password", "my_secret" )

  val ds: HikariDataSource = new HikariDataSource(new HikariConfig(db_props))
  var db = DSL.using(ds, SQLDialect.MYSQL)

  val sql = "SELECT * FROM users"
  val rs: ResultSet = ds.getConnection.createStatement().executeQuery(sql)
  val result: Result[Record] = db.fetch(rs)
  println ( result )

Adapt with your own DB and needs : at least you’ll have to change the username and password.

Jooq configuration and usage see there : http://www.jooq.org/doc/3.7/manual-single-page/

So now the full MyAPIServer.scala class looks like this (with all the imports) :

package fr.quidquid.my_api

import java.sql.ResultSet
import java.util.Properties

import fr.quidquid.my_api.controllers._
import com.twitter.finatra.http.HttpServer
import com.twitter.finatra.http.routing.HttpRouter
import com.zaxxer.hikari.{HikariConfig, HikariDataSource}
import org.jooq.{Record, Result, SQLDialect}
import org.jooq.impl.DSL
import scala.collection.JavaConversions._

object MyAPIMain extends MyAPIServer

class MyAPIServer extends HttpServer {

  override def configureHttp(router: HttpRouter): Unit = {
    router.add[TestsController]
  }

  val db_props: Properties = new Properties
  db_props.setProperty("dataSourceClassName", "com.mysql.jdbc.jdbc2.optional.MysqlDataSource")
  db_props.setProperty("dataSource.serverName", "127.0.0.1")
  db_props.setProperty("dataSource.port", "3306")
  db_props.setProperty("dataSource.databaseName", "quidquid_db")
  db_props.setProperty("dataSource.user", "my_user")
  db_props.setProperty("dataSource.password", "my_secret" )

  val ds: HikariDataSource = new HikariDataSource(new HikariConfig(db_props))
  var db = DSL.using(ds, SQLDialect.MYSQL)

  val sql = "SELECT * FROM users"
  val rs: ResultSet = ds.getConnection.createStatement().executeQuery(sql)
  val result: Result[Record] = db.fetch(rs)
  println( result )
}

Just launch your server with : sbt "run -http.port=:8888 -admin.port=:8889"

and you should see this output somewhere in the logs:

If you want to retrieve row by row just replace println( result ) by :

for (r:Record <- result) {
  val id = r.getValue("id")
  val email = r.getValue("email")
  val pass = r.getValue("password")
  println("User : "  + id + ", " + email + ", " + pass + "\n")
}

In order to make this loop work import scala.collection.JavaConversions._ otherwise you’ll have an error (value filter is not a member of) during compilation
Note that HikariCP deals with the statement and connection close stuffs…

Database connection module

Now your database connection is operational, we are going to move the database initialization inside a module and take advantage of the flags mecanism.

So we create the modules package : mkdir src\main\scala\fr\quidquid\my_api\modules

And create the DBConfModule.scala source file with inside the code we move out of MyAPIServer.scala:

package fr.quidquid.my_api.modules

import java.util.Properties

import com.google.inject.Provides
import com.twitter.inject.TwitterModule

/**
  * Reads from flags.
  * Flag usage by starting server with arguments ex:
  * java -jar target/myserver-1.0.0-SNAPSHOT.jar \
  * -db.class=com.mysql.jdbc.jdbc2.optional.MysqlDataSource \
  * -db.host=127.0.0.1 -db.port=3306 \
  * -db.name=quidquid_db -db.user=my_user -db.pass=my_secret
  */
object DBConfModule extends TwitterModule {

  val ds_class = flag("db.class", "com.mysql.jdbc.jdbc2.optional.MysqlDataSource", "Mysql for dev")
  val ds_host = flag("db.host", "127.0.0.1", "Host name or IP address. (127.0.0.1)")
  val ds_port = flag("db.port", "3306", "Database port. (3306)")
  val ds_name = flag("db.name", "quidquid_db", "Database name. (quidquid_db)")
  val ds_user = flag("db.user", "my_user", "DB user name")
  val ds_pass = flag("db.pass", "my_secret", "The password")

  @Provides
  def get_props: Properties = {
    val db_props: Properties = new Properties
    db_props.setProperty("dataSourceClassName", ds_class())
    db_props.setProperty("dataSource.serverName", ds_host())
    db_props.setProperty("dataSource.port", ds_port())
    db_props.setProperty("dataSource.databaseName", ds_name())
    db_props.setProperty("dataSource.user", ds_user())
    db_props.setProperty("dataSource.password", ds_pass())
    db_props
  }
}

This object extends TwitterModule and defines values obtained via twitter flag, a flag is defined like this :

val my_val = flag( "key", "default value", "description" )

In the scaladoc you’ll find flags argument to pass to your program if you want to override params, in test, or production (db host, user / password / …) easily, example:

sbt "run -db.host=10.0.2.15 -db.name=db_prod -db.user=prod_user -db.pass=dfRT564hg!lN$12DsFr"

We could now add a convention over configuration mechanism based on a parser that takes all flags starting with ‘db.’ and putting them into the Properties for us… but that’s not the point here.

In the function get_props we just create a Properties that will be used by Jooq to instanciate the DB connection’s pool

Now we have to tell the MyAPIServer that it has to load a module, just add inside MyAPIServer.scala :

  override val modules = Seq(
    DBConfModule
  )

don’t forget to import it first : 

import fr.quidquid.my_api.modules.DBConfModule

Now we just create an object DBSupport.scala inside the fr.quidquid.my_api package (you could place it where you want) that will help us to deal with DB context :

package fr.quidquid.my_api

import java.util.Properties

import com.zaxxer.hikari.{HikariConfig, HikariDataSource}
import org.jooq.impl.DSL
import org.jooq.{DSLContext, SQLDialect}

object DBSupport {

  var db: DSLContext = _
  var ds: HikariDataSource = _

  def db_configure(db_props: Properties) {
    ds = new HikariDataSource(new HikariConfig(db_props))
    db = DSL.using(ds, SQLDialect.MYSQL)
  }
}

This object will keep for us the DSLContext variable, that we will use for every DB related requests

Previously the database was initialized directly from the MyAPIServer class. Now we will move it inside a WarmupHandler that will make the http server wait for the end of the initialization before allowing connections to controllers.
Create the class MyAPIWarmupHandler.scala and put it into the modules package.
Add this code inside :

package fr.quidquid.my_api.modules

import java.util.Properties
import javax.inject.Inject

import fr.quidquid.my_api.DBSupport
import com.twitter.finatra.utils.Handler

class MyAPIWarmupHandler @Inject()(db_props: Properties) extends Handler {

  override def handle() = {
    DBSupport.db_configure(db_props)
  }
}

The @Inject()(db_props: Properties) is just an injection of what we loaded before in the DBConfModule: the DB connection properties red from flags arguments.

Adapt the MyAPIServer class in order to add this module :

  override def warmup() {
    run[MyAPIWarmupHandler]()
  }

So the full MyAPIServer class looks like this (note that it is still small and readable) :

package fr.quidquid.my_api

import fr.quidquid.my_api.controllers._
import fr.quidquid.my_api.modules.{DBConfModule, MyAPIWarmupHandler}
import fr.twitter.finatra.http.HttpServer
import fr.twitter.finatra.http.routing.HttpRouter

object MyAPIMain extends MyAPIServer

class MyAPIServer extends HttpServer {

  override def configureHttp(router: HttpRouter): Unit = {
    router.add[TestsController]
  }

  override val modules = Seq(
    DBConfModule
  )

  override def warmup() {
    run[MyAPIWarmupHandler]()
  }
}

Now the database is initialized we just have to add a new controller path in order to list users.
Inside the controllers open the TestsController.scala and add a new function to reflect this:

package fr.quidquid.my_api.controllers

import java.sql.ResultSet

import com.twitter.finagle.http.Request
import com.twitter.finatra.http.Controller
import org.jooq.{Result, Record}
import fr.quidquid.my_api.DBSupport._

class TestsController extends Controller {

  get("/ping") { request: Request =>
    "pong"
  }

  get("/users") { request: Request =>
    val sql = "SELECT * FROM users"
    val rs: ResultSet = ds.getConnection.createStatement().executeQuery(sql)
    val result: Result[Record] = db.fetch(rs)
    result
  }
}

Start your server:

sbt "run -http.port=:8888 -admin.port=:8889"

and test your new endpoint:

Not really usable isn’t it?

Let’s make that nicer and more computer friendly. Objectives :
– Parsing the result from database
– put them inside a scala case class
– convert to json

3 really easy steps :
Create a case class named User.scala inside your controllers package. You’ll change that to any dto or vo, or pojo or poso or whateveryoucallit package later, it doesn’t matter for our purpose…

package fr.quidquid.my_api.controllers

case class User(id: Int, email: String, password:String)

This one was hard…

And just change your TestController get("/users") function to parse the request, instantiate User case class and send a json response :

get("/users") { request: Request =>
  val sql = "SELECT * FROM users"
  val rs: ResultSet = ds.getConnection.createStatement().executeQuery(sql)
  val result: Result[Record] = db.fetch(rs)
  var list = new ListBuffer[User]()
  for (r: Record <- result) {
    val id: Int = r.getValue("id").asInstanceOf[Int]
    val email: String = r.getValue("email").asInstanceOf[String]
    val pass: String = r.getValue("password").asInstanceOf[String]
    list += User(id, email, pass)
  }
  response.ok.json(list)
}

With these new imports :

import scala.collection.JavaConversions._
import scala.collection.mutable.ListBuffer

Once again, don’t forget to import scala.collection.JavaConversions._ or you’ll have compilation problems => value filter is not a member of ...

Launch your server once again 

sbt "run -http.port=:8888 -admin.port=:8889"

and call your api :

D:\DEV\finatra2-startup>curl http://127.0.0.1:8888/users
[{"id":1,"email":"arthur@dent.net","password":"FourTy2"},{"id":2,"email":"tricia@millan.mac","password":"eArTh"}]

Formatted view:

[{
  "id" : 1,
  "email" : "arthur@dent.net",
  "password" : "FourTy2"
 }, 
{
  "id" : 2,
  "email" : "tricia@millan.mac",
  "password" : "eArTh"
}]

And what about flags

Previously we used the flags system. so you can now try it easily. Try launching this one and you’l probably have HikaryCP being not very polite :

sbt "run -http.port=:8888 -admin.port=:8889 -db.host=10.0.2.15 -db.name=db_prod -db.user=prod_user -db.pass=dfRT564hg!lN$12DsFr"

the flags are an extremely simple solution to override parameters on different environments.

And what about warmup

That’s right you had to believe me when I said the HTTP server was waiting for the WarmupHandler to end before allowing external connections, but you ca test it easily:
Launch your server with wrong DB credentials or a wrong IP address and while it’s checking the DB connection (30s timeout by default), you can try to access the ping / pong function :

sbt "run -http.port=:8888 -admin.port=:8889 -db.host=10.0.2.15"

D:\DEV\finatra2-startup>curl http://127.0.0.1:8888/ping
curl: (7) Failed connect to 127.0.0.1:8888; No error

Warmup is fully operational !

Have a lovely day

Previous
Du travail d’architecte
Next
Protected: Twilio’s train app

4 Comments

Leave a comment

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.