GraalVM is a universal virtual machine for running applications written in JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, Clojure, and LLVM-based languages such as C and C++.

GraalVM removes the isolation between programming languages and enables interoperability in a shared runtime.

In lay man's terms, Graal VM allows languages JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, Clojure to talk to each other, thus removing isolation between these languages.

But, as a Java/JVM platform developer, I am more interested in Graal VM's ability to ahead-of-time compile Java to run a Java application as a native application.

GraalVM Native Image allows you to ahead-of-time compile Java code to a standalone executable, called a native image.

native-image is a utility that processes all the classes of your application and their dependencies, including those from the JDK. It statically analyses these classes to determine which classes and methods are reachable and used during application execution. Then it passes all this reachable code as the input to the GraalVM compiler which ahead-of-time compiles it to the native binary.

The key benefits of AOT compiling Java application to a native binary are:

1. Speed of startup and
2. Reduced memory footprint

For example here's a micronaut application: https://github.com/RaviH/graal-micronaut

When I try to run the jar, it takes about 1.3 secs to start:

 $ java -jar target/graal-micronaut-0.1.jar
 18:30:22.244 [main] INFO  io.micronaut.runtime.Micronaut - Startup completed in 1320ms. Server Running: http://localhost:8080

But, when I create the native image, it starts in 29 ms (vs 1.3 s) – 45 times faster boot time.

# rhasija ~/dev/projects/tmp/graal-micronaut on git:master x  C:130
$ native-image -jar target/graal-micronaut-0.1.jar
Build on Server(pid: 5608, port: 50229)
[graal-micronaut:5608]    classlist:   3,788.29 ms
[graal-micronaut:5608]        (cap):   1,547.58 ms
[graal-micronaut:5608]        setup:   2,041.59 ms
[graal-micronaut:5608]   (typeflow):  27,169.03 ms
[graal-micronaut:5608]    (objects):  32,584.30 ms
[graal-micronaut:5608]   (features):   1,562.28 ms
[graal-micronaut:5608]     analysis:  63,945.36 ms
[graal-micronaut:5608]     universe:   1,593.43 ms
[graal-micronaut:5608]      (parse):   4,368.97 ms
[graal-micronaut:5608]     (inline):   9,400.59 ms
[graal-micronaut:5608]    (compile):  36,820.75 ms
[graal-micronaut:5608]      compile:  53,848.37 ms
[graal-micronaut:5608]        image:   4,955.53 ms
[graal-micronaut:5608]        write:   1,971.52 ms
[graal-micronaut:5608]      [total]: 132,379.50 ms

# rhasija ~/dev/projects/tmp/graal-micronaut on git:master x
$ ./graal-micronaut
18:36:13.776 [main] INFO  io.micronaut.runtime.Micronaut - Startup completed in 29ms. Server Running: http://localhost:8080

Also, the native image took 14M of memory (vs  387M Java based app) – roughly 3% of the memory taken by Java app. This is awesome.

Micronaut Java app:
13068 java 0.1 00:03.00 23 1 84 387M 0B 0B 13068 5096

Native image:
13136 graal-micron 0.0 00:00.03 3 1 33 14M 0B 0B 13136 5096

Below are the performance metrics for the endpoint:

For Graal VM native binary:

$ bombardier -n 100000 -c 100 "http://localhost:8080/simple"
Bombarding http://localhost:8080/simple with 100000 request(s) using 100 connection(s)
 100000 / 100000 [=========================================================] 100.00% 18s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      5308.20    1376.18    9776.58
  Latency       18.70ms    23.93ms   305.78ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    16.31MB/s%


$ bombardier -n 100000 -c 100 "http://localhost:8080/simple"
Bombarding http://localhost:8080/simple with 100000 request(s) using 100 connection(s)
 100000 / 100000 [=============================================================================================] 100.00% 19s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      5164.61    1269.64   11265.72
  Latency       19.22ms    19.82ms   249.97ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    15.87MB/s%


$ bombardier -n 100000 -c 100 "http://localhost:8080/simple"
Bombarding http://localhost:8080/simple with 100000 request(s) using 100 connection(s)
 100000 / 100000 [=============================================================================================] 100.00% 19s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      5041.83    1283.07   10297.43
  Latency       19.66ms    22.36ms   310.20ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    15.51MB/s%  

For Java application:

$ bombardier -n 100000 -c 100 "http://localhost:8080/simple"
Bombarding http://localhost:8080/simple with 100000 request(s) using 100 connection(s)
 100000 / 100000 [=============================================================================================] 100.00% 17s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      5629.80    1057.21   13696.04
  Latency       17.70ms     8.91ms   182.28ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    17.15MB/s%

  $ bombardier -n 100000 -c 100 "http://localhost:8080/simple"
Bombarding http://localhost:8080/simple with 100000 request(s) using 100 connection(s)
 100000 / 100000 [=============================================================================================] 100.00% 14s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      7034.21     592.21   15347.18
  Latency       14.22ms     2.91ms    65.69ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    21.45MB/s%


  # rhasija ~/dev/projects/tmp/graal-micronaut on git:master x
$ bombardier -n 100000 -c 100 "http://localhost:8080/simple"
Bombarding http://localhost:8080/simple with 100000 request(s) using 100 connection(s)
 100000 / 100000 [=============================================================================================] 100.00% 14s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec      7060.21    2076.69   61322.98
  Latency       14.31ms     2.79ms    57.63ms
  HTTP codes:
    1xx - 0, 2xx - 100000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    21.31MB/s%

The java application performed consistently better.

Details:

$ mvn -v
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T11:41:47-07:00)
Maven home: /Users/rhasija/.sdkman/candidates/maven/current
Java version: 1.8.0_202, vendor: Oracle Corporation, runtime: /Users/rhasija/.sdkman/candidates/java/1.0.0-rc-16-grl/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"

To get more information on it I asked the GraalVM folks and Oleg Selajev got back with this information:

native image does offer superior startup / memory overhead doesn't it? In general peak performance will benefit more from the powerful JIT compiler, so it's expected that running for a longer period of time or under a heavy load will perform better in the jvm mode.

also, if you use GraalVM Enterprise, you can provide --pgo-instrument to build an instrumented binary, and use a profile from that to build (with --pgo) a better performing native image.

IMO:

I could be wrong but this is what I have learnt: GraalVM is awesome for quick start applications / lamdas that will do a job and be shutdown right after or for applications where performance is a non-issue. But for a long running applications that are supposed to be performant under load, java application (because of JIT compiler) are still better. This might change in future, but for now this seems to be the case.

Thoughts/feedback?

With the latest release of micronaut v1.0.0, it matches Spring without any changes to bombardier i.e. it's keeping the connection alive by default now.

You can find the latest test output here

Summary:

micronaut with 1.0.0 release beats Spring in startup time and performs same or slightly better in run time with a very simplistic one database entity based test. I would expect micronaut to perform better with it's strategy of providing implementations at compile time vs Spring's run time. I will perform a test with more number of enitites and with GETs and PUTs and provide a link to that article here.

Update 10/23/2018 8:40 PDT:

With the keep-alive connection header while testing with bombardier, micronaut provides the same results as Spring. So the updated comparison table is:

You can find the result output for all of the run(s) here: https://github.com/RaviH/spring-data-micronaut-gorm/tree/master/performance_run_output

If you are interested in latest run with bombardier with keep-alive that output is here

All the result outputs are here

Update 10/23/2018 7:38 PDT:

I have been working with Micronauts team and they have quickly found the reason why bombardier tests provide such different results. bombardier closes the connection after every request and hence the huge discrepancy in the result. To address this Micronaut 1.0.GA would release with Http 1.1 default of keeping the connection open. I will be updating this article as soon as I can get hands on it. I will be updating the article pretty soon here with the results of provoding Connection:keep-alive header with bombardier on Micronaut 1.0.0.RC3

Recently, I came across Micronaut framework and found it to be very interesting. It provides low memory footprint and fast bootup time. I have used Micronaut now for couple non-database backed applications and it starts up really fast and is performing really well. In my performance tests comparing Micronaut app to Spring app, Micronaut performed better (will be covered in a separate post).

The next step for me was to see if I could recommend Micronaut for database backed applications. Micronaut works with straight up hibernate and even provides GORM (a Spring Data alternative). I am already familiar with Spring Data so I am curious about GORM. I decided to give it a spin and see how it holds up compared to Spring Data app.

You can find my repository here: https://github.com/RaviH/spring-data-micronaut-gorm

Details about the test:

a. This is a very basic application with a single domain object: Person.
b. 3 records were inserted into an in-memory H2 database at startup time.
c. Only GETs are tested
d. I haven't tweaked any of the apps for better performance i.e. this is as out of the box as we can get.
e. Spring Boot v2.0.6 is used Micronaut: 1.0.0.RC3 is used.
f. OS: MacOS High Sierra, 16 GB RAM, 2.5 Ghz Intel Core I7
g. mvn and java version:

$ mvn -version
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T12:39:06-07:00)
Maven home: /usr/local/Cellar/maven/3.5.0/libexec
Java version: 1.8.0_162, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"

i. Used different tools (bombardier, ab, loadtest for performance tests since each one of them offers different results.

Results

Tabular comparison: Total time to complete requests = total time

Summary:

• Micronaut has a much faster startup time than Spring.
• With bombardier: Spring data app blows Micronaut app out of the water as far as throughput is concerned.
• With ab: micronaut wins by a slight margin.
• With loadtest: it's too small a difference to decide.

Next Steps
This is probably going to be an in progress article. I will add more information as I gain it.

a. Try a different load test tool.
b. Try loading these applications on a container to take my machine out of the picture.
c. Try a more real world scenario i.e. more entities and real relationships between them, constraints on domain classes.

Please feel free to share your thoughts/comments on the article. Try this out on your machine and feel free to share your results.

Just be nice ;-)

I want to create 2 databases inside one MySQL container and have the user of first database full access to 2nd database. With official mysql image one can easily create a database, and allow a user access to that database. But, creating a 2nd database is not easily provisioned.

Solution:

Docker images work on the concept of layers. Each new command so to speak creates a new layer and here in lies our solution.

Simply create a SQL file with following commands:

# Create second_db database if it doesn't exist
CREATE DATABASE IF NOT EXISTS second_db;
# Grant all privilidges on second_db to org_user
GRANT ALL PRIVILEGES ON second_db.* TO 'org_user' identified by 'org_user_password';

Let's say you created a file called create_second_db.sql in your project folder (or wherever you really want it to be). Than you can mount this file into docker /docker-entrypoint-initdb.d folder. Files in this folder are loaded in alphabetical order at startup time under root user.

Below is the corresponding Dockerfile (that would mount the create_second_db.sql into the /docker-entrypoint-initdb.d) folder:

# Use base mysql image with tag 5.7
FROM mysql:5.7
# Copy our custom SQL file to /docker-entrypoint-initdb.d folder
COPY ./create_second_db.sql /docker-entrypoint-initdb.d/create_second_db.sql

NOTE: You can mount as many SQL files as you would like into that folder.

Once you have created the image (via docker build), than when you start the container, that container would have 2 databases created.

Build the image

docker build -t custom_mysql -f Dockerfile .</code>

Start the container:

docker run --name custom_mysql \
-e MYSQL_ROOT_PASSWORD=my-secret-pw \
-e MYSQL_USER=org_user \
-e MYSQL_PASSWORD=org_user_password \
-e MYSQL_DATABASE=first_db \
-d custom_mysql

Result:

# rhasija ~/tmp/docker
$ docker exec -it 0fdbf85f6218 /bin/bash
root@0fdbf85f6218:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.22 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| first_db           |
| mysql              |
| performance_schema |
| second_db          |
| sys                |
+--------------------+
6 rows in set (0.00 sec)

mysql>

Notice first_db and second_db above.

Boom! Job done. Let's open champagne! ;-)

Comments / questions welcome!

Link to this article published on DZone

• For the first x calls respond with y
• Then for x+1 respond with z

Wiremock by itself doesn't provide a direct way to to stub different responses akin to Mockito's thenReturn or Spock's multiple stubs. Wiremock however provides a ResponseTransformer. I used Wiremock's ResponseTransformer to store state, and return y by default but return z on x+1 try.

  static class SuccessfulAtLastTransformer extends ` {

        int count = 1

        SuccessfulAtLastTransformer() {}

        @Override
        ResponseDefinition transform(
                final Request request, final ResponseDefinition responseDefinition, final FileSource fileSource) {

            if (request.getUrl().contains("/customer/cust1234")) {
                if (count == 3) {
                    def responseJson = new JsonSlurper().parseText(responseDefinition.body)
                    // Set the customerId on the final try
                    responseJson.customerCar = "toyota"
                    def newRequestBody = JsonOutput.toJson(responseJson)
                    return new ResponseDefinitionBuilder()
                            .withStatus(200)
                            .withBody(newRequestBody)
                            .build()
                }

                return new ResponseDefinitionBuilder()
                        .withHeader("COUNT", count++ as String)
                        .withStatus(200)
                        .withBody(responseDefinition.body)
                        .build()
            }

            return responseDefinition
        }

        @Override
        boolean applyGlobally() {
            return false
        }

        @Override
        String name() {
            return "SuccessfulAtLastTransformer"
        }
    }

Line #1: extend import com.github.tomakehurst.wiremock.extension.ResponseTransformer

Line #8: override the transform function.

Line #11: Check if this is the api you are interested in.

Line #12: Check if the count has reached a certain threshold and if so, then add customerCar property to response JSON

Line #23: Add count to the response header. That acts as our state storage. Once the count will reach x in this case 3, then change the response (line #13).

Line #34: Set apply globally false i.e. don't use this ResponseTransformer as an interceptor anywhere else but where I put it.

Line #40: Give your transformer a name.

Once you have defined a ResponseTransformer like above, you can then attach it to wiremock instance like so:

@Shared
def customerServiceMock = 
      wireMockConfig()
     .port(8090)
     .extensions(SuccessfulAtLastTransformer)

Done.

Please let me know if you have any questions or if there's a better way to do the same thing in wiremock via comments on this post.

fun String.addOne() = this + "1"

And then:
println("Boba".addOne()) leads to an output of: Boba1

So above we have added a new function to String class.

Generic Extension Functions

With the basic introduction out of the way, I wanted to write a Generic extension function. We use Netflix OSS at work and are constantly getting property values via code that looks like this:

DynamicPropertyFactory.getInstance().getStringProperty("SOME_CONFIG_PROP_NAME", "I am a good default value").get()

Basically, above code allows us to fetch (dynamically) values of a property: SOME_CONFIG_PROP_NAME in this example, and provide default value I am a good default value in this case.

We have several variations of this, for example:

DynamicPropertyFactory.getInstance().getIntProperty
DynamicPropertyFactory.getInstance().getFloatProperty
DynamicPropertyFactory.getInstance().getDoubleProperty
DynamicPropertyFactory.getInstance().getLongProperty
DynamicPropertyFactory.getInstance().getBooleanProperty

So on and so forth. I wanted to have a generic extension function available on String class because property names are always String type. Here's the generic function:

fun  String.getConfig(default: T): T {
    val dynamicPropertyFactory = DynamicPropertyFactory.getInstance()
    return when (default) {
        is String  -> dynamicPropertyFactory.getStringProperty(this, default).get()
        is Int     -> dynamicPropertyFactory.getIntProperty(this, default).get()
        is Float   -> dynamicPropertyFactory.getFloatProperty(this, default).get()
        is Double  -> dynamicPropertyFactory.getDoubleProperty(this, default).get()
        is Boolean -> dynamicPropertyFactory.getBooleanProperty(this, default).get()
        is Long    -> dynamicPropertyFactory.getLongProperty(this, default).get()
        else       -> default
    } as T
}

Kotlin would know based on the input param what "type" it is. And we can use that power to use the when expression along with is to check what type it is and call appropriate function based on the type of default value.

So now our code can become as simple as:

"SOME_CONFIG_PROP_NAME".getConfig("I am a good default value")

"CallExternalService".getConfig(false)

Boom. Done.

package com.example.groovy

import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
import groovy.transform.ToString

/**
 * Created by rhasija on 1/15/16.
 */
class JsonToObject {

    public static void main(String[] args) {
        // Person object
        def person = new Person(firstName: "John", lastName: "Doe")
        // Json String
        def personJSON = new JsonBuilder(person).toPrettyString()
        // Json String to Map
        def personMap = new JsonSlurper().parseText(personJSON)
        // using Map to convert to Person object type
        def newPerson = new Person(personMap)

        println(person)
        println(newPerson)

        assert newPerson.firstName.equals(person.firstName)
        assert newPerson.lastName.equals(person.lastName)
    }
}

@ToString
class Person {
    String firstName
    String lastName
}

Please let me know if you found this helpful. Thanks!

I love to use RoboMongo^[1] as the Mongo DB Query tool. In my case the problem is that the MongoDb server is on a private network and the only way to get to the private server from my machine is via ssh. Which means logging into the box, using mongo command to connect to the database and then issue my queries. Of course manually typing in my queries on a command line isn't my favorite thing to do. Being able to use RoboMongo is a major productivity boost and I want to be able to use it to connect to remote DB.

Solution:

Enter SSH tunnel.

Issue the following command on your terminal (I am using Mac hence a UNIX based system):

ssh -fN -i ~/.ssh/YourKeyFile.pem -L 6666:localhost:27017 userId@1.2.3.4

The command above is setting up: traffic on your local machine's 6666 port to be forwarded to remote server's localhost:27017. The userId is the userId you use to login to remote box. And 1.2.3.4 is remote server's ip address. The -i parameter allows you to connect using a .pem file i.e. a password-less connection.

Please make sure not to use low port numbers (lower than 1024) as local machine's port, since you won’t be able to bind them without root privileges.

Once you have the SSH tunnel set you can connect via RoboMongo on localhost:6666.

Any traffic will be diverted to 1.2.3.4 then to localhost:27017.

Hope you found this useful. Please let me know if you have any questions/comments/feedback.

References:

Now like a good developer, you are trying to test customer-service to verify that it is adhering to the requirements contract. How do you test the service such that it goes through the system flow, makes the necessary calls to account service and returns responses?

Assuming that the account-service is configured to be available at http://localhost:8090, we can configure Wiremock ^{[1] [1:1]} to intercept all calls going to localhost:8090 and return stubbed responses.

Here's a sample Integration Test:

    WireMockServer wireMockServer = new WireMockServer(wireMockConfig().port(8090))
    WireMock wireMock = new WireMock("localhost", 8090)
    def baseUrl = 'http://localhost:8889/customerservice/v1/customerInfo'

    def setup() {
        wireMockServer.start()
    }

    def cleanup() {
        wireMockServer.stop()
    }

    def "get customer info"() {
        given:
        wireMock.register(get(urlPathEqualTo( "/accountservice/v1/account/account1234"))
                .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/json")
                .withBody(JSON_RESPONSE_DATA))
        )

        def http = new HTTPBuilder(baseUrl)
        def headers = ['Accept': 'application/json', 'Content-Type': 'application/json', 'X-FORWARDED-FOR': '1.2.3.4']
        def requestBody = '{"customerNumber":"cust1234","ZipCode":"12345"}'

        http.handler.failure = http.handler.success

        when:
        def response = http.post(contentType: ContentType.JSON, headers: headers, body: requestBody)

        then:
        wireMock.verifyThat(new RequestPatternBuilder(RequestMethod.GET, urlPathEqualTo("/accountservice/v1/account/account1234")))
        response
        !response.errors
    }

Line: 1-2 are setting up Wiremock server to respond to http requests to http://localhost:8090 (account-service in this case)

Line 5: Starts the wiremock proxy server before each test.

Line 9: Stops the wiremock proxy server after each test.

Line 14: Stubs the wiremock for a GET call for path matching /accountservice/v1/account/account1234

Line 15-19: Will return a 200 response with response body defined as a String in JSON_RESPONSE_DATA

Line 29: Calls the actual service via HTTP Post based on the baseUrl defined on line 3 with headers and request body defined on line 22 and 24.

Line 32: Verifies that the wiremock call was actually called.

Line 34-35: Verify that there was a valid response and response had no error.

Note:
The test code is written in Groovy and is using Spock^[1:2] test framework but you can also use Java along with JUnit as the test framework.

To add the wiremock library to your code (via pom.xml)

<dependency>
    <groupId>com.github.tomakehurst</groupId>
    <artifactId>wiremock</artifactId>
    <version>1.57</version>

    <exclusions>
        <exclusion>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>servlet-api</artifactId>
        </exclusion>
    </exclusions>
</dependency>   

In summary:

You can mock any external http/RESTful service based on the host and port you expect it at. Just configure Wiremock server to listen to that host and port, and start the wiremock server. Wiremock then would intercept all the calls that are made to that host and port.

Advantages:

a. Mock external services based on the contracts they provide.
b. Verify marshaling and un-marshaling of JSON/XML data to their respective objects.
c. Test your service end-points as if they are being called via external HTTP calls and verify parameter naming and marshaling to request object.
d. Verify contracts provided by your service (and other external services).

That's it. Thank you for reading. Please let me know if you have any thoughts/questions/feedback via comments below.

References:

I was trying to get spring-cloud-server started on my laptop and I was getting:

Downloaded: http://repo.spring.io/libs-snapshot/org/codehaus/mojo/maven-metadata.xml (21 KB at 16.6 KB/sec)
[WARNING] Could not transfer metadata org.apache.maven.plugins/maven-metadata.xml from/to central (https://repo.maven.apache.org/maven2): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[WARNING] Could not transfer metadata org.codehaus.mojo/maven-metadata.xml from/to central (https://repo.maven.apache.org/maven2): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Cause:

The issue is I don't have a valid certificate for http://repo.spring.io.

Solution:

Step 1: Get the certificate from the location. In this case it is: http://repo.spring.io.

Here's how:

I used Firefox to browse to the link. On the top left there is a lock icon. Click on it > Click on the right arrow to get to information > Click on More Information > Click on View Certificate > Click on Details > Click on Export and save the certificate on your drive as a file say: Certificate.cer.

** It's important you chose X.509 Certificate (PEM) as the type.

Step 2: Now that you have downloaded the certificate, you need to install the certificate on your machine.

On my machine I did this:

keytool -import -file ~/Downloads/Certificate.cer -keystore /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/lib/security/cacerts -alias <your_custom_alias_name>

Of course the location of your java's cacerts might be different. To find out the location of cacerts directory, do:

echo $JAVA_HOME

In my case the output of echo $JAVA_HOME was: /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home. Then I just looked for cacerts directory in it and found it to be at: jre/lib/security/cacerts

Note:

• -alias foo_bar is important so that you don't override default alias which goes to my_key. Of course foo_bar is an example and not a good alias name. You should use a more descriptive alias name than foo_bar ;-)
• If you dont't want to install a certificate you can also do below:

export MAVEN_OPTS="$MAVEN_OPTS -Djavax.net.ssl.trustStore=/Users/rhasija/Downloads/Certificate.cer"

This will solve the problem for maven but not necessarily for java.

Summary

The problem was java was missing the certificate from a specific website, in my case, http://repo.spring.io. So I got the certificate file using Step 1 and then imported the certificate in my machine's java cacerts using keytool as show in Step 2.

Hope you found this information helpful. Please let me know if you have any questions.

References:

We wanted to see what Mongo was getting. There are a couple ways to do it:

1. Mongo Oplog

To quote Mongo documentation on Oplog^[1]:

The oplog (operations log) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. MongoDB applies database operations on the primary and then records the operations on the primary’s oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog, in the local.oplog.rs collection, which allows them to maintain the current state of the database.

So here's how we can use it:

use local
db.oplog.rs.find();

This will return you all the records in the op log collection of the format:

/* 0 */
{
    "ts" : Timestamp(1422739093, 110),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicBooks",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Spider Man",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.480Z")
    }
}

/* 1 */
{
    "ts" : Timestamp(1422739093, 110),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicVideos",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Spider Man Video",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.480Z")
    }
}

Notice: They are returning operations on different collections since we have not restricted the query to a particular collection.

Here's what the fields mean:

• ts: the time this operation occurred.
• h: a unique ID for this operation. Each operation will have a different value in this field.
• op: the write operation that should be applied to the slave. n indicates a * no-op, this is just an informational message.
• ns: the database and collection affected by this operation. Since this is a no-op, this field is left blank.
• o: the actual document representing the op. Since this is a no-op, this field is pretty useless.

If you want to check the recent operations on a particular collection you can:

db.oplog.rs.find({ns: 'Comic.ComicBooks'}).sort({ ts : -1 });

In Comic.ComicBooks: Comic is the database name and ComicBooks is the collection name.

/* 0 */
{
    "ts" : Timestamp(1422739093, 110),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicBooks",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Spider Man",
        "status: "NEW",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.480Z")
    }
}

/* 1 */
{
    "ts" : Timestamp(1422739093, 110),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicBooks",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Superman",
        "status: "NEW",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.480Z")
    }
}

If you want to query for a certain record you can get all the logs for it by:

db.oplog.rs.find({ns: 'Comic.ComicBooks', 'o._id': ObjectId("54d3977de4b0ad9775570661")}).sort({ ts : -1 });

Here we are querying for ComicBooks collection in Comic db, for id: 54d3977de4b0ad9775570661

And when I did so for the problem at hand I got the response below:

/* 0 */
{
    "ts" : Timestamp(1422739093, 110),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicBooks",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Spider Man",
        "status": "PROCESSING",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.480Z")
    }
}

/* 1 */
{
    "ts" : Timestamp(1422739093, 108),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicBooks",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Spider Man",
        "status": "OK",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.580Z")
    }
}

/* 2 */
{
    "ts" : Timestamp(1422738003, 107),
    "h" : NumberLong(-5537608931132355814),
    "op" : "u",
    "ns" : "Comic.ComicBooks",
    "o2" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01")
    },
    "o" : {
        "_id" : ObjectId("54caab29368a16b1f1a00c01"),
        "_class" : "com.rackspace.automation.support.mongo.AgentFile",
        "name" : "Spider Man",
        "status": "NEW",
        "lastUpdated" : ISODate("2015-01-31T21:18:13.180Z")
    }
}

The query sorts the records in descending order of time. So Mongo saw: NEW -> OK -> PROCESSING but if you notice the lastUpdated field you will notice that the Java application updated the object/entity in NEW -> PROCESSING -> OK sequence. So it seems like something is going wrong in Spring Data or MongoDB Java driver.

Another way to get research issues with operations is System Profile.

2. Mongo System Profile

MongoDb provides system profiling^[1:1] that allows you to profile database queries, operations etc. This can help identify inefficient queries and operations.

You can run the profile command on a particular database:

use Comic
db.setProfilingLevel(2)
// Perform whatever operation you want to debug
db.system.profile.find();

Sample output:

/* 0 */

{

"ts" : ISODate("2015-02-05T19:49:19.354Z"),

"op" : "update",

"ns" : "Comic.ComicBooks",

"query" : {

"_id" : ObjectId("54d3ba5a836c6476930566de")

},

"updateobj" : {

"_id" : ObjectId("54d3ba5a836c6476930566de"),

"_class" : "com.foobar.ComicBook",

"name" : "Superman",

"synchronizationStatus" : "OK",

"lastUpdated" : ISODate("2015-02-05T19:49:08.428Z")

},

"millis" : 0,

"client" : "10.14.15.16"

}

This will return you all the queries, and operations occurring in Comic db. Here's what the fields mean:

• ts: the time this operation occurred.
• op: the write operation that should be applied to the slave. n indicates a * ns: the database and collection affected by this operation. Since this is a no-op, this field is left blank.
• query: the query used to find the record.
• updateobj: the actual document representing the op. The contents of updateobj are prety self explanatory.
• millis: The time in milliseconds from the perspective of the mongod from the beginning of the operation to the end of the operation.
• client: The IP address or hostname of the client connection where the operation originates.

There are other fields that are included which are outside the scope of this post. You can read more about them here^[1:2].

NOTE: Please know that system.profile is a capped collection. So you can lose records over time since oldest records are removed if collection is near it's max size. Please set the profiling level to 0 once you are done researching the issue in order to not slow down the database. Also, this will help save hard disk space.

If you would like to query about specific collection you can do so by:
db.system.profile.find({ns: 'Comic.ComicBooks'});

If you want to query about a certain document id in a collection you can do so by:
db.system.profile.find({ns: 'Comic.ComicBooks', 'updateobj._id': ObjectId("54d3ba5a836c6476930566de")})

Basically, you can query this collection as you would query any other collection in Mongo.

Now, I could not find the records I was looking for in system.profile. The possible reason for that (according to team DBA is that we do not have logging on), but once I turn logging on, I would update this post to share if I could find what I was looking for with this. Regardless, I think this is a very handy tool to research issues with queries.

Great! I hope that was helpful.

Please help me out with your feedback/question in the comments section of this blog post. Muchas gracias.

References

Solution: Here's a simple python script to delete tags locally as well as from remote branch.

__author__ = 'ravi'
from subprocess import call
import logging as log
import os
 
os.chdir('/home/ravi/dev/projects/spiderman')
for i in range(115, 280):
    try:
        # Remove locally
        call(['git', 'tag', '-d', '1.0.{}'.format(i)])
    except Exception as ex:
        log.exception(ex)
        pass
    try:
        # Remove from master
        call(['git', 'push', 'origin', ':refs/tags/1.0.{}'.format(i)])
    except Exception as ex:
        log.exception(ex)
        pass

NOTE:

1. Line #6: Provide the path of the git repo.
2. Line #7: Change the format and numbers to the tag format and numbers you want to remove.
3. Please be careful!

Pretty straightforward!

Releases are a workflow for shipping software to end users. Releases are first-class objects with changelogs and binary assets that present a full project history beyond Git artifacts. They're accessible from a repository's homepage.

Thus anybody within the company who has access to the repository would be able to access the release artifact, view the release history and know what went into each release by looking at the release notes/comments. It's pretty nifty!

You can of course create each release manually, but we wanted to automate the releases via Jenkins! Now, Jenkins has Github Publisher plugin but it does not have the ability to perform releases. Not as of yet. But, Github provides an api^[1:1] which is super easy to use. And it is pretty straightforward to create a simple Python script that would integrate with the API to create releases, upload artifacts etc.

So I came up with a simple python script that allows us to:
a. Get all the releases for a given repo,
b. Create a release,
c. Delete a release given the release name,
d. Download release artifacts given the release name,
e. List artifacts for a release.

I have included the full code below:

You can use the script like so:
python github\_release\_management.py -c -t <Github_Token> -o <Github Organization> -r <repo name>

Creating a release is now a simple two step process:

export BUILD_NUMBER=1
python github_release_management.py -c -t  -o Spiderman -r awesome-api

Creating the Github token for Jenkins:

In order to create and upload releases you need access to your github. If you have 2FA (Two Factored Authentication) ON then you need to generate a token that would allow you to create/upload release artifacts. You can create a token by going to Account Settings -> Application and generating a new token:

Once you have created the token, copy it some place so you can easily access it for the next step.

Integrating with Jenkins:

The reason why I am using BUILD_NUMBER environment variable in the script above is because Jenkins automatically sets the BUILD_NUMBER variable for each build. So I can use that as part of the release number.

To setup Jenkins build to run the python script above you need to install the Post Build Task^[1:2] plugin. Once you have the plugin installed, you can then go to your actual build configuration and setup the Post Build Action like so:

Notice the grayed out area after -t-, that's where you should put the token you generated above.

Save. And voila! You are done!

Have you automated releases via Jenkins on GitHub? How did you do it?

Also, please feel free to comment below any questions/comments/ideas you might have? Your comments help me understand what I need to tweak on my blog so as to make the content more explanatory!

References:

1. Save the file onto the server and serve it from there.
2. Mongo[^n] provide GridFS[^n] store that allows you not only to store files but also metadata related to the file. For example: you can store author, tags, group etc right with the file. You can provide this functionality via option 1 too, but you would need to make your own tables and link the files to the metadata information. Besides replication of data is in built in Mongo.

Bottle

You can upload and download mongo files using Bottle[^n] like so:

import json

from bottle import run, Bottle, request, response
from gridfs import GridFS
from pymongo import MongoClient

FILE_API = Bottle()
MONGO_CLIENT = MongoClient('mongodb://localhost:27017/')
DB = MONGO_CLIENT['TestDB']
GRID_FS = GridFS(DB)

@FILE_API.put('/upload/< file_name>')
def upload(file_name):
    response.content_type = 'application/json'
    with GRID_FS.new_file(filename=file_name) as fp:
        fp.write(request.body)
        file_id = fp._id
    # If the file is found in the database then the save
    # was successful else an error occurred while saving.
    if GRID_FS.find_one(file_id) is not None: 
        return json.dumps({'status': 'File saved successfully'})
    else:
        response.status = 500
        return json.dumps({'status': 'Error occurred while saving file.'})


@FILE_API.get('/download/< file_name>')
def index(file_name):
    grid_fs_file = GRID_FS.find_one({'filename': file_name})
    response.headers['Content-Type'] = 'application/octet-stream'
    response.headers["Content-Disposition"] = "attachment; filename={}".format(file_name)
    return grid_fs_file


run(app=FILE_API, host='localhost', port=8080)

And here's the break down of the code:

Upload method:

Line 12: Sets up upload method to recieve a PUT request for /upload/<file_name> url with file_name variable holding the value that user passed in.
Line 15-17: Create a new GridFS file with name: file_name and get the content from request.body. request.body may be StringIO type or a File type because Python is smart enough to decipher the body type based on the content.

Download method:

Line 29: Find the GridFS file.
Line 30-31: Set the response Content-Type as application-octet-stream and Content-Disposition to attachment; filename=<file_name> so the client can download the file.
Line 33: Return the GridOut object. Based on Bottle documentation (below), we can return an object which has .read() method available and Bottle understands that to be a File object. Bottle handles return of File object(s) such that they can be downloaded.

File objects
Everything that has a .read() method is treated as a file or file-like object and passed to the wsgi.file_wrapper callable defined by the WSGI server framework. Some WSGI server implementations can make use of optimized system calls (sendfile) to transmit files more efficiently. In other cases this just iterates over chunks that fit into memory.

That is as simple as it gets as far as Bottle is concerned. Now on to implementing the same functionality in Flask.

Flask

You can upload/download files using Flask[^n] like so:

import json
from gridfs import GridFS
from pymongo import MongoClient
from flask import Flask, make_response
from flask import request

__author__ = 'ravihasija'

app = Flask(__name__)
mongo_client = MongoClient('mongodb://localhost:27017/')
db = mongo_client['TestDB']
grid_fs = GridFS(db)

@app.route('/upload/', methods=['PUT'])
def upload(file_name):
    with grid_fs.new_file(filename=file_name) as fp:
        fp.write(request.data)
        file_id = fp._id

    if grid_fs.find_one(file_id) is not None:
        return json.dumps({'status': 'File saved successfully'}), 200
    else:
        return json.dumps({'status': 'Error occurred while saving file.'}), 500

@app.route('/download/')
def index(file_name):
    grid_fs_file = grid_fs.find_one({'filename': file_name})
    response = make_response(grid_fs_file.read())
    response.headers['Content-Type'] = 'application/octet-stream'
    response.headers["Content-Disposition"] = "attachment; filename={}".format(file_name)
    return response

app.run(host="localhost", port=8081)

You might notice that the Flask upload and download method(s) are very similar to Bottle's. It differs only in a few places listed below:

Line 14: Routing is configured differently in Flask. You mention the URL and the HTTP methods that apply for that URL.
Line 17: Instead of request.body you use request.data to get the request content.
Line 28-31: In Flask, if you want to add additional headers, one way to do so is to "make the response" with the file content and set up the appropriate headers. Finally, return the response object.

Questions? Thoughts? Please feel free to leave me a comment below. Thank you for your time.