Aliens and Avatars – Explorative Product Deveploment in Action

On this years W-JAX I had the pleasure to host a session about explorative product development. The session was about the appropriate mixture of methods and software architecture especially, but not only, for projects in the pioneering phase. A good example of the Creative Software Workbench in action. You can see the session recording on entwickler.de (subscription required).

The Data Lakehouse – Best of Data Lake and Data Warehouse

Almost every company today utilizes a kind of data warehouse or business intelligence solution for data analysis and reporting. Those solutions are primarily based on relational data, ETL jobs and reporting. Although powerful they are limited when it comes to very large data sets or realtime processing.

Some years ago the paradigm of Data Lakes was born to process very large data sets. Data Lakes are based on the idea of raw data processing, streaming data, ELT and machine learning.

What about combining the strengths of both into something even more powerful? This is what is called the Data Lakehouse, a term conceived by Databricks.

Evolution of data storage, from data warehouses to data lakes to data lakehouses
Data Lakehouse. Source: https://databricks.com/de/glossary/data-lakehouse

As the name suggests, it combines the strengths of Data Warehouses with the power of Data Lakes. Although the term Data Lakehouse was not really used in 2020, we built a Data Lakehouse for a logistics company already then.

One of the main datasets in this project comprised 16 years or freight offers plus live data. The historical data was transferred from Oracle Databases to a new Data Lake. In addition stream sources were set up to ingest live data directly from the source applications into the Data Lake. The result was a huge active archive including historical and live data based on Hadoop, Spark, Kafka and HBase. The raw data was stored and continuously transformed into a normalized form ready to be processed by reporting and machine learning jobs. A logical structure, metadata and governance were added using Apache Atlas and Avro schemas. Reporting and end user security was implemented using Microsoft Power BI.

The result was something we would probably call a Data Lakehouse today. The combination of BI and Data Lake was very successful, so we created a success story to describe it.
To me is seems that Data Lakehouse is a very useful concept. It is an evolutionary step towards an integrated solution for processing and analysis of massive amounts of data by applying good practices in terms of governance, security and reporting. Surely something BI-Teams should have an eye on.

Success Story: Big Data in Logistics

In the years 2019 and 2020 I had the pleasure to support TIMOCOM in the implementation of their brand new Big Data Platform.
TIMOCOM is an international logistics platform provider and a true champion in its area.

When we started the initiative the company had an existing BI-System to perform reporting and statistical analysis. The aim was to extend the capapabilities of the company to collect, store and analyse huge amount of data. A Big Data solution comprising best of breed open source products was chosen. The new technology stack is able to scale not only technically but also business wise as it ist completely license cost free. It is based on technologies such as Java, Python, Hadoop, Hive, Kafka, Spark and HBase.
A major challenge in the beginning was that the staff had almost no knowledge of the applied technologies. To cope with this situation and to establish the solution quickly and in high quality, we’ve set up a Creative Software Workbench (CSW). A CSW combines the areas of modern technology, agile methodology and team dynamics to create an enviroment in which digital products can be created in the best possible way. It is based on more than 25 years of practical experience from many successful and of course some not so successful projects. In this enviroment agile engineering and active learning are important parts which helped us to master the Big Data ecosystem in a reasonable amount of time.

The new platform enables the company to gain new insights from their data today an tomorrow. It is an important step in the future to support their data driven business model.
You can read about the project in the success story “Wissen aus Daten”. I am glad that I can add this story to our list of success stories. If you want to know more about it, don’t hesitate to contact me.


Workshop: Big Data you can Touch

Today I released the brand new Workshop:Big Data you can Touch.

If you start researching about Big Data Platforms, you will find an overwhelming amount of possible technologies. But if you dig deeper you’ll find that many platforms are based on the same proven Open Source products.

This workshop teaches how to set up your own Big Data platform using professional Open Source products. Together we’ll build a end-to-end use case using a Lambda-Architecture and Machine Learning.

It is intended for all people who are generally intested in Big Data platforms, e.g. developers, architects, analysts or decision makers, who want to know how those technologies work together.

The workshop takes 4 hours and can be booked as On-Site-Training and Online-Webinar. Hope to see you there…

Upcoming events:.

7. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

13. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

21. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

Fintech Success Story

In 2017 PLEUS Consulting supported Yareto GmbH in the development of their new independent comparision site for the automotive finance industry.
Yareto is a fintech company that specializes in automotive financing. In 2016 the corporate startup was founded to build a brand new comparision site for the automotive finance industry. The site enables car dealers to compare credit offers in the areas of sales financing and purchase financing. Lenders get access to sales channels they couldn’t serve before.

PLEUS Consulting supported Yareto in setting up a Creative Software Workbench. The Creative Software Workbench aligns technology, processes and people in a way that creates an environment in which high quality digital products can be developed in short time.

The front-ends were developed using modern web technologies such as Java-/Typescript, HTML5, CSS and Angular. For the backend Java Enterprise (JEE) and a Sustainable Service Design approach was utilized to design and build a backend with a high degree of reuse and scalability. The service landscape was established using Domain Driven Design principles. Operations was performed using cloud platforms.

On the technical side, PLEUS Consulting supported the teams as Lead Developer. In the area of agile techniques, PLEUS Consulting supported the development teams as Agile Coach. The combination of those roles worked quite well especially in the phases of seed and growth. With these roles the company received thorough support in the areas of technology and methodology.

The project has shown that with a combination of modern technologies, agile approaches and the right people a very short concept to market cycle can be achieved, creating competitive advantages. This is what the Creative Software Workbench is all about.

You can read more details about the project in the official success story. More info about the Creative Software Workbench can be found on the official website.

Bank11 Success Story

In 2016 PLEUS Consulting supported Bank11 in the development of their brand new sales financing system VICTOR 3.0.

Success Story Bank11 is a credit institution that specializes in sales financing. In 2016 the bank decided to replace their existing software with something new. To be able to meet the challenging requirements in terms of quality, customer satisfaction and process efficiency they decided to build their own solution.

The front-ends were developed using modern web technologies such as Javascript, HTML5, CSS and Angular. For the backend Java Enterprise (JEE) and a Sustainable Service Design approach was utilized to design and build a backend with a high degree of reuse and scalability. The service landscape was established using Domain Driven Design principles.

On the technical side, PLEUS Consulting supported the teams as Master Developer and Architecture Owner. In the area of agile techniques, PLEUS Consulting supported the development teams as Scrum Master and Agile Coach. Although not 100% tension free, the combination of those roles worked quite well. With these roles the bank received thorough support in the areas of technology and methodology.

From the beginning we tried to align technology and business as much as possible, creating a people centered architecture. Central to the strategy were BPMN process models, graphical business rules and visual service contracts. In order to create appealing front-ends for the car dealers and the back office of the bank we worked closely with user interface specialists which were members of the cross functional teams. Web stack technologies allowed us to create individual and great looking front-ends. Agile frameworks such as Scrum organized the development teams and created valuable software together with the customer within a short period of time.

The project has shown that with a combination of modern technologies and agile approaches a very short concept to market cycle can be achieved, creating competitive advantages. It also demonstrates that it is possible to establish an agile culture in rather traditional business domains.

You can read more details about the project in the official success story. If you want to find out more come to watch my talks at JAX 2017 in Mainz.

Sustainable Microservices with Spring Boot

In my article series about Sustainable Service (SSD) design I described a design and implementation approach to develop services with technical decoupling to improve reuse.

At the level of IT infrastructure sustainable means that service implementations can be used in different technical environments without a major rewrite. Technical decoupling is a key factor to achieve that. In the second part of the article I have provided an example on how to implement SSD with a JEE stack. Part of this example is a  calculator service which performs simple arithmetic operations.

Due to the technical decoupling, services can be moved with little effort, for example from a JEE Server to other runtimes like Apache Karaf (OSGi) or Spring Boot, just to name a few.

In this blog post I would like to demonstrate how to move the Calculator JEE example from Wildfly JEE Container to Spring Boot. The main difference in the Spring Boot deployment is the fact that each service is bundled with its own HTTP server. The deployment unit is not a JEE EAR which is deployed on a JEE Server, but a so-called über-jar which includes the complete HTTP infrastructure. The über-jar just requires a Java runtime and no additional infrastructure. This kind of deployment creates a high level of service autonomy which is often used in Microservice architectures.

Let me tell you a little story:

Assume Peter is an IT professional who is working on a fictitious software project for a large insurance company. One day a colleague, let’s call him Max, from another project enters Peterʼs office and starts the following conversation:

Max: I’ve heard you’ve implemented some very useful services. I saw them on your service repository Wiki and think we could use some of them in our new project.

Peter: Yes, that’s right. I am glad we’ve created something valuable.

Max: But…I also heard that you are using a full blown JEE Applicationserver to run your services.

Peter: Yes, this is the best runtime for our project, as it helps us to manage centralized deployment. Each service is deployed in its own EAR file, which gives us great flexibility.

Max: For our project we decided to use Spring Boot and deploy each service together with its own HTTP Server. I guess we can’t use your service without a major rewrite then?

Peter: You don’t have to rewrite the services because we’ve implemented them based on Sustainable Service Design.

Max: Sounds great, could you please show me what we have to do to run your services?

Peter: Of course. Let’s start by downloading the Calculator Example, which demonstrates how to build SSD-Services for JEE. First, build the example like so:

mvn clean install

Peter: Now you have the following maven artifacts (jars) in your local maven repository.

net.pleus.services.calculator:calculator_api
net.pleus.services.calculator:calculator_impl

Peter: You can use this artifacts without any modifications.

Max: Ok, I can see that api and impl form the core service. Our project decided to just use JSON/HTTP as the protocol for service interaction. I saw that in the orignal example REST, EJB and SOAP are provided.

Peter: No problem, just add the bindings when you need them. With SSD you can add additional bindings any time. So you can start with the bare minimum and expand later. This gives you great flexibility. Add the JSON/HTTP-binding first.

Max: In the original example I saw that it was called REST-Binding.

Peter: Although the original calculator example uses the term REST-Binding, I prefer to call it JSON/HTTP-Binding because it better describes what it is. An SSD-Service can manage multiple resources (nouns) and can support arbitrary operations (verbs). This representation is very well suited to modelling the real world (the domain) which is important for proper service design and reuse. If you really have the requirement to create REST-Style APIs, you can do it in the respective binding. But be aware that in this case you create a variation of your service contract (subcontract) which also relies on HTTP-Verbs instead of the verbs in the primary contract. Although it is possible, I would not recommend it. Ok, let’s not digress but move on with Spring Boot.

Max: Ok, please show me how to create the bootable service.

Peter: Sure, start with the following Maven-POM , which is based on the tutorial Building a RESTful Web Service with Spring Boot.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.5.RELEASE</version>
    </parent>

    <groupId>net.pleus.services.calculator</groupId>
    <artifactId>calculator_boot</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>Services :: calculator :: boot</name>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <dependency>
            <groupId>net.pleus.services.calculator</groupId>        
            <artifactId>calculator_api</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>net.pleus.services.calculator</groupId>        
            <artifactId>calculator_impl</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
    </dependencies>
    
    <properties>
        <java.version>1.8</java.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Peter: From lines 25-34 you can see the existing calculator jars. The rest is required to create a minimal Spring Boot über-jar. Now we create a JSONHTTP-Binding using Spring MVC.

@RestController
@RequestMapping(value = "/services/calculator/rest/api", method= RequestMethod.POST)
public class CalculatorJSONHTTPBinding {

    @Autowired
    private Calculator service;

    @RequestMapping(value = "/performcalculations", method= RequestMethod.POST)
    @ResponseBody
    public PerformCalculationsResponse performCalculations(@RequestBody PerformCalculationsRequest request) {
        return service.performCalculations(request);
    }
}

Peter: To add this binding we need some boilerplate code. First an application class.

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        ApplicationContext ctx = SpringApplication.run(Application.class, args);
    }
}

Peter: And second a little factory to create a service instance, so that it can be injected using @Autowired in the binding.

@Configuration
public class Factory{
    @Bean public Calculator createCalculator(){
        return new CalculatorImpl();
    }
}

Peter: That’s all. Build it and run the über-jar with the following command.

mvn clean install
java -jar target/calculator_boot-1.0-SNAPSHOT.jar

Max: Wow, that’s all? Can you prove that it works?

Peter: Of course. For example fire up SOAP-UI and send this request to the service at http://localhost:8080/services/calculator/rest/api/performcalculations

{
	"correlationid":"4711",
	"calculations": {
	 "value": [
	   {
	     "operation": "ADD",
	     "inputs": {
	       "value": ["1","2"]
	     }
	   }]
	}
}

Peter: This is what you get.

{
   "correlationid": "4711",
   "calculations": {"value": [{"result": 3}]}
}

Max: Can I reuse the existing binding from the original example instead of the one we created?

Peter: Yes, but it is technically coupled to JAX-RS. If you want to use it please read the blog Using JAX-RS With Spring Boot Instead of MVC.

Max: I saw that the original example contains a handy Java client to access the service. Can I reuse it?

Peter: You mean net.pleus.services.calculator:calculator_binding_rest_client. Yes, you can use it as it is. And it makes sense, as it gives you a nice fluent Java-API to access the service. First add the following Maven artifact to your pom.xml.

        
            net.pleus.services.calculator        
            calculator_binding_rest_client
            1.0-SNAPSHOT
        

Peter: Now you can use the Java client in your tests like this.

@Before
public void setUp() throws Exception {
  client = new CalculatorClient("localhost",port);
}
    
@Test
public void add() throws Exception {

  // Create calculation inputs
  Calculation calculation = new Calculation()
   .withOperation(Operation.DIVIDE)
   .withInputs(new ArrayOfInt()
     .withValue(9,3,2)
   );
				 
  // Create request
  PerformCalculationsRequest request = new PerformCalculationsRequest()
    .withCorrelationid(UUID.randomUUID().toString())
    .withCalculations(new ArrayOfCalculation().withValue(calculation));
		
  // Call service	
  PerformCalculationsResponse response = client.performCalculations(request);

  // Check correlation
  Assert.assertEquals(request.getCorrelationid(), response.getCorrelationid());
		
  // Check result
  Assert.assertEquals(new BigDecimal(1.5), response.getCalculations().getValue().get(0).getResult());	
}

Peter: When you run the test Spring Boots starts the HTTP Server and calls the service.

Max: It seems that I can easily run your service and even use the Java client within Spring Boot. We have a sustainable service and a lightweight runtime. Perfect! That saves us a lot of time and money. Maybe we should try to evolve the service together? This way we could create further value for other projects.

Peter: Good idea! If you want to try the example I’ve packaged it for download. For convenience it also contains identical copies from the original calculator example.
Feel free to use it as you like.

Sustainable Service Design

We all know the idea of sustainability from our daily life. But is it possible to apply this idea to software development? I think yes.
Sustainable Service Design is a practical approach to design and implement software services with a great level of reuse both at technical and business levels. It is based on the following four principles:

  1. Technology-agnostic service definition
  2. Unified request/response
  3. Consequent contract first
  4. Technology bindings
00_cover_0 If you want to know more, please read my latest article about sustainable service design which has been publised in issue 2.2015 of Javamagazin (German).The interview can be found on the jaxenter site.
In the second part in issue 3.2015 I am showing how to implement a sustainable approach using JEE and JBoss Wildfly.