Thursday, 9 July 2020

Linux Authentication and Access - A different approach

A small briefing that covers the basic points of creating a new user in a Linux host. We start with a simple username/password old and insecure authentication method and then we add RSA certificate to make the user authentication modern and more robust. In the second section, we copy the private key to our windows machine and we configure windows PuTTY so that we connect to the Linux server using RSA certificates.
In the second part of the article, we make a very short introduction to Chef Technology and how this can be used in order to automate user provisioning on a host. This example does exactly the same user provisioning operations shown in the first part but this time with Chef.
In the last section we introduce we introduce a simple Jenkins pipeline that demonstrates the use of the sftp and ssh commands for the linux user we previously created. 
Just 10 minutes reading.

Read the full article here:

Monday, 22 June 2020

Web based CRUD operations made with ZKoss ZK Framework with JPA, Spring, using Intellij Idea

Hi folks!

This month we will continue the ZKoss/ZK framework. series and we present you a full functional example of a simple ZKoss ZK web application that makes JPA/CRUD operations on a database entity. The user is introduced to a web form that is build around a ZK listbox and presents the contents of a DB table. The user through the web form can perform CRUD operations (Create, Retrieve, Update and Delete  table records that are subsequently persisted in the DB level using Hibernate JPA implementation. In the article we come across with the DAO/Adapter pattern, we explain how ZK implements the the MVVM Pattern and we explain the design using  simple elements of the UML methodology.
The implementation of the  example utilizes technologies such as J2EE JPA, Spring framework and Maven in order to compile and build the WAR artifact over  the Intelij IDEA  programming environment/IDE.

Read the full article in Illumine IT articles GitHub:

https://github.com/illumine/articles/tree/master/ZK-JPA-Spring-Tutorial-with-IDEA-Maven

Thursday, 28 May 2020

ZKoss ZK framework for Java Application Development





Publish those two training sessions dealing with ZKoss hashtag ZK framework for Java web application development.

Both those training sessions were given to the engineers/analysts of MOU S.A. some years ago.



Wednesday, 15 April 2020

ακατάληπτα σύμβολα στην ελληνική γραμματοσειρά του microworlds pro

MICROWORLDS PRO Πρόβλημα εγκατάστασης στα Microsoft Windows 10



Το φοβερό MICROWORLDS PRO - το επίσημο εποπτικό μέσο που προτείνει το Υπ. Παιδέιας - δε παίζει πάντα στα Microsoft Windows 10. Γιατί? Γιατί απλά η εφαρμογή που προτείνει το Υπ. Παιδείας είναι αρχαία βασισμένη σε 32 bit βιβλιοθήκες, χρησιμοποιεί άλλο code page από τις περισσότερες σύγχρονες εγκαταστάσεις των Microsoft Windows 10 και φυσικά κατά την εγκατάσταση στην οθόνη βγαίνουν μπαρμπουτσαλα στην καθομιλουμένη και _"ακατάληπτα σύμβολα στην ελληνική γραμματοσειρά του microworlds pro" όπως το έθεσε η γραφουσσα εδω



Για την υποστήριξη της εφαρμογής MICROWORLDS PRO που δυστυχώς ακόμη τυραννάει τους μαθητές Γυμνασίων και Λυκείων η Illumine IT Consulting έγραψε το παρακάτω άρθρο

Friday, 3 January 2020

Creating a Full Web Based Business

Hi folks! Merry Christmas to everyone!

One of my old clients asked me how to create a full web business site. the site should be able to do the common basic usual stuff:
  • Promote the company - Inform customers about the brand
  • Present products 
  • Do online sales
The interesting part of the story is that this guy did not want me to do the actual site work. Instead he asked me:
  •  to make a TODO list of ll basic steps with a correct order so that nothing is missing and everything is in order.  Write down a well formed procedure. 
  • also to put all those artifacts that should be delivered in each of the steps from the side of the implementing contract company, so that the contractor is checked - ensuring no hidden details
Read the full article  on Illumine IT Consulting GitHub Space

Thursday, 17 October 2019

Elementary Linux Performance Monitoring

The basic tool here is top
Monitoring a single process can be done with -p option, in the next example we measure the MySQL process:

[root@(db-master) ~]# top -p 2521
top - 15:42:54 up 40 days, 10:46,  4 users,  load average: 0.14, 0.24, 0.48
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32551020 total, 32285684 used,   265336 free,   149660 buffers
KiB Swap:  3129340 total,   402572 used,  2726768 free. 16662620 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2521 mysql     20   0 18.725g 0.014t   4548 S 6.000 46.50   2735:03 mysqld

Load Average is a linux/unix mystery: Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. 
For an extended excellent article around Linux Load Average, refer to Brendan Gregg's Blog

On the other hand good old ps which is available on all UNIX flavors and LINUX distributions can also help. The following command shows the most CPU consuming processes  in ascending order along with their virtual size 

[root@(db-master) ~]# ps -e -o pid,pcpu,vsz,comm= | sort -n  --key=3
...
 1669  0.0 752396 isecespd
 1759  0.0 1561472 isectpd
 2521 52.4 19634584 mysqld

To get the process tree try pstree -aAl:

[root@(db-master) ~]# pstree -aAl
systemd --switched-root --system --deserialize 24
  |-VGAuthService -s
  |-agetty --noclear tty1 linux
  |-automount -p /var/run/automount.pid
  |   `-5*[{automount}]
  |-cron -n
  |-dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  |-discagnt /etc/init.d/discagnt start
  |   `-discagnt
  |-haveged -w 1024 -v 0 -F
...

For systems that do not have  pstree  try ps -ejH  

To get information about threads created by processes  try  ps -eLf

To get information about disk performance try iostat:

 [root@(mmcp_prod_corp)(db-master) ~]# iostat -dcm
Linux 4.4.121-92.117-default (mo-1400a55c2)     10/17/19        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.22    0.00    0.59    1.19    0.00   91.00

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               1.56         0.01         0.01      44144      51244
sdb             146.49         5.48         1.79   19159479    6250758

Finally to see all open files by a process such as data/shared objects/dynamic libraries and sockets use lsof. In the following example we can see all open files of mysql process:

[root@(db-master) ~]# lsof -p 2521
COMMAND  PID  USER   FD   TYPE             DEVICE     SIZE/OFF     NODE NAME
mysqld  2521 mysql  cwd    DIR              254,2         4096  6815769 /monsoon/mysql/data
mysqld  2521 mysql  rtd    DIR              254,0         4096        2 /
mysqld  2521 mysql  txt    REG              254,0    250387936   794500 /usr/sbin/mysqld
mysqld  2521 mysql  mem    REG              254,0        97056  1065145 /lib64/libresolv-2.22.so
mysqld  2521 mysql  mem    REG              254,0        26976  1065107 /lib64/libnss_dns-2.22.so


To see the TCP listening server sockets on a linux server, we can do that with netstat -tulpn

[root@(db-master) ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      2521/mysqld
tcp        0      0 0.0.0.0:2738            0.0.0.0:*               LISTEN      3282/discagnt
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3289/sshd
tcp        0      0 127.0.0.2:25            0.0.0.0:*               LISTEN      3671/master
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3671/master
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      38622/0
tcp        0      0 :::7938                 :::*                    LISTEN      3317/nsrexecd
tcp        0      0 :::5666                 :::*                    LISTEN      1/systemd
udp     4352      0 0.0.0.0:68              0.0.0.0:*                           1521/wickedd-dhcp4
udp        0      0 10.97.6.160:123         0.0.0.0:*                           3343/ntpd


while for all open TCP sockets:

[root@(db-master) ~]# netstat -t
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 mo-1400a55c2.zone:mysql mo-6740a22da.zone:46138 ESTABLISHED
tcp        0     64 mo-1400a55c2.zone1.:ssh mo-657dabf53.zone:58606 ESTABLISHED
tcp        0      0 mo-1400a55c2.zone:mysql mo-23acddcc0.zone:50068 ESTABLISHED





Friday, 20 September 2019

Creating a RSA Key pair, a Self Signed Certificate and put it on a JKS Java Key Store

Generating a Key Pair (Private/Public key) and a Self-Signed Certificate and store them to a JKS Java Key Store 

Job done on a Linux box using the openssl tools and JDK 's keytool

1) Generate RSA key pair of 2048 bits
openssl genrsa -out illumineit.com.key 2048  

2) Generate certificate request for CA (.csr)
openssl req -x509 -sha256 -new -subj '/C=CY/ST=Nikosia/L=Center/CN=illumineit.com'  -key illumineit.com.key -out illumineit.com.csr

3) Generate self signed certificate expiry-time 10 years from the certificate request
openssl x509 -sha256 -days 3652 -in illumineit.com.csr -signkey illumineit.com.key -out illumineit.com.crt


4) Import the pair (private key and selfsigned certificate) in a new JKS (Trustore and Keystore together)
# Create PKCS12 keystore from private key and public certificate.
openssl pkcs12 -export -name illumineit.com -in illumineit.com.crt -inkey illumineit.com.key -out illumineit.com.p12 -passin pass:welcome -password pass:welcome

# Convert PKCS12 keystore into a JKS keystore
keytool -importkeystore -destkeystore illumineit.com.jks -srckeystore illumineit.com.p12 -srcstoretype pkcs12 -alias illumineit.com -srcstorepass welcome  -storepass welcome  -noprompt

Wednesday, 19 June 2019

Retrieving the Posts and Pages from Wordpress Database.



Sometimes shit happens. Client took backup only the Wordpress DB without taking the PHP files. In other words, configuration, plugins, custom templates, skins and images.... just lost. Last Well Known good backup gone with the server.... and what we only got is a Wordpress DB without images. So practically, it would be a better idea to backup one by one pages from browser by clicking Save as.... Just jogging of course...




Now he has to write everything from scratch.

Step 1.

Examine your backup file:


-- MySQL dump 10.13 Distrib 5.5.55, for Linux (x86_64)
--
-- Host: localhost Database: wordpress9
-- ------------------------------------------------------
-- Server version 5.5.55




Step 2:

Go to MySQL and download the exact server version your previous installation comes from. Here is our link: https://dev.mysql.com/downloads/mysql/5.5.html

Install the MySQL temporarily in your PC or a simple VM or anything else.




Step 3:

Create a Database in your server just like the backup specifies:

C:\Users\>mysql -uroot -pmypass

mysql> CREATE DATABASE wordpress9 CHARACTER SET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.01 sec)







Step 4:

Copy your backup file. Edit your backup file

Restore your last backup in the server

C:\Users\>mysql -uroot -pmypass wordpress9 < backup_2019_01_18_1547817726_4347121_wpdb.sql



Step 5:

Create a file called restore.sql with the following query to retrieve your posts, pages and news:

select '<h1>',post_title,'</h1>',post_content, '<hr/> End Post <hr/>' from wp_posts where post_status='publish' and post_type in ('page','post','nooz_release') order by post_name, post_date ;

Step 6:
Run the query command as follows:
C:\Users\>mysql -uroot -pmypass wordpress9 < restore.sql > restored.html

The results are inside restored.html and can be viewed with a browser.
More careful backup next time....

Saturday, 3 February 2018

Enable SSL for your Wordpress/Plesk site using a free authority-signed certificate

This article explains how to  replace HTTP with HTTPS on your site. This is an easy task if your site is relatively small and can be accomplished with 5 to 6 basic steps.

The article assumes site deployment with Wordpress and Plesk dashboards and suggests the creation of a free trusted authority signed SSL sertificate from  Comodo Cyber Security trusted authority which is valied for 3 months (90 days).
 
Read the overall article here:
Illumine IT Consulting GitHub Articles.

 

 

Tuesday, 12 September 2017

disk-benchmark A mutlipurpose benchmark program that can simulate your application's I/O performance

disk-benchmark tool - get it here!

Sometimes we need to have a prior estimation of I/O performance of a program we plan to develop or we currently posses.
This may be triggered by a number of reasons:
  • Order specific Disk hardware in advance
  • Plan to rent cloud based volume from a cloud provider
  • Estimate the total performance of your application in order to establish operational scenarios and calculate KPIs.
  • Check the cloud providers SLA compliance.
In the past I dealt with all those challenges using standard Linux methods for benchamarking a volume like the classic one:


dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

Or other similar methods or tools like iostat.

The problem with all those methods, is that you can have an idea of how your disk performs in general, but not according to a given scenario, for example:
  • 20 concurrent users each of them reads and writes of a random file of size between 20k and 1 MB with a pause of 2 seconds for 5 mins.
  • 10 concurrent users each of them reads/ writes a file of 60kb with a pause of 2 seconds after read repeatedly for 100 times. 


Unless you go to very sophisticated tools like JMeter,  you don't really have something very handy. On the other hand, sophisticated tools most of the times, have a significant learning curve but of course in most cases, you want something to use it in the next 5 mins with very simple options just like the above scenarios. To amend this situation, last year, I developed a small C program that can be used to do the job, the disk-benchmark program available on Illumine IT Consulting GitHub URL:

https://github.com/illumine/disk-benchmark

This is a benchmark program to test Hard Drives, SSD Drives, HBAs, RAID Adapters & Storage Controllers. This is a really simple C program that you can compile using the standard GNU/gcc compiler that comes with your Linux distribution.

How to setup the disk-benchmark in your Linux system:
Installation of the disk-benchmark is as simple as this:

# git clone https://github.com/illumine/disk-benchmark
# cd disk-benchmark/src/
# gcc disk-benchmark.c -o disk-benchmark  -l pthread -lrt  -O3  -Wall
# ls -l disk-benchmark
-rwxr-xr-x 1 root root 23365 Apr 15 10:23 disk-benchmark

A simple scenario implementation using disk-benchmark

Scenario: 10 concurrent users each writing and reading a file of size ~10MB in /var.  Each user pauses for some seconds randomly picked from the interval [2,10] sec.  The command that implements the above scenario has as follows:

[root@mo-8f752419d src]# ./disk-benchmark -p /var -t 10 -a 10000000 -E 2:10

Test scenario:
test path=/var
Threads=10, sleep sec between write/read = 1, repeats per thread=5, random pick sleep sec from [2 10]
Lower file size=1024, Upper file size=10240, Absolute file size=10000000
Read/Write buffer size=8192,  Buff Siz W 0, Buf Siz R 0,
Do write only=0, Delete files=1
Print values only=0 dont print scenario info= 0, dont print clocks=0 dont print headers=0 print date=1
Work Continously=0  Work Continously Sleep Brake=5

T=7, Avg W=0.016134 Avg R=0.002160 Total W=0.080671 Total R=0.010801 Total Time=0.091473 Sleep=4.600000  Avg File Size =10000000.000000
T=2, Avg W=0.014436 Avg R=0.002411 Total W=0.072179 Total R=0.012056 Total Time=0.084234 Sleep=4.800000  Avg File Size =10000000.000000
T=4, Avg W=0.016104 Avg R=0.002189 Total W=0.080520 Total R=0.010943 Total Time=0.091463 Sleep=4.800000  Avg File Size =10000000.000000
T=9, Avg W=0.011966 Avg R=0.002069 Total W=0.059829 Total R=0.010347 Total Time=0.070176 Sleep=4.800000  Avg File Size =10000000.000000
T=6, Avg W=0.013065 Avg R=0.001826 Total W=0.065323 Total R=0.009128 Total Time=0.074451 Sleep=5.000000  Avg File Size =10000000.000000
T=1, Avg W=0.015399 Avg R=0.003005 Total W=0.076996 Total R=0.015025 Total Time=0.092021 Sleep=5.200000  Avg File Size =10000000.000000
T=8, Avg W=0.012883 Avg R=0.002303 Total W=0.064416 Total R=0.011513 Total Time=0.075930 Sleep=5.200000  Avg File Size =10000000.000000
T=3, Avg W=0.015850 Avg R=0.002492 Total W=0.079251 Total R=0.012458 Total Time=0.091709 Sleep=5.400000  Avg File Size =10000000.000000
T=0, Avg W=0.013430 Avg R=0.002697 Total W=0.067151 Total R=0.013487 Total Time=0.080637 Sleep=5.600000  Avg File Size =10000000.000000
T=5, Avg W=0.016659 Avg R=0.002387 Total W=0.083293 Total R=0.011934 Total Time=0.095226 Sleep=5.600000  Avg File Size =10000000.000000

T=-1, Avg W=0.014593 Avg R=0.002354 Total W=0.072963 Total R=0.011769 Total Time=0.084732 Sleep=5.100000  Avg File Size =10000000.000000
Wall time 28.000000, CPU time 0.880000
Tue Sep 12 13:36:26 2017


Wednesday, 18 May 2016

Web Service Client with Basic Authentication and SSL

Web Service Client with Basic Authentication and SSL

Recently, I had to create a web service client for a web service that uses a number of Web Service Policies. In general, the web service utilizes the following policies:
  • Transport: Service uses one way certificates. Client had to download and check server´s certificate in order to prove the server´s  identity.
  • Authentication: Basic authentication is required to access the URL and the service WSDL.
The following steps were used.
  • Creating the TrustStore: Access the Web Service URL, download the web service certificate and create a x509 trustStore to host the server´s certificate.
  • Create the client Stub: Access the Web Service URL and create the client stub by compiling the WSDL with wsimport.
  • Code and complete the service client. This has the following sub tasks:
    • Code the client to use Basic Authentication
    • Code the client to utilize the trustStore in order to setup SSL session with the server
    • Code the client to call the web method.

Creating the SSL Trustore.

During SSL handshake, the trustStore is used to verify server´s id.
Download the Server´s certificate by hitting the Web Service URL. There you will be prompted for login. You can login with the given user/password.

Then, the certificate is stored in your browser. You can export it easy but that depends to you browser. Chrome for example, the certificate can be downloaded directly as a x509 trustStore like the following image illustrates:



If you want to create the a trustStore manually you need to create a X509 keystore file using Java keytool and then import the server´s public certificate in it. The trustStore will be password protected and the certificate inside the trustStore will be password protected using "password" passphrase:

$ keytool -genkey -alias replserver -keyalg RSA -keystore mykeystore.jks -dname "cn=localhost, ou=IT, o=Continuent, c=DE"  -storepass password -keypass password

Now you have the keyStore. Next you need to import the server´s public certificate in it. In the general case, supposing the Server certificate is the following one plain text file server-certificate.txt then do one of the following actions to:

Check the server´s certificate:
openssl x509 -in server-certificate.txt -text -noout

Delete previous certificate version from the trustStore if any:
keytool -delete -alias myserver-name.com  -keystore mykeystore.jks 

Re-import the server certificate to the trustStore:
keytool -import -alias myserver-name.com -keystore mykeystore.jks  -file server-certificate.txt

Access the Web Service URL and create the client stub by compiling the WSDL with wsimport.

After running your wsimport command directly you should get a message complaining about a missing web authorization file.
What you need to do is create an authorization file (usually the default name/location for it is $HOME_DIRECTORY/.metro/auth, but check the previous error message, you'll get the hint from there).
Inside this file you just write the line: "https://username:password@url?wsdl"

 Now create a file called: wsimport_mysvc.bat and code the following commands:
setlocal
set _JAVA_OPTIONS=%_JAVA_OPTIONS% -Djavax.net.ssl.trustStore=mykeystore.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=mykeystore.jks
wsimport -s . -verbose -keep -p gr.illumine.wsclient.stub  -extension https://myserver-name.com/wsd/alc_interface?wsdl
endlocal

Doing so, you fulfill both conditions for basic authentication and also for transport/SSL by asking wsimport to examine what is been sent from server against to what is stored in mykeystore.jks

Run the wsimport_mysvc.bat and the client stub files will be created in the package gr.illumine.wsclient.stub

C:\>set _JAVA_OPTIONS= -Djavax.net.ssl.trustStore=cacerts -Djavax.
net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=cacerts

C:\>wsimport -s . -verbose -keep -p gr.illumine.wsclient.stub  -extension https://myserver-name.com/wsd/alc_interface?wsdl
Picked up _JAVA_OPTIONS:  -Djavax.net.ssl.trustStore=cacerts -Djavax.net.ssl.key
StorePassword=changeit -Djavax.net.ssl.trustStore=cacerts
parsing WSDL...

Code the client

The first thing you have to do is to add a static initializer that will provide the username and password for basic authentication:

public class AlcClient {
 
 private static final Logger log= Logger.getLogger( AlcClient.class.getName() );
 
 /* 
  * Use this static initializer to provide Basic Authentication for the Web Service Consumption
  */
 static {
     java.net.Authenticator.setDefault(new java.net.Authenticator() {

         @Override
         protected java.net.PasswordAuthentication getPasswordAuthentication() {
             return new java.net.PasswordAuthentication("happyuser", "mypassword".toCharArray());
         }
     });
 }

Next, configure your SSL settings in the code, by adding the following system parameters:
        /*
         * Use the following settings to specify how this client will utilize the X509 trust store
         * called mykeystore.jks. In this trustore, it is stored the server´s public certificate
         * Also the trustore/keystores are password protected with a password "password"
         */
        System.setProperty("java.protocol.handler.pkgs","com.sun.net.ssl.internal.www.protocol");
        System.setProperty("javax.net.ssl.keyStore","mykeystore.jks");
        System.setProperty("javax.net.ssl.keyStorePassword","password");
        System.setProperty("javax.net.ssl.keyStoreType", "JKS");
        System.setProperty("javax.net.ssl.trustStore","mykeystore.jks");
        System.setProperty("javax.net.ssl.trustStorePassword","password");
        System.setProperty("javax.net.ssl.trustStoreType", "JKS");

Then add some debugging options to debug your SSL session. You are strongly advised to comment out the following code after testing it since it will affect the SSL performance.
        /* Following options enable logging of all communication to the console
         * We are most interested in the request response SOAP Messages   */
        System.setProperty("com.sun.xml.ws.transport.http.client.HttpTransportPipe.dump", "true");
        System.setProperty("com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.dump", "true");
        System.setProperty("com.sun.xml.ws.transport.http.HttpAdapter.dump", "true");
        System.setProperty("com.sun.xml.internal.ws.transport.http.HttpAdapter.dump", "true");

Now code the Web Service client instance by using the stub you have created with the wsimport:
 ZALCINTERFACE_Service service = new ZALCINTERFACE_Service( new URL("myserver-name.com/wsd/alc_interface?wsdl"),
     new QName("urn:com:myserver-name:document:sap:soap:functions:mc-style", 
                      "ZALC_INTERFACE"));
  
 /*
  * From this service get the proper port
 */
 ZALCINTERFACE port = service.getZALCINTERFACE(); 

        /* Make the web service call */
        String responseMessage = port.callMyWebMethod();

Get the entire web service client java implementation can be downloaded here

Tuesday, 15 March 2016

Set your HTML META tags in WordPress

Hi folks!

I think there is no much need to tell you how important are HTML META tags for SEO for your site.
So in order to have a simple touch of perfection, you just have to edit the following META tags:

 
<meta name="" content="Illumine Consulting - Europe" property="og:title"/>
<meta name="" content="website" property="og:type"/>
<meta name="" content="technology" property="website:tag"/>
<meta name="" content="cloud computing" property="website:tag"/>
<meta name="" content="b2b" property="website:tag"/>
<meta name="" content="science" property="website:tag"/>
<meta name="" content="http://www.illumine.gr" property="og:url"/>
<meta name="" content="https://www.linkedin.com/company/illumine-it-consulting?trk=company_logo" property="og:image"/>
<meta name="" content="Illumine IT Consulting - Greece" property="og:site_name"/>
<meta name="" content="For more than ten years Illumine IT Consulting " property="og:description"/>
<meta name="" content="1392144595" property="og:updated_time"/>
<link href="https://plus.google.com/{+PageId}" rel="publisher" />
<meta name="" content="https://media.licdn.com/media/p/2/005/020/2ca/29e39f7.png" 
property="og:image"/>
<meta name="robots" content="index, follow" />
<meta name="keywords" content="illumine, IT, technology, consulting, services, software,mountrakis" />
 
<meta name="generator" content="illumine it consulting" />
<meta name="author" content="michael mountrakis" />
<meta name="copyright" content="Copyright (c) Illumine Consulting. All Rights Reserved." />
  
  
To do so, go to your WordPress admin panel Then in the left menu select Appearance, Editor and select to edit header.php file. Then add your meta tags just like the following picture illustrates:


Tuesday, 8 March 2016

Implement Redirects withing WordPress and Eggplant 301 Redirects

The easiest way in order to add a redirect in your Wordpress site is by installing Eggplant 301 Redirects Plugin. 


To do so login to WordPress as administrator

On the Left side menu go to Plugins --> Add New -->




Now in the Add Plugin page you have to add the word "eggplant" in the textfield and click "Install now"





The last part is to add a redirect. To do so, go on  the Left side menu go to Settings --> EPS Redirects

 And finally add the redirect to your Wordpress page using Eggplant plugin redirect management:



Monday, 7 March 2016

Apache httpd reverse proxy for Tomcat with SSL self signed certificates.

Recalling from the previous article on how to install Apache Tomcat 7 and Httpd on Fedora 22 we are now going to present how to configure Apache Httpd working as a reverse proxy for Apache Tomcat.

In more details, we are going to implement the following setup:
  • Setup Tomcat 7 listening on port 8080
  • Redirect port 80 (HTTP) to port 443 (HTTPS)
  • Use self signed RSA server certificates to authenticate our HTTPs server on clients and secure the TCP session.

Public and Private Server Key

In order to create the Server Public/Private key set we are going to use openSSL tools. 
 To install them in you Fedora 22 server do:
# dnf install openssl
# or for older Fedora systems
# yum install openssl

Then openssl tools are installed to:
# which openssl
/bin/openssl

Go to the apache httpd configuration directory and do the following:
# cd  /etc/httpd/conf/

Generate a PEM RSA private key key using DES3
# openssl genrsa -des3 -passout pass:mypass  -out server.pass.key 2048
Generating RSA private key, 2048 bit long modulus
..............................+++
...................................................................................+++
e is 65537 (0x10001)

Create a Server PEM certificate request using the server key:
# openssl req -new -key server.pass.key -out server.csr
Enter pass phrase for server.pass.key:     # put mypass here
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:GR
State or Province Name (full name) [Some-State]:Athens
Locality Name (eg, city) []:Athens
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Illumine IT Consulting
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:www.illumineit.com
Email Address []:info@illumine.gr

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:                      # press enter here to skip password
An optional company name []:  Illumine IT Consulting 


Finally, create the server certificate using the PEM Certificate Request
# openssl x509 -req -in server.csr -signkey server.pass.key -out server.crt  -days 365

Signature ok
subject=/C=GR/ST=Athens/L=Athens/O=Illumine IT Consulting/CN=www.illumineit.com/emailAddress=info@illumineit.com
Getting Private key
Enter pass phrase for server.pass.key:  # put mypass here


By the end of this operation you should have the following files created:
# ls -l
-rw-r--r--. 1 root root 1318 Mar  7 18:11 server.crt
-rw-r--r--. 1 root root 1115 Mar  7 18:07 server.csr
-rw-r--r--. 1 root root 1743 Mar  7 18:05 server.pass.key
  • server.ctr: is the server certificate
  • server.csr: is the server PEM certificate request
  • server.pass.key : server´s private RAS key.

Configure Apache HTTPd working with SSL certificates and reverse proxy to Tomcat

# vi /etc/httpd/conf/httpd.conf

Add the following section:
ServerRoot "/etc/httpd"
# Port 80 (HTTP) will be redirected to 443 (HTTPS)
Listen 80

   ServerName www.illumineit.com
   Redirect permanent / https://www.illumineit.com

# Port 443 HTTPS will be default
Listen 443

  ServerName www.illumineit.com
  ServerAdmin my-mail-here
  #
  # Configure SSL engine on and add your certificates
  #
  SSLEngine on
  SSLCertificateFile     conf/server.crt
  SSLCertificateKeyFile  conf/server.key
  #
  # proxypass configuration to your tomcat server running on 8080
  #
  ProxyPass        /zsecure-pdf/   http://www.illumineit.com:8080/zsecure-pdf/
  ProxyPassReverse /zsecure-pdf/   http://www.illumineit.com:8080/zsecure-pdf/
  ProxyPassReverseCookieDomain www.illumineit.com www.illumineit.com
  ProxyPassReverseCookiePath /zsecure-pdf  /zsecure-pdf
  
     ProxyPassReverse /
     SetOutputFilter  proxy-html
     RequestHeader    unset  Accept-Encoding
  

  BrowserMatch "MSIE [2-5]" \
  nokeepalive ssl-unclean-shutdown \
  downgrade-1.0 force-response-1.0

The first section VirtualHost configures Apache to redirect whatever goes to port 80 to be redirected to port 443 (HTTPS)

The second section VirtualHost configures Apache to use Tomcat as reverse Proxy. So if someone requests URI path /zsecure-pdf/ this will be redirected to port 8080 where tomcat listens.

Save and restart the Apache HTTPD:
# service httpd restart
Redirecting to /bin/systemctl restart  httpd.service

Test Apache

Hit with browser http://www.illumineit.com this will redirect you to https://www.illumineit.com

if you also navigate to the path that was reverse pass: https://31.171.245.82/zsecure-pdf/secure-my-pdf-to-image-password-encrypt-and-watermark.html then you will be served from Tomcat serving your application.

Potential problems

AH01114: HTTP: failed to make connection to backend
To get rid of this log to your server as root and run those commands:
/usr/sbin/setsebool httpd_can_network_connect 1
/usr/sbin/setsebool -P httpd_can_network_connect 1


Page does not renders correctly: images and CSS are missing. That is very common since HTML pages might taken from other sites by A HREF. The only think you can do is copy them locally to WebContent directory of your WAR deployment.

Fedora 22 Apache Tomcat and Httpd. Publishing an application in minutes.

Recalling from the previous article "Quest of the Holy Cloud" I got a provider and started a simple VM over there.
One of my first actions was to baptize my server and give it a fancy hostname.
Now lets come to the juicy part. In this article I am going to build a simple application server to handle PDF trans-code to images with a custom Java application I built.
The actions I am going to demonstrate are how to:
  • Setup OpenJKD on Fedora 22
  • Install Ghostscript libraries required for my application.
  • Download, install and configure Apache Tomcat 7
  • Install and configure Apache HTTPd.
  • Installing Open JDK

Install OpenJDK

The first step is really easy. We need a JDK or a JRE in order to run Tomcat that hosts our application. The straight option is to use opensource community JAVA: OpenJDK.
To do so, I entered the following commands:
# dnf install java
Last metadata expiration check performed 1:09:31 ago on Mon Mar  7 12:20:26 2016.
...
To check where java is and what has been installed:
# which java
/bin/java
# java -version
openjdk version "1.8.0_72"
OpenJDK Runtime Environment (build 1.8.0_72-b15)
OpenJDK 64-Bit Server VM (build 25.72-b15, mixed mode)

Install Ghostscript

Most of the software I wrote rely to Ghostscript shared libraries that are called from the corresponding Java API. To install them I entered the following commands:
# dnf install ghostscript
Last metadata expiration check performed 1:15:36 ago on Mon Mar  7 12:20:26 2016.
..
The library got installed at:
# ls -lh /lib64/libgs*
..
-rwxr-xr-x. 1 root root 16M Mar 31  2015 /lib64/libgs.so.9.16

# file  /lib64/libgs.so.9.16
/lib64/libgs.so.9.16: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=6601d742a4829cb3e4fe8197f1b1457f665ce130, stripped

Install Apache Tomcat 7

Apache Tomcat 7 can be downloaded from here as a tar.gz file by picking up a binary distribution as follows:
# cd /opt
# wget http://mirror.serversupportforum.de/apache/tomcat/tomcat-7/v7.0.68/bin/apache-at-7.0.68.tar.gz
# tar -xvf apache-tomcat-7.0.68.tar.gz

Now tomcat is not provided as a service from Fedora. To do so, we need to create a simple start script in /etc/init.d:

# cd /etc/init.d
# vi tomcat
paste the following to the script tomcat:
#!/bin/bash
# start/ stop Tomcat script
# Since you are using OpneJDK put this as your java home
JAVA_HOME=/
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
# Where you have placed tomcat
CATALINA_HOME=/opt/apache-tomcat-7.0.68

case $1 in
start)
sh $CATALINA_HOME/bin/startup.sh
;;
stop)
sh $CATALINA_HOME/bin/shutdown.sh
;;
restart)
sh $CATALINA_HOME/bin/shutdown.sh
sh $CATALINA_HOME/bin/startup.sh
;;
esac
exit 0
Now tomcat needs to be registered as a Linux service. To do so add those commands:
# cd /etc/init.d
# chmod 755 tomcat  
# chkconfig --add tomcat  
# chkconfig --level 234 tomcat on  
# chkconfig --list tomcat 

Installing Apache HTTPD

This comes as a standard service supported from Fedora distribution. To install it:
# dnf install httpd
...
For a very fast configuration of http you can edit httpd.conf and add a simple virtual host:
#  vi /etc/httpd/conf/httpd.conf
# add where "Listen 80" is:
Listen My.Host.IP.Here:80

    DocumentRoot "/www/illumineit.com"
    ServerName www.illumineit.com

    # Other directives here

Since in modern Cloud environments the linux firewall IP Tables may block everything, here are the commands to unlock the ports:
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
You can start the HTTP service and get its status:

# service httpd start
Redirecting to /bin/systemctl start  httpd.service
# service httpd status
Redirecting to /bin/systemctl status  httpd.service
 httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-03-07 14:09:27 UTC; 4s ago
 Main PID: 1760 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─1760 /usr/sbin/httpd -DFOREGROUND
           ├─1761 /usr/sbin/httpd -DFOREGROUND
           ├─1762 /usr/sbin/httpd -DFOREGROUND
           ├─1763 /usr/sbin/httpd -DFOREGROUND
           ├─1764 /usr/sbin/httpd -DFOREGROUND
           └─1765 /usr/sbin/httpd -DFOREGROUND

Mar 07 14:09:27 securepdf.illumineit.com systemd[1]: Starting The Apache HTTP Server...
Mar 07 14:09:27 securepdf.illumineit.com systemd[1]: Started The Apache HTTP Server.
The deployment directory for tomcat where you can place your WAR files is: /opt/apache-tomcat-7.0.68/webapps/ since I have donwloaded and installed tomcat on /opt.
You can use WinSCP to copy your WAR file there:

# ls -lh  /opt/apache-tomcat-7.0.68/webapps/
total 27M
drwxr-xr-x. 14 root root 4.0K Mar  3 11:00 docs
drwxr-xr-x.  7 root root 4.0K Mar  3 11:00 examples
drwxr-xr-x.  5 root root 4.0K Mar  3 11:00 host-manager
drwxr-xr-x.  5 root root 4.0K Mar  3 11:00 manager
drwxr-xr-x.  3 root root 4.0K Mar  3 11:00 ROOT
drwxr-xr-x.  4 root root 4.0K Mar  4 16:59 zsecure-pdf
-rw-r--r--.  1 root root  27M Mar  4 16:59 zsecure-pdf.war

Friday, 4 March 2016

Set you linux host name and domain

Recently I have created a new VM linux server on CloudSigma. The Server runs Fedora 22. In order to setup the hostname and network domain I have changed the following files:

[root@illumine ~]# cat  /etc/host
securepdf

[root@illumine ~]# cat /etc/hostname
securepdf

[root@illumine ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

178.XXX.XXX.132   securepdf.illumineit.com securepdf

Test it using ping:

[root@securepdf ~]# ping securepdf
PING securepdf.illumineit.com (178.XXX.XXX.132) 56(84) bytes of data.
64 bytes from securepdf.illumineit.com (178.XXX.XXX.132): icmp_seq=1 ttl=64 time=0.036 ms



The quest for the Holy Cloud.

The last 10 days I am struggling myself to choose a cloud provider.  

My selection criteria:

  • Free of charge for a trial use. No credit card registration.
  • Easy to use with what I know without having to invest on extra study
  • The resources the cloud provider offers for trial/free tries, like CPU, allowed Network bandwidth. The more resources offered, the best scoring for the cloud provider.
  • Technology used for Automation and VM provisioning. 

I tried several cloud providers by the order they appear on Google. 

First of all, I dumped Amazon Cloud Services only for one reason: I don not really want to put my Credit Card even before I have to pay for something just because the site asks it. If it was not Amazon behind the site- would you put your card? So no Amazon for me.

Second try with Openshift from Red Hat. I registered there and created a VM with Tomcat7/JBoss "cartridge". Cool - worked out easy and in about 10 minutes I managed to register and create a VM. However:  the machine has too many restrictions, for example you cannot add the packages you like with RPM or yum. Moreover, the Tomcat7 differs from the standard tomcat you download from Apache. When I tried to deploy one of my apps in the new machine there the deployment failed. Also, I did not like the approach of automation implementation with rhc tools. It reminded me some nightmares I had with Chef´s knife. 

My next try was with DigitalOcean. They do not have a free plan but instead they offer a voucher with discount. Again when I tried to register, after following the link in the confirmation email that was sent from them, I was redirected to their page asking for my Credit Card details again: "Thanks! Please add a credit card to activate your account." Thanks but no thanks guys. "There are other orange trees that also make oranges" as an old Greek piece of mind says.

Finally I got there:  cloudsigma.com.  No credit card requirement for a test drive of 7 days. So I created a VM with Fedora 22 in less than a minute. If you register with them you can run your instance for free for 7 days with a limitation about port 25 for email. They offer VNC client on their site to connect to the running VM. I also got connected using Putty and OpenSSL tools with a minimal configuration of the security keys. At some point, I could not find the Super user credentials for the VM but there was a message box with 24/7 online help even for the trial users. The operator responded instantly and gave me some hints.  The extra bonus for this cloud provider is the billing scheme they apply: they bill the usage of the resources not the resources. So you pay if you exceed your contract threshold per 5 minutes sampling. They utilize HTTP/HTTPS API for cloud management and Operations, a design that according to my opinion is the most flexible way to build your applications on top. 

From my quest for the holy cloud I think I made the correct decision with cloudsigma.com.

Tuesday, 15 July 2014

Pattern to Deliver Different Automation Templates per server group

The Problem: 
I have 3 groups servers that utilize different settings like LDAP, Apache config, Splunk. Each group has around 30 servers. Each of the configuration file for LDAP, Apache and Splunk does NOT have the same format, so a general automation Ruby template cannot be used for all three groups of servers. 

For example I cannot have a Splunk authentication.conf.erb for all groups like:
[default]

[Corporate AD]
bindDN = <%= @node['splunk']['ldap-bindDN'] %>
charset = utf8
bindDNpassword = <%= @node['splunk']['ldap-bindDNpassword'] %>
SSLEnabled = 0
port = 389
userBaseDN =  <%= @node['splunk']['ldap-userBaseDN'] %>
host =  <%= @node['splunk']['ldap-binddn'] %>

[authentication]
authType = LDAP
authSettings = <%= @node['splunk']['ldap-authSettings'] %>

# Here the splunk Stanga is always different for all 4 group of servers!!!
[roleMap_Corporate]
admin = wewvffsf3f
myreporting = 0110052012E
power = 0110052012E;0110052012E;0110052012E;0110052012E;

Question: 
How to apply automation for all four server groups by having templates of different formats ?

Solution:
 I give each server group a group id as an attribute:
node['splunk']['group-id'] = groupA or groupB or groupC 

Then in my Chef project I organize my templates folder as follows:

Contents of  my-chef-project/templates/default
  • groupA-authentication.conf.erb : describes LDAP settings for Group A
  • groupA-authorization.conf.erb : describes Splunk Authorization settings for Group A
  • groupB-authentication.conf.erb : describes LDAP settings for Group B
  • groupB-authorization.conf.erb : describes Splunk Authorization settings for Group B
  • groupC-authentication.conf.erb : describes LDAP settings for Group C
  • groupC-authorization.conf.erb : describes Splunk Authorization settings for Group C
Each of those templates is bare simple text without any parameters or anything else except perhaps node IP, hostname... See an example groupA-authentication.conf.erb :
[default]

[Corporate Settings]
bindDN = CN=splunk,OU=Services,OU=Company Page,OU=Resources,DC=illumine,DC=gr
SSLEnabled = 1
port = 437
host = ldap.illumine.com
client =  <%= @node['ip'] %>

[authentication]
authType = LDAP
authSettings = Corporate Settings

[roleMap_Corporate kl]
admin = nottellingya
blog = 0110341333450057252012E
puser = 0110003532234123412342012E;0110003532234123412342012E;0110003532234122412342012E

In my automated delivery chef recipe for any type of those templates I do something like the following chef ruby illustrates:

  template "/opt/splunk/etc/system/local/authentication.conf" do
    source "#{node['splunk']['group-id']}_authentication.conf.erb"
    owner 'splunk'
    group 'splunk'
    mode 0600
    variables()
    ignore_failure true
  end

Note that:

  • The template that is sourced is bound to the server´s group ID. 
  • Any server is the group will take the same group template. 
  • The parameter ignore_failure true denotes that if a template is not found for this group-id then no configuration is delivered and Chef automation will continue without brake.

Friday, 14 March 2014

Concurrent mode failure: Tuning JVM GC for Solr

The machine
I have an 8 CPU VM server with 32GB RAM running Solr. My JVM is 1.6.0_37 with the following JVM settings:
-Xms28g
-Xmx28g
-XX:NewSize=6g
-XX:MaxNewSize=6g
-XX:SurvivorRatio=4
-XX:PermSize=512m
-XX:MaxPermSize=512m
-XX:SoftRefLRUPolicyMSPerMB=500
-XX:+PrintCommandLineFlags
-XX:+HeapDumpOnOutOfMemoryError
-XX:+DumpGCHistoryOnOutOfMemory
-XX:+DumpDetailedClassStatisticOnOutOfMemory
-XX:HeapDumpPath=/opt/alfresco/tomcat/dumps
-verbose:gc
-Xloggc:/opt/alfresco/tomcat/dumps/gc-logs/gc-2014-03-13-10-00-07.log
-XX:+GCHistory
-XX:+CMSClassUnloadingEnabled
-XX:+DisableExplicitGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC

The reason for such huge heap is that the Solr data are about 130 GB and Sorl is heavily utilized from around 100 concurrent threads performing text search on documents.
I notice that Sorl application pauses for some time without responding. I discovered on the GC logs the following problem:
2014-03-17T14:29:23.438+0100: 7991.661: [GC2014-03-17T14:29:23.438+0100: 7991.661: [ParNew (promotion failed)
Desired survivor size 268435456 bytes, new threshold 15 (max 15)
- age   1:  125233576 bytes,  125233576 total
: 2621440K->2228808K(2621440K), 5.9399450 secs]2014-03-17T14:29:29.378+0100: 7997.601: [CMS2014-03-17T14:29:32.715+0100: 8000.938: [CMS-concurrent-sweep: 21.573/33.920 secs] [Times: user=118.80 sys=3.29, real=33.91 secs]
 (concurrent mode failure): 20465547K->11174034K(26214400K), 33.5774490 secs] 22674213K->11174034K(28835840K), [CMS Perm : 47873K->47648K(524288K)], 39.5176760 secs] [Times: user=46.19 sys=2.36, real=39.51 secs]

This issue is summarized in the official ORACLE documentation for JVM v6 as follows:

..a concurrent collection needs to be started at a time such that the collection can finish before the tenured generation becomes full; otherwise the application would observe longer pauses due to concurrent mode failure. There are several ways a concurrent collection can be started. 

See:  Concurrent Mode Failure

The message "concurrent mode failure" signifies that the concurrent collection of the tenured generation did not finish before the tenured generation became full. In other words, the new generation is filling up too fast, it is overflowing to tenured generation but the CMS could not clear out the tenured generation in the background. When a concurrent mode failure happens, the low pause collector does a stop-the-world (STW) collection. All the application threads are stopped, a different algorithm is used to collect the tenured generation (our particular flavor of a mark-sweep-compact), the applications threads are started again, and life goes on....

Seems that a concurrent mode failure is responsible for a "Stop the World" JVM pausing.
See also another wonderful Blog about the same issue here :

In order to treat problem we tune the following JVM flags:
-XX:CMSInitiatingOccupancyFraction=10
Indicating that a concurrent collection will start if the occupancy of the tenured generation exceeds 10% instead of 92% that is the default threshold.
-XX:CMSIncrementalSafetyFactor=100
Indicating to the JVM GC to start a concurrent collection at the next opportunity without any delay.
See also Oracles GC tunning instructions here