Friday 3 January 2020

Creating a Full Web Based Business

Hi folks! Merry Christmas to everyone!

One of my old clients asked me how to create a full web business site. the site should be able to do the common basic usual stuff:
  • Promote the company - Inform customers about the brand
  • Present products 
  • Do online sales
The interesting part of the story is that this guy did not want me to do the actual site work. Instead he asked me:
  •  to make a TODO list of ll basic steps with a correct order so that nothing is missing and everything is in order.  Write down a well formed procedure. 
  • also to put all those artifacts that should be delivered in each of the steps from the side of the implementing contract company, so that the contractor is checked - ensuring no hidden details
Read the full article  on Illumine IT Consulting GitHub Space

Thursday 17 October 2019

Elementary Linux Performance Monitoring

The basic tool here is top
Monitoring a single process can be done with -p option, in the next example we measure the MySQL process:

[root@(db-master) ~]# top -p 2521
top - 15:42:54 up 40 days, 10:46,  4 users,  load average: 0.14, 0.24, 0.48
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32551020 total, 32285684 used,   265336 free,   149660 buffers
KiB Swap:  3129340 total,   402572 used,  2726768 free. 16662620 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2521 mysql     20   0 18.725g 0.014t   4548 S 6.000 46.50   2735:03 mysqld

Load Average is a linux/unix mystery: Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. 
For an extended excellent article around Linux Load Average, refer to Brendan Gregg's Blog

On the other hand good old ps which is available on all UNIX flavors and LINUX distributions can also help. The following command shows the most CPU consuming processes  in ascending order along with their virtual size 

[root@(db-master) ~]# ps -e -o pid,pcpu,vsz,comm= | sort -n  --key=3
...
 1669  0.0 752396 isecespd
 1759  0.0 1561472 isectpd
 2521 52.4 19634584 mysqld

To get the process tree try pstree -aAl:

[root@(db-master) ~]# pstree -aAl
systemd --switched-root --system --deserialize 24
  |-VGAuthService -s
  |-agetty --noclear tty1 linux
  |-automount -p /var/run/automount.pid
  |   `-5*[{automount}]
  |-cron -n
  |-dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  |-discagnt /etc/init.d/discagnt start
  |   `-discagnt
  |-haveged -w 1024 -v 0 -F
...

For systems that do not have  pstree  try ps -ejH  

To get information about threads created by processes  try  ps -eLf

To get information about disk performance try iostat:

 [root@(mmcp_prod_corp)(db-master) ~]# iostat -dcm
Linux 4.4.121-92.117-default (mo-1400a55c2)     10/17/19        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.22    0.00    0.59    1.19    0.00   91.00

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               1.56         0.01         0.01      44144      51244
sdb             146.49         5.48         1.79   19159479    6250758

Finally to see all open files by a process such as data/shared objects/dynamic libraries and sockets use lsof. In the following example we can see all open files of mysql process:

[root@(db-master) ~]# lsof -p 2521
COMMAND  PID  USER   FD   TYPE             DEVICE     SIZE/OFF     NODE NAME
mysqld  2521 mysql  cwd    DIR              254,2         4096  6815769 /monsoon/mysql/data
mysqld  2521 mysql  rtd    DIR              254,0         4096        2 /
mysqld  2521 mysql  txt    REG              254,0    250387936   794500 /usr/sbin/mysqld
mysqld  2521 mysql  mem    REG              254,0        97056  1065145 /lib64/libresolv-2.22.so
mysqld  2521 mysql  mem    REG              254,0        26976  1065107 /lib64/libnss_dns-2.22.so


To see the TCP listening server sockets on a linux server, we can do that with netstat -tulpn

[root@(db-master) ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      2521/mysqld
tcp        0      0 0.0.0.0:2738            0.0.0.0:*               LISTEN      3282/discagnt
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3289/sshd
tcp        0      0 127.0.0.2:25            0.0.0.0:*               LISTEN      3671/master
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3671/master
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      38622/0
tcp        0      0 :::7938                 :::*                    LISTEN      3317/nsrexecd
tcp        0      0 :::5666                 :::*                    LISTEN      1/systemd
udp     4352      0 0.0.0.0:68              0.0.0.0:*                           1521/wickedd-dhcp4
udp        0      0 10.97.6.160:123         0.0.0.0:*                           3343/ntpd


while for all open TCP sockets:

[root@(db-master) ~]# netstat -t
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 mo-1400a55c2.zone:mysql mo-6740a22da.zone:46138 ESTABLISHED
tcp        0     64 mo-1400a55c2.zone1.:ssh mo-657dabf53.zone:58606 ESTABLISHED
tcp        0      0 mo-1400a55c2.zone:mysql mo-23acddcc0.zone:50068 ESTABLISHED





Friday 20 September 2019

Creating a RSA Key pair, a Self Signed Certificate and put it on a JKS Java Key Store

Generating a Key Pair (Private/Public key) and a Self-Signed Certificate and store them to a JKS Java Key Store 

Job done on a Linux box using the openssl tools and JDK 's keytool

1) Generate RSA key pair of 2048 bits
openssl genrsa -out illumineit.com.key 2048  

2) Generate certificate request for CA (.csr)
openssl req -x509 -sha256 -new -subj '/C=CY/ST=Nikosia/L=Center/CN=illumineit.com'  -key illumineit.com.key -out illumineit.com.csr

3) Generate self signed certificate expiry-time 10 years from the certificate request
openssl x509 -sha256 -days 3652 -in illumineit.com.csr -signkey illumineit.com.key -out illumineit.com.crt


4) Import the pair (private key and selfsigned certificate) in a new JKS (Trustore and Keystore together)
# Create PKCS12 keystore from private key and public certificate.
openssl pkcs12 -export -name illumineit.com -in illumineit.com.crt -inkey illumineit.com.key -out illumineit.com.p12 -passin pass:welcome -password pass:welcome

# Convert PKCS12 keystore into a JKS keystore
keytool -importkeystore -destkeystore illumineit.com.jks -srckeystore illumineit.com.p12 -srcstoretype pkcs12 -alias illumineit.com -srcstorepass welcome  -storepass welcome  -noprompt

Wednesday 19 June 2019

Retrieving the Posts and Pages from Wordpress Database.



Sometimes shit happens. Client took backup only the Wordpress DB without taking the PHP files. In other words, configuration, plugins, custom templates, skins and images.... just lost. Last Well Known good backup gone with the server.... and what we only got is a Wordpress DB without images. So practically, it would be a better idea to backup one by one pages from browser by clicking Save as.... Just jogging of course...




Now he has to write everything from scratch.

Step 1.

Examine your backup file:


-- MySQL dump 10.13 Distrib 5.5.55, for Linux (x86_64)
--
-- Host: localhost Database: wordpress9
-- ------------------------------------------------------
-- Server version 5.5.55




Step 2:

Go to MySQL and download the exact server version your previous installation comes from. Here is our link: https://dev.mysql.com/downloads/mysql/5.5.html

Install the MySQL temporarily in your PC or a simple VM or anything else.




Step 3:

Create a Database in your server just like the backup specifies:

C:\Users\>mysql -uroot -pmypass

mysql> CREATE DATABASE wordpress9 CHARACTER SET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.01 sec)







Step 4:

Copy your backup file. Edit your backup file

Restore your last backup in the server

C:\Users\>mysql -uroot -pmypass wordpress9 < backup_2019_01_18_1547817726_4347121_wpdb.sql



Step 5:

Create a file called restore.sql with the following query to retrieve your posts, pages and news:

select '<h1>',post_title,'</h1>',post_content, '<hr/> End Post <hr/>' from wp_posts where post_status='publish' and post_type in ('page','post','nooz_release') order by post_name, post_date ;

Step 6:
Run the query command as follows:
C:\Users\>mysql -uroot -pmypass wordpress9 < restore.sql > restored.html

The results are inside restored.html and can be viewed with a browser.
More careful backup next time....

Saturday 3 February 2018

Enable SSL for your Wordpress/Plesk site using a free authority-signed certificate

This article explains how to  replace HTTP with HTTPS on your site. This is an easy task if your site is relatively small and can be accomplished with 5 to 6 basic steps.

The article assumes site deployment with Wordpress and Plesk dashboards and suggests the creation of a free trusted authority signed SSL sertificate from  Comodo Cyber Security trusted authority which is valied for 3 months (90 days).
 
Read the overall article here:
Illumine IT Consulting GitHub Articles.

 

 

Tuesday 12 September 2017

disk-benchmark A mutlipurpose benchmark program that can simulate your application's I/O performance

disk-benchmark tool - get it here!

Sometimes we need to have a prior estimation of I/O performance of a program we plan to develop or we currently posses.
This may be triggered by a number of reasons:
  • Order specific Disk hardware in advance
  • Plan to rent cloud based volume from a cloud provider
  • Estimate the total performance of your application in order to establish operational scenarios and calculate KPIs.
  • Check the cloud providers SLA compliance.
In the past I dealt with all those challenges using standard Linux methods for benchamarking a volume like the classic one:


dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct

Or other similar methods or tools like iostat.

The problem with all those methods, is that you can have an idea of how your disk performs in general, but not according to a given scenario, for example:
  • 20 concurrent users each of them reads and writes of a random file of size between 20k and 1 MB with a pause of 2 seconds for 5 mins.
  • 10 concurrent users each of them reads/ writes a file of 60kb with a pause of 2 seconds after read repeatedly for 100 times. 


Unless you go to very sophisticated tools like JMeter,  you don't really have something very handy. On the other hand, sophisticated tools most of the times, have a significant learning curve but of course in most cases, you want something to use it in the next 5 mins with very simple options just like the above scenarios. To amend this situation, last year, I developed a small C program that can be used to do the job, the disk-benchmark program available on Illumine IT Consulting GitHub URL:

https://github.com/illumine/disk-benchmark

This is a benchmark program to test Hard Drives, SSD Drives, HBAs, RAID Adapters & Storage Controllers. This is a really simple C program that you can compile using the standard GNU/gcc compiler that comes with your Linux distribution.

How to setup the disk-benchmark in your Linux system:
Installation of the disk-benchmark is as simple as this:

# git clone https://github.com/illumine/disk-benchmark
# cd disk-benchmark/src/
# gcc disk-benchmark.c -o disk-benchmark  -l pthread -lrt  -O3  -Wall
# ls -l disk-benchmark
-rwxr-xr-x 1 root root 23365 Apr 15 10:23 disk-benchmark

A simple scenario implementation using disk-benchmark

Scenario: 10 concurrent users each writing and reading a file of size ~10MB in /var.  Each user pauses for some seconds randomly picked from the interval [2,10] sec.  The command that implements the above scenario has as follows:

[root@mo-8f752419d src]# ./disk-benchmark -p /var -t 10 -a 10000000 -E 2:10

Test scenario:
test path=/var
Threads=10, sleep sec between write/read = 1, repeats per thread=5, random pick sleep sec from [2 10]
Lower file size=1024, Upper file size=10240, Absolute file size=10000000
Read/Write buffer size=8192,  Buff Siz W 0, Buf Siz R 0,
Do write only=0, Delete files=1
Print values only=0 dont print scenario info= 0, dont print clocks=0 dont print headers=0 print date=1
Work Continously=0  Work Continously Sleep Brake=5

T=7, Avg W=0.016134 Avg R=0.002160 Total W=0.080671 Total R=0.010801 Total Time=0.091473 Sleep=4.600000  Avg File Size =10000000.000000
T=2, Avg W=0.014436 Avg R=0.002411 Total W=0.072179 Total R=0.012056 Total Time=0.084234 Sleep=4.800000  Avg File Size =10000000.000000
T=4, Avg W=0.016104 Avg R=0.002189 Total W=0.080520 Total R=0.010943 Total Time=0.091463 Sleep=4.800000  Avg File Size =10000000.000000
T=9, Avg W=0.011966 Avg R=0.002069 Total W=0.059829 Total R=0.010347 Total Time=0.070176 Sleep=4.800000  Avg File Size =10000000.000000
T=6, Avg W=0.013065 Avg R=0.001826 Total W=0.065323 Total R=0.009128 Total Time=0.074451 Sleep=5.000000  Avg File Size =10000000.000000
T=1, Avg W=0.015399 Avg R=0.003005 Total W=0.076996 Total R=0.015025 Total Time=0.092021 Sleep=5.200000  Avg File Size =10000000.000000
T=8, Avg W=0.012883 Avg R=0.002303 Total W=0.064416 Total R=0.011513 Total Time=0.075930 Sleep=5.200000  Avg File Size =10000000.000000
T=3, Avg W=0.015850 Avg R=0.002492 Total W=0.079251 Total R=0.012458 Total Time=0.091709 Sleep=5.400000  Avg File Size =10000000.000000
T=0, Avg W=0.013430 Avg R=0.002697 Total W=0.067151 Total R=0.013487 Total Time=0.080637 Sleep=5.600000  Avg File Size =10000000.000000
T=5, Avg W=0.016659 Avg R=0.002387 Total W=0.083293 Total R=0.011934 Total Time=0.095226 Sleep=5.600000  Avg File Size =10000000.000000

T=-1, Avg W=0.014593 Avg R=0.002354 Total W=0.072963 Total R=0.011769 Total Time=0.084732 Sleep=5.100000  Avg File Size =10000000.000000
Wall time 28.000000, CPU time 0.880000
Tue Sep 12 13:36:26 2017


Wednesday 18 May 2016

Web Service Client with Basic Authentication and SSL

Web Service Client with Basic Authentication and SSL

Recently, I had to create a web service client for a web service that uses a number of Web Service Policies. In general, the web service utilizes the following policies:
  • Transport: Service uses one way certificates. Client had to download and check server´s certificate in order to prove the server´s  identity.
  • Authentication: Basic authentication is required to access the URL and the service WSDL.
The following steps were used.
  • Creating the TrustStore: Access the Web Service URL, download the web service certificate and create a x509 trustStore to host the server´s certificate.
  • Create the client Stub: Access the Web Service URL and create the client stub by compiling the WSDL with wsimport.
  • Code and complete the service client. This has the following sub tasks:
    • Code the client to use Basic Authentication
    • Code the client to utilize the trustStore in order to setup SSL session with the server
    • Code the client to call the web method.

Creating the SSL Trustore.

During SSL handshake, the trustStore is used to verify server´s id.
Download the Server´s certificate by hitting the Web Service URL. There you will be prompted for login. You can login with the given user/password.

Then, the certificate is stored in your browser. You can export it easy but that depends to you browser. Chrome for example, the certificate can be downloaded directly as a x509 trustStore like the following image illustrates:



If you want to create the a trustStore manually you need to create a X509 keystore file using Java keytool and then import the server´s public certificate in it. The trustStore will be password protected and the certificate inside the trustStore will be password protected using "password" passphrase:

$ keytool -genkey -alias replserver -keyalg RSA -keystore mykeystore.jks -dname "cn=localhost, ou=IT, o=Continuent, c=DE"  -storepass password -keypass password

Now you have the keyStore. Next you need to import the server´s public certificate in it. In the general case, supposing the Server certificate is the following one plain text file server-certificate.txt then do one of the following actions to:

Check the server´s certificate:
openssl x509 -in server-certificate.txt -text -noout

Delete previous certificate version from the trustStore if any:
keytool -delete -alias myserver-name.com  -keystore mykeystore.jks 

Re-import the server certificate to the trustStore:
keytool -import -alias myserver-name.com -keystore mykeystore.jks  -file server-certificate.txt

Access the Web Service URL and create the client stub by compiling the WSDL with wsimport.

After running your wsimport command directly you should get a message complaining about a missing web authorization file.
What you need to do is create an authorization file (usually the default name/location for it is $HOME_DIRECTORY/.metro/auth, but check the previous error message, you'll get the hint from there).
Inside this file you just write the line: "https://username:password@url?wsdl"

 Now create a file called: wsimport_mysvc.bat and code the following commands:
setlocal
set _JAVA_OPTIONS=%_JAVA_OPTIONS% -Djavax.net.ssl.trustStore=mykeystore.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=mykeystore.jks
wsimport -s . -verbose -keep -p gr.illumine.wsclient.stub  -extension https://myserver-name.com/wsd/alc_interface?wsdl
endlocal

Doing so, you fulfill both conditions for basic authentication and also for transport/SSL by asking wsimport to examine what is been sent from server against to what is stored in mykeystore.jks

Run the wsimport_mysvc.bat and the client stub files will be created in the package gr.illumine.wsclient.stub

C:\>set _JAVA_OPTIONS= -Djavax.net.ssl.trustStore=cacerts -Djavax.
net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStore=cacerts

C:\>wsimport -s . -verbose -keep -p gr.illumine.wsclient.stub  -extension https://myserver-name.com/wsd/alc_interface?wsdl
Picked up _JAVA_OPTIONS:  -Djavax.net.ssl.trustStore=cacerts -Djavax.net.ssl.key
StorePassword=changeit -Djavax.net.ssl.trustStore=cacerts
parsing WSDL...

Code the client

The first thing you have to do is to add a static initializer that will provide the username and password for basic authentication:

public class AlcClient {
 
 private static final Logger log= Logger.getLogger( AlcClient.class.getName() );
 
 /* 
  * Use this static initializer to provide Basic Authentication for the Web Service Consumption
  */
 static {
     java.net.Authenticator.setDefault(new java.net.Authenticator() {

         @Override
         protected java.net.PasswordAuthentication getPasswordAuthentication() {
             return new java.net.PasswordAuthentication("happyuser", "mypassword".toCharArray());
         }
     });
 }

Next, configure your SSL settings in the code, by adding the following system parameters:
        /*
         * Use the following settings to specify how this client will utilize the X509 trust store
         * called mykeystore.jks. In this trustore, it is stored the server´s public certificate
         * Also the trustore/keystores are password protected with a password "password"
         */
        System.setProperty("java.protocol.handler.pkgs","com.sun.net.ssl.internal.www.protocol");
        System.setProperty("javax.net.ssl.keyStore","mykeystore.jks");
        System.setProperty("javax.net.ssl.keyStorePassword","password");
        System.setProperty("javax.net.ssl.keyStoreType", "JKS");
        System.setProperty("javax.net.ssl.trustStore","mykeystore.jks");
        System.setProperty("javax.net.ssl.trustStorePassword","password");
        System.setProperty("javax.net.ssl.trustStoreType", "JKS");

Then add some debugging options to debug your SSL session. You are strongly advised to comment out the following code after testing it since it will affect the SSL performance.
        /* Following options enable logging of all communication to the console
         * We are most interested in the request response SOAP Messages   */
        System.setProperty("com.sun.xml.ws.transport.http.client.HttpTransportPipe.dump", "true");
        System.setProperty("com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.dump", "true");
        System.setProperty("com.sun.xml.ws.transport.http.HttpAdapter.dump", "true");
        System.setProperty("com.sun.xml.internal.ws.transport.http.HttpAdapter.dump", "true");

Now code the Web Service client instance by using the stub you have created with the wsimport:
 ZALCINTERFACE_Service service = new ZALCINTERFACE_Service( new URL("myserver-name.com/wsd/alc_interface?wsdl"),
     new QName("urn:com:myserver-name:document:sap:soap:functions:mc-style", 
                      "ZALC_INTERFACE"));
  
 /*
  * From this service get the proper port
 */
 ZALCINTERFACE port = service.getZALCINTERFACE(); 

        /* Make the web service call */
        String responseMessage = port.callMyWebMethod();

Get the entire web service client java implementation can be downloaded here