Concurrent mode failure: Tuning JVM GC for Solr

The machine
I have an 8 CPU VM server with 32GB RAM running Solr. My JVM is 1.6.0_37 with the following JVM settings:

-Xms28g
-Xmx28g
-XX:NewSize=6g
-XX:MaxNewSize=6g
-XX:SurvivorRatio=4
-XX:PermSize=512m
-XX:MaxPermSize=512m
-XX:SoftRefLRUPolicyMSPerMB=500
-XX:+PrintCommandLineFlags
-XX:+HeapDumpOnOutOfMemoryError
-XX:+DumpGCHistoryOnOutOfMemory
-XX:+DumpDetailedClassStatisticOnOutOfMemory
-XX:HeapDumpPath=/opt/alfresco/tomcat/dumps
-verbose:gc
-Xloggc:/opt/alfresco/tomcat/dumps/gc-logs/gc-2014-03-13-10-00-07.log
-XX:+GCHistory
-XX:+CMSClassUnloadingEnabled
-XX:+DisableExplicitGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC

The reason for such huge heap is that the Solr data are about 130 GB and Sorl is heavily utilized from around 100 concurrent threads performing text search on documents.
I notice that Sorl application pauses for some time without responding. I discovered on the GC logs the following problem:

2014-03-17T14:29:23.438+0100: 7991.661: [GC2014-03-17T14:29:23.438+0100: 7991.661: [ParNew (promotion failed)
Desired survivor size 268435456 bytes, new threshold 15 (max 15)
- age   1:  125233576 bytes,  125233576 total
: 2621440K->2228808K(2621440K), 5.9399450 secs]2014-03-17T14:29:29.378+0100: 7997.601: [CMS2014-03-17T14:29:32.715+0100: 8000.938: [CMS-concurrent-sweep: 21.573/33.920 secs] [Times: user=118.80 sys=3.29, real=33.91 secs]
 (concurrent mode failure): 20465547K->11174034K(26214400K), 33.5774490 secs] 22674213K->11174034K(28835840K), [CMS Perm : 47873K->47648K(524288K)], 39.5176760 secs] [Times: user=46.19 sys=2.36, real=39.51 secs]

This issue is summarized in the official ORACLE documentation for JVM v6 as follows:

..a concurrent collection needs to be started at a time such that the collection can finish before the tenured generation becomes full; otherwise the application would observe longer pauses due to concurrent mode failure. There are several ways a concurrent collection can be started. 

See:  Concurrent Mode Failure

The message "concurrent mode failure" signifies that the concurrent collection of the tenured generation did not finish before the tenured generation became full. In other words, the new generation is filling up too fast, it is overflowing to tenured generation but the CMS could not clear out the tenured generation in the background. When a concurrent mode failure happens, the low pause collector does a stop-the-world (STW) collection. All the application threads are stopped, a different algorithm is used to collect the tenured generation (our particular flavor of a mark-sweep-compact), the applications threads are started again, and life goes on....

Seems that a concurrent mode failure is responsible for a "Stop the World" JVM pausing.
See also another wonderful Blog about the same issue here :

In order to treat problem we tune the following JVM flags:

-XX:CMSInitiatingOccupancyFraction=10

Indicating that a concurrent collection will start if the occupancy of the tenured generation exceeds 10% instead of 92% that is the default threshold.

-XX:CMSIncrementalSafetyFactor=100

Indicating to the JVM GC to start a concurrent collection at the next opportunity without any delay.
See also Oracles GC tunning instructions here

Overriding finalize() for reference count on active threads

The need for this small article came from the implementation of a custom thread controller. In my design I am using a ThreadGroup that holds threads that dynamically load classes and invoke methods of objects created on the fly.

The problem here comes with the ThreadGroup. I want to have the number of all threads in the group and we mean absolute number not something like myThreadGroup.activeCount() that shows only the active threads of the group. Normally ThreadGroup.activeCount() returns an estimate of the number of active threads in this thread group, so we cannot rely on the this.
One solution is to hold a container of references to all the new threads, but then I have to implement custom synchronized code for accessing the container and blah blah...

The solution I finally followed was to implement an object reference count on the parent class CustomPlugin. When a new object of any descendant class rooted from CustomPlugin, reference count is augmented. When a plugin exits execute() method, the curring thread exits run().
Plugin object reference terminated and destroyed by the JVM GC and the number of concurent plugins is decreased to something less than MAX_TOTAL_PLUGINS. Only then a new plugin can be created.

public abstract class CustomPlugin extends Thread implements Plugin{
 
 public static final int MAX_TOTAL_PLUGINS = 3;
 
 private static int refCount = 0;

 public CustomPlugin(String name) throws Exception{
    if(!allowLoad()){
       throw new Exception("Maximum plugin objects already loaded!");
    }
    Thread.currentThread().setName(name + "-" + Thread.currentThread().getId() );
    refCount++;
 }
 

 @Override
 protected void finalize(){
    try {
      super.finalize();
    } catch (Throwable e) {
      e.printStackTrace();
    }finally{
      refCount--; 
    }
 }

 public static boolean allowLoad() {
    return (refCount>=MAX_TOTAL_PLUGINS?false:true);
 }

 public abstract execute(String [] args);
}

Remember that you have only CustomPlugin.MAX_TOTAL_PLUGINS=3 available threads that you can use for your plugins, meaning that only MAX_TOTAL_PLUGINS can be used in total, regardless if their threads are sleeping! When a plugin exits execute() method, the curring thread exits run(). Plugin object reference terminated and destroyed by the JVM GC and the number of concurrent plugins is decreased to something less than MAX_TOTAL_PLUGINS. Only then a new plugin can be called.

Implementing 2-way SSL in Java using TLS and Self Signed Certificates part4

Debug the Client/Server Communication 


Entire Source Code of the tutorial here: 
https://github.com/illumine/articles/tree/master/Implementing-2-way-SSL-in-Java-using-TLS-and-Self-Signed-Certificates


This part depends on the:

• Communication credentials, or the Keystore/Trustore file created in Part-1
• Server implemented in Part-2
• Client implemented in Part-3

To start with we need to include on our JVM arguments on both client and server the following option:
-Djavax.net.debug=all

So for example to run the server we do:
$ java TwoWaySslServer -Djavax.net.debug=all

As a result, we have all the debugging information we need for network operations from the JVM and the imported classes, like SSLServerSocket.

Having our server in debug mode, we can observe that the Keystore (mysystem.jks) was loaded correctly if we notice the following private key log entry:

***
found key for : mysystem
chain [0] = [
[
  Version: V3
  Subject: CN=mysystem, L=Chalandri, ST=Athens, C=GR
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 1024 bits
  modulus: 127165120252867655295291767293201905001859144644390543231567027664179957281793248832544049047501722299712701237862474932181664724853946779349563166371011412260964043029373627517538842247060193170649833260910804612805979354599504164270912367917881965338674535760796311997608873587262396297225200721624071184029
  public exponent: 65537
  Validity: [From: Wed Jan 22 12:46:44 CET 2014,
               To: Mon Jan 22 12:46:44 CET 2024]
  Issuer: CN=mysystem, L=Chalandri, ST=Athens, C=GR
  SerialNumber: [    af1b0164 bd3095fa]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 03 B2 09 B4 89 36 32 E4   D6 A3 51 4A 5D 3B CD ED  .....62...QJ];..
0010: B2 00 77 EC                                        ..w.
]

]

[2]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 03 B2 09 B4 89 36 32 E4   D6 A3 51 4A 5D 3B CD ED  .....62...QJ];..
0010: B2 00 77 EC                                        ..w.
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 78 58 C9 F5 C0 42 2D 62   B5 0D 8A 79 6B 57 5A 85  xX...B-b...ykWZ.
0010: 8C 85 20 4D 7B B3 8A 0A   DF 83 D9 D1 5A FA F6 26  .. M........Z..&
0020: 53 56 DB FE B3 82 42 35   0C BF E8 BD 75 0A 18 7A  SV....B5....u..z
0030: D7 B0 36 E5 4E F9 82 FB   23 57 EC 23 3F D0 92 9E  ..6.N...#W.#?...
0040: 9C D6 FA 26 32 7C B6 4A   62 A4 4B AB F7 D3 64 7C  ...&2..Jb.K...d.
0050: 37 92 ED F2 2B 62 BC E7   A6 35 E6 87 67 9E BD 0D  7...+b...5..g...
0060: 97 5E 0F 31 A9 B1 AB 64   CC F9 4B 51 3E 90 7B 2F  .^.1...d..KQ>../
0070: E9 2E 23 E5 BC D3 DA 32   20 3B 6C 2C B8 E2 7C 6B  ..#....2 ;l,...k

]

If we don´t find our private key in the above debugging information, we need to check again the parameters:

  System.setProperty("javax.net.ssl.keyStore","mysystem.jks");
  System.setProperty("javax.net.ssl.keyStorePassword","welcome");

Similarly again in debugging mode, we can observe that the Trustore (mysystem.jks) was loaded correctly from server if we notice the following log entry:

***
trustStore is: mysystem.jks
trustStore type is : jks
trustStore provider is : 
init truststore
adding as trusted cert:
  Subject: CN=mysystem, L=Chalandri, ST=Athens, C=GR
  Issuer:  CN=mysystem, L=Chalandri, ST=Athens, C=GR
  Algorithm: RSA; Serial number: 0xaf1b0164bd3095fa
  Valid from Wed Jan 22 12:46:44 CET 2014 until Mon Jan 22 12:46:44 CET 2024

trigger seeding of SecureRandom
done seeding SecureRandom
SERVER
 socket class: class com.sun.net.ssl.internal.ssl.SSLServerSocketImpl
 Socker address = 0.0.0.0/0.0.0.0
 Socker port = 8095
 Need client authentication = true
 Want client authentication = false
 Use client mode = false

If we don´t find the self signed certificate key in the above debugging information, we need to check again the parameters:

System.setProperty("javax.net.ssl.trustStore","mysystem.jks");
System.setProperty("javax.net.ssl.trustStorePassword","welcome");

Some really interesting Exceptions you may come across while debugging your application:

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
Caught on Server
Probably you have created the client socket with something like:

clientSocket = new Socket();

Instead of :

SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
clientSocket = (SSLSocket) factory.createSocket("localhost",port);

java.lang.IllegalStateException: KeyManagerFactoryImpl is not initialized
You have not initialized KeyManagerFactory object, meaning that you should call method init() with a valid KeyStore and Certificate Password:

  String keystoreFile = "/opt/mysystem/etc/mysystem.jks";
  String keystorePassword = "welcome";
  KeyStore ks = KeyStore.getInstance("JKS");
  ks.load(new FileInputStream(keystoreFile), keystorePassword );
  KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
  kmf.init(ks,"myCertificatePassword");

java.security.cert.CertificateException: No X509TrustManager implementation avaiable
Check the trustrore. Ensure that property javax.net.ssl.trustStore is set appropriately.

javax.net.ssl.SSLHandshakeException: null cert chain
Received on server
Check that both Client and server share the trustore.
Check that trustore contains the same signed certificate

java.io.IOException: Keystore was tampered with, or password was incorrect
Check the Keystore password, system property javax.net.ssl.keyStorePassword

Implementing 2-way SSL in Java using TLS and Self Signed Certificates part3

Step 3: The Client (Get the complete code here)


Entire Source Code of the tutorial here: 
https://github.com/illumine/articles/tree/master/Implementing-2-way-SSL-in-Java-using-TLS-and-Self-Signed-Certificates

The client also requires the Keystore/Trustore created in Part-1

Again in the client we have to do a couple of things similar to the server:

The first is to specify the Java Keystore/Trustore we created in  Part-1 of this article:

System.setProperty("javax.net.ssl.keyStore","mysystem.jks");
System.setProperty("javax.net.ssl.keyStorePassword","welcome");

System.setProperty("javax.net.ssl.trustStore","mysystem.jks");
System.setProperty("javax.net.ssl.trustStorePassword","welcome");

Similarly with the server side described in Part-2, we have to create the client socket as an SSLSocket:

SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();    
SSLSocket sslSock = (SSLSocket) factory.createSocket("localhost",8095);

The entire code of the client can be downloaded here.

Next article, Part-4, of this Blog series will assist you to debug the SSL/TLS client/server communication.

Implementing 2-way SSL in Java using TLS and Self Signed Certificates part2

Step 2: The server (Get the complete server code here)


Entire Source Code of the tutorial here: 
https://github.com/illumine/articles/tree/master/Implementing-2-way-SSL-in-Java-using-TLS-and-Self-Signed-Certificates


Requires the Trustore/Keystore created in Step-1.

To write the server process in Java is pretty simple. You just have to do a couple of steps:
Specify a couple of properties so that the Trustore/Keystore can be loaded like the following code fragment shows:

System.setProperty("javax.net.ssl.keyStore","mysystem.jks");
System.setProperty("javax.net.ssl.keyStorePassword","welcome");

System.setProperty("javax.net.ssl.trustStore","mysystem.jks");
System.setProperty("javax.net.ssl.trustStorePassword","welcome");

Create the ServerSocket as anSSLServerSocketlike the following code fragment shows:

 
char ksPass[] = "welcome".toCharArray();
char ctPass[] = "welcome".toCharArray();

//Create and load the Keystore
KeyStore ks = KeyStore.getInstance("JKS");
ks.load(new FileInputStream("ianalyzer.jks"), ksPass);
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(ks, ctPass);

//Create the ServerSocket as an SSLServerSocket
SSLContext secureSocket = SSLContext.getInstance("TLS");
secureSocket.init(kmf.getKeyManagers(), null, null);
SSLServerSocketFactory ssf = secureSocket.getServerSocketFactory();
ssocket = (SSLServerSocket) ssf.createServerSocket(8095);
SSLServerSocket ss = (SSLServerSocket) ssocket;

//This explicitly states TLS with 2-way authentication
ss.setNeedClientAuth(true); 

The entire code for server implementation can be downloaded here.

Implementing 2-way SSL in Java using TLS and Self Signed Certificates part1

Consider that we want to implement in Java a secure communication (Transport Layer Security ) for a system called MySystem.

Entire Source Code of the tutorial here: 
https://github.com/illumine/articles/tree/master/Implementing-2-way-SSL-in-Java-using-TLS-and-Self-Signed-Certificates

The problem

The security scenario for the implementation of  MySystem is simple:

• Authentication only between peers that both share the Keystore/Trustore file
• Session establishment only between peers that have the Keystore/Trustore file

Doing so, the entire communication between client and server requires authentication and is encrypted:

Before going further on this study, pay a visit to this site for Java SSL: ssljavaguide.

To implement the scenario, there are three basic steps:

1. Create the Java Keystore/Trustore that will be used for Authentication and Encryption of Transport/Session. This will be used from both Client and Server parties. (Current Part)
2. Implement the Client side: (See blog article Part-2)
3. Implement the Server side: (See blog article Part-3)

Part-4 deals with debugging the Client/Server SSL/TLS communication.

Step 1: Create the Keystore/Trustore
Following steps of this section, results in the creation of a  Keystore/Trustore .jks file that contains:

• MySystem Private key 
• MySystem Selfsigned Certificate

To do so we are going to use the tools openssl  and keytool. We prefer using openssl because it can work silently - without prompt the user to put passwords, domains, server names....

The steps are:
1) Generate RSA 1024 bit private key. The key will be password protected:
openssl genrsa -out mysystem.key 1024 -passin pass:welcome

2) Generate Certificate Request for CA (.csr) using the private key
openssl req -x509 -sha256 -new -subj '/C=GR/ST=Athens/L=Chalandri/CN=mysystem'  -key mysystem.key -out mysystem.csr

3) Generate self signed certificate expiry-time 10 years from the certificate request
openssl x509 -sha256 -days 3652 -in mysystem.csr -signkey mysystem.key -out mysystem.crt


4) Import the pair (private key and selfsigned certificate) in a new JKS (Trustore/Keystore together)
First we need to create PKCS12 keystore from private key and self signed certificate.
openssl pkcs12 -export -name mysystem -in mysystem.crt -inkey mysystem.key -out mysystem.p12 -passin pass:welcome -password pass:welcome

Then we need to convert PKCS12 keystore into a JKS keystore
keytool -importkeystore -destkeystore mysystem.jks -srckeystore mysystem.p12 -srcstoretype pkcs12 -alias mysystem -srcstorepass welcome  -storepass welcome  -noprompt

At this point we have created the Java  Keystore/Trustore mysystem.jks file.

Copy mysystem.jks on both client and server machines.

Download all the commands for the Keystore/Trustore .jks file generation here

MySQL Master Slave Synchronization

In MySQL when Master/slave synchronization brakes, we can start the debugging with show slave status command like the following example illustrates:

[root@my-db-slave ~]# echo "show slave status\G" | mysql -u root -ppass
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: my-db-master.mydomain.com
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: binlog.000522
          Read_Master_Log_Pos: 50224067
               Relay_Log_File: relay.000005
                Relay_Log_Pos: 10328614
        Relay_Master_Log_File: binlog.000520
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1032
                   Last_Error: Could not execute Delete_rows event on table mydatabase.example_table; Can't find record in 'example_table', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000520, end_log_pos 10330060
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 10328471
              Relay_Log_Space: 259941828
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1032
               Last_SQL_Error: Could not execute Delete_rows event on table mydatabase.example_table; Can't find record in 'example_table', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000520, end_log_pos 10330060
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 49


On Slave MYSQL_HOME/logs/mysql.err this can be shown as:

131108 18:22:55 [Note] Slave I/O thread: connected to master 'replication@my-db-master.mydomain.com:3306',replication started in log 'binlog.000522' at position 50224067
131109  2:26:19 [ERROR] Could not execute Delete_rows event on table mydatabase.example_table; Can't find record in 'example_table', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binlog.000520, end_log_pos 10330060, Error_code: 1032
131109  2:26:19 [Warning] Slave: Can't find record in 'alf_node' Error_code: 1032
131109  2:26:19 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'binlog.000522' at position 50224067

There are several treatments for this issue that depend on what is the state of binlogs on master and how long the slave has been out of synchronization.
The following were tested on  MySQL 5.5.31 /  RedHat  2.6.32-358.2.1.el6.x86_64

The simple case : Restart Slave 

When synchronization fails between MASTER/SLAVE due to some error, normally the slave retries to reconnect after MASTER_CONNECT_RETRY seconds.
Slave will try to reconnect MASTER_RETRY_COUNT times.

In some cases, when synchronization lost within a small time frame and not many updates have been to the Master, then a simple "stop slave" / "start slave" may restore sync:

On Slave:
[root@my-db-slave ~]# echo "stop slave" | mysql -u root -ppass
[root@my-db-slave ~]# echo "start slave" | mysql -u root -ppass
[root@my-db-slave ~]# echo "show slave status\G" | mysql -u root -ppass

Resetting the Slave (Ignore Master Updates )

Another treatment is to reset the Slave to follow the master in the future: doing this you are lossing all updates from master and you just want slave to ignore all changes happened to the master while slave was out of sync!

On Master
[root@my-db-master ~]#echo "SHOW MASTER STATUS" | mysql -uroot -ppass 

On Slave:
[root@my-db-slave ~]#Take from master the binlog and pos
[root@my-db-slave ~]#echo "slave stop"| mysql -uroot -ppass
[root@my-db-slave ~]#echo "CHANGE MASTER TO MASTER_HOST='my-db-master',MASTER_USER='replication', MASTER_PASSWORD='apassword', MASTER_LOG_FILE='binlog.001006', MASTER_LOG_POS=  102018129;"| mysql -uroot -ppass
[root@my-db-slave ~]#echo "start slave"| mysql -uroot -ppass
[root@my-db-slave ~]#echo "SHOW SLAVE STATUS\G" | mysql -uroot -ppass  

Forcing Slave to get binlog files from Master

Sometimes, it might help to force slave get the binlogs again from the master. This will work if master has not rotate its binlogs: in other words if master have not started writting the binlogs again. That may happened because for example someone issued a RESET MASTER  on master. To force SLAVE retrieve the binlogs from start do the following steps:

On Slave:
Stop slave & Database
[root@my-db-slave ~]#echo "slave stop"| mysql -uroot -ppass
[root@my-db-slave ~]#/etc/init.d/mysql stop

Be sure that you have enough space when you do the next step:
[root@my-db-slave ~]#rm -rf $MYSQL-HOME/relaylogs/* 
[root@my-db-slave ~]#rm -rf $MYSQL-HOME/mysqld-relay-bin.*  $MYSQL-HOME/mysqld-relay-bin.index  $MYSQL-HOME/relay-log.info
[root@my-db-slave ~]#/etc/init.d/mysql start
[root@my-db-slave ~]#echo "stop slave" | mysql -u root -ppass
[root@my-db-slave ~]#echo "reset slave" | mysql -u root -ppass
[root@my-db-slave ~]#echo "start slave" | mysql -u root -ppass

Restore Slave from clear export from Master

Use this when you cannot restore the slave with any of the above ways.


On Master:
I need only mydatabase database, do not care about the others. Even if mydatabase db is in usage from application clients:

Take the dump of current Master database
[root@my-db-master ~]# sudo -i
[root@my-db-master ~]# cd /dbexport/
[root@my-db-master ~]# nohup mysqldump --master-data  -u root  -ppass mydatabase > mydatabase.dump &

With the previous command mysql dumps mydatabase database to the mydatabase.dump by:
Reserving the master binlog  status at the moment of backup transaction
Locking each backuped table in mydatabase, so it corresponds to the binlog state.

Note that: with the previous command I export only one database instance : mydatabase
In case your server hosts more that one databases and those are replicated on different slaves, you should consider taking an export for them also.

Check the last command output and also the dump status:
[root@my-db-master ~]# head mydatabase.dump
[root@my-db-master ~]# tail mydatabase.dump

Put a new password for trasport the dump
[root@my-db-master ~]# passwd

On Slave:
[root@my-db-slave ~]# sudo -i
[root@my-db-slave ~]# cd /monsoon/dbexport/
[root@my-db-slave ~]# scp root@my-db-master:/dbexport/mydatabase.dump .

[root@my-db-slave ~]# echo "show slave status\G" |  mysql -u root -ppass | egrep "Slave_IO_Running|Slave_SQL_Running"
             Slave_IO_Running: No
            Slave_SQL_Running: No

[root@my-db-slave ~]# nohup mysql  -u root  -ppass mydatabase < mydatabase.dump &

[root@my-db-slave ~]#  cat nohup.out
--there must be nothing here--

[root@my-db-slave ~]# echo "start slave" |  mysql -u root -ppass
[root@my-db-slave ~]# echo "show slave status\G" |  mysql -u root -ppass | egrep "Slave_IO_Running|Slave_SQL_Running"
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes