AndrewCode Blog

Using csshX to Manage Multiple Virtual Machines

I was looking for a cluster ssh tool to use to run updates and installs on multiple Virtual Machine servers. I had been having to ssh into each server individually, but with four servers this is already monotonous enough.

Searching the web I found Cluster SSH which looked cool, but I could not afford to use the X11 visual forwarding. The next tool that came up was a tmux script that would use tmux to synchronize tiles. Unfortunately tmux is not the easiest to learn, so I continued my search.

Then I came across csshX which is cluster ssh for OSX. This tool was very easy to use, first I created a file /etc/clusters. With this structure:

clustername user@ user@
custername2 user2@ user2@

Once I had my /etc/clusters set up I could call the command by entering csshX clustername. When you run this command you can control multiple shells easily.


Check out csshX for simple management of servers using OSX

Finding the Semaphore Source Code

Finding the Semaphore source code in the kernel was quite easy, I found it by running the command find . -name semaphore.c. You can find the semaphore code in linux/kernel/locking/semaphore.c and the header files in linux\include\kernel\semaphore.h.

The main semaphore struct is located in semaphore.h

struct semaphore {
    raw_spinlock_t      lock;
    unsigned int        count;
    struct list_head    wait_list;

The function to lock acquire the semaphore is the down(struct semaphore *sem) function. The function to release the function is up(struct semaphore *sem).

The down() function is depreciated for these other functions:

  • down_interruptible(struct semaphore *sem) - acquire the semaphore unless interrupted
  • down_killable(struct semaphore *sem) - acquire the semaphore unless killed
  • down_trylock(struct semaphore *sem) - try to acquire the semaphore, without waiting
  • down_timeout(struct semaphore *sem, long jiffies) - acquire the semaphore within a specified time

These additional functions will give the kernel engineers more functionality and control. Overall the semaphore source code was simple to find, but is a very important piece of code for the Linux kernel.

Least Amount of Threads to Start Fedora

In this OS experiment we were tasked in finding the minimum threads in Linux that the operating system will need to boot. I am using a Fedora 20 virtual machine running on VMWare Fusion 7.

The first thing I did was try to figure out how to start Fedora with no GUI. This was quite simple:

  • When the GRUB boot loader appears hit the arrow key
  • Click the 'e' key to edit the boot of your choosing
  • Scroll to the line that starts with 'linux'
  • At the end of that line, add the number '3'
  • Fedora should boot in command line mode

Fedora CL Boot

Command Line Boot on Fedora 20

Then I started to do some research, for example how to find the number of threads running? ps -eLf would do the trick but it would blow my screen up. Combining it with wc -l would give me the total number of lines counted.

Running ps -eLf | wc -l gave me a number of 391 current threads running.

I wanted to change the number of max threads to 400. Then increase the number of threads until I went over the maximum thread amount found in /proc/sys/kernel/threads-max.

To change the number of threads temporarily I ran echo 400 > /proc/sys/kernel/threads-max and started to create threads by executing the command sleep 1000 & over and over to get my thread count up to 400.

Once I hit the limit bash started to give me the error -bash: fork: retry: Resource temporarily unavailable

Fork Error

This wasn't really the main goal of the experiment, I was looking to see what type of error I would get when the max thread number was exceeded. I needed to go edit the maximum number of threads in the file fork.c of the kernel code. Then recompile the code, and select that kernel when the GRUB boot loader loads. I am going to set the number of max threads to 500 in my fork.c and see if it will even load.

I edited max_threads = 500; in fork.c and built the kernel. This worked in both the GUI and command line Fedora 20. So I decreased the max_threads to 100.

With 100 threads the GUI just stopped and hung at the loading screen. I would have to shut down my Fedora Virtual Machine and restart.

Fedora Stuck

When booting into the command line interface with 100 threads, things got interesting because the operating system booted in emergency mode.

Emeregency Mode

Setting 100 threads maximum the Command Line mode booted in emergency mode.

Once I exited the emergency mode I found that I logged into a shell, and was able to see that the total number of threads running in this emergency shell was 85.

When looking through the code in fork.c around line 278 it says

 * we need to allow at least 20 threads to boot a system
if (max_threads < 20)
    max_threads = 20;

I then decided to change the kernel code once again to max_threads = 0; to see if the kernel could truly load with only 20 threads.

Setting the max_thread count to 0 and then letting the kernel set it to 20 lead the kernel to panic. I had to "pull the plug" on my virtual and make sure to boot to the correct kernel or else it will panic while attempting to boot.

Kernel Panic

Changing the max_thread count to 20 lead to a kernel panic.

The exact number of threads needed to boot according to the kernel is 20, but I was unable to boot using that number. I had success in booting in to the emergency mode shell with 100 threads, but it seems like you will still need more to launch the command line smoothly.

Killing Pid Number One

In my Operating Systems class at Boise State we were given a experiment to run a loop that kills all the processes starting at Pid number one. So I wrote the program:

 * kill.c 
 * Andrew Gable
 * Sept 9, 2014
 * ****************/

#include <sys/types.h>
#include <signal.h>

int main()
    int i = 1;
    for(i; i < 2000000; i++)
        kill(i, SIGKILL);


The result was a brief black screen, then my login screen appeared. This will happen regardless of whether the program was run as normal user or super user.

Killing PID number one seems to kill the virtual and force a restart.

Beginning of the End

As I start back my last year I look at how I started at Boise State four years ago. In a new state and at a new school where I had no connections. Over the past four years I have learned so much it blows my mind.

From humble beginnings in my first programming class learning Java:

/** Hello my name is Andrew Gable and this is my first attempt at 
 *  a Java program. It is called Quote, and it will display one 
 *  of my favorite quotes. 

/** this is the public class */
public class Quote {

     * This is what will print out my favorite quote
    public static void main(String[] args) {
        System.out.println("From the Movie and Book \"Into the Wild\" ");
        System.out.println("If you want something in life, reach out and grab it.");
        System.out.println("\t-Christopher McCandless");


Fast forwarding to my last year taking Operating Systems, Parallel Processing, and a Cloud Computing course it is amazing how much I have progressed as a student and programmer. I am truly looking forward to continue my development with school, internship, and side projects that seem to always come my way.

I'd like to thank my family, friends, classmates, and teachers for helping me get to this great point in my academic career because without them I wouldn't be where I am. I've had Teaching Assistants help me study for tests on the weekends and professors respond to my programming questions during Spring Break. Boise State has created a great network of support that has helped me immensely.

I cannot wait to see what the future holds in store for software engineering as a whole and my future as a developer. I'm excited to keep continuing to learn as a developer and progress my career as a professional.