array-speed-test.c

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <sys/time.h>

void compare_arrays(int **screen1, int **screen2)
{
    int x;
    int y;
    int diff;
    int mult;

    for(x = 0; x < 1920; x++) {
        for(y = 0; y < 1080; y++)
        {
            mult = screen1[x][y] * screen2[x][y];
            diff = abs(screen1[x][y] - screen2[x][y]);
        }
    }
}

int main(int argc, char **argv)
{
    srand(time(NULL));
    printf("Generating arrays...\n");
    int **screen1;
    int **screen2;
    int i = 0;
    int j = 0;
    struct timeval now;
    struct timeval end;
    int usecs_passed;

    screen1 = malloc(1920 * sizeof(int *));
    screen2 = malloc(1920 * sizeof(int *));
    for(i = 0; i < 1920; i++)
    {
        screen1[i] = malloc(1080 * sizeof(int));
        screen2[i] = malloc(1080 * sizeof(int));
    }

    for(i = 0; i < 1920; i++)
    {
        for(j = 0; j < 1080; j++)
        {
            screen1[i][j] = rand() % 255;
            screen2[i][j] = rand() % 255;
        }
    }

    printf("Comparing arrays...\n");

    gettimeofday(&now, NULL);
    compare_arrays(screen1, screen2);
    gettimeofday(&end, NULL);

    usecs_passed = end.tv_usec - now.tv_usec;

    printf("Time passed: %dms\n", (usecs_passed / 1000));
    for(i = 0; i < 1920; i++)
    {
        free(screen1[i]);
        free(screen2[i]);
    }

    return 0;
}

And my python code:

array-speed-test.py

#!/usr/bin/python
import random
import time
from math import fabs

def generateArray():
    array_to_gen = [None] * 1920
    for i in range(0, 1920):
        array_to_gen[i] = [None] * 1080

    for x in range(0,1920):
        for y in range(0, 1080):
            array_to_gen[x][y] = random.randrange(0,255)

    return array_to_gen

def compareArrays(screen1, screen2):
    for x in range(0, 1920):
        for y in range(0, 1080):
            diff = fabs(screen1[x][y] - screen2[x][y])
            combo = screen1[x][y] * screen2[x][y]

if __name__ == "__main__":
    print "Generating arrays..."
    screen1 = generateArray()
    screen2 = generateArray()

    print "Created two screens.  Comparing..."
    startTime = time.time()
    compareArrays(screen1, screen2)
    print "Time taken: " + str((time.time() - startTime) * 1000) + "ms"

So far, the C program runs in 25ms, while the python program consistently takes 1100ms. Might have to ditch python for real time analysis, unless someone wants to point out how I am doing this completely wrong (I am assuming the comments will be use Numpy?)

Related posts:

1. Python threads
2. Why?
3. Python Commands Module

Here is how to limit the command to running for roughly 5 seconds:

#!/usr/bin/env python
import commands
import datetime
import time
import threading

class runCommand(threading.Thread):
    def run(self):
        status, output = commands.getstatusoutput('sleep 8')

if __name__ == "__main__":
    startTime = datetime.datetime.now()
    maximumRunTime = datetime.timedelta(seconds = 5)
    mythread = runCommand()
    mythread.start()

    while 1:
        if mythread.isAlive():
            print "Thread has not returned yet..."
        else:
            print "Thread is all done."
            break

        if (datetime.datetime.now() - startTime) > maximumRunTime:
            print "Thread has ran too long, giving up."
            break

        time.sleep(1)

    print "Program done"

So when we enter main, I assign the current time to startTime. Then I define maximumRunTime as a difference of 5 seconds. The thread is started, and every second we check to see if it is still running. If it is, we go ahead and check how much real time has elapsed.

Related posts:

1. Python Commands Module
2. Python and real time graphical analysis
3. Cherokee / Django tip: Timeout value

As long as at the top of your file you had import rpm that should work file. That gives a few basic fields. To get a full list of fields, I actually used a reference from a perl module: http://cpansearch.perl.org/src/RJRAY/Perl-RPM-1.51/RPM/Constants.pm.

If you expect more information, it really isn’t out there. Every mailing list post tells you to visit slides at people.redhat.com, on a webpage which no longer exists from 2004. I was able to find it again, so this may help you also: http://www.ukuug.org/events/linux2004/programme/paper-PNasrat-1/rpm-python-slides/toc.html. ActiveState’s website also has a snippet showing a bit of what is contained in that hdr class, which really helped trying to decipher this: http://code.activestate.com/recipes/576767/.

And for all future python programmers out there, THERE IS NO NEED TO ABBREVIATE VARIABLE NAMES. We have big computers, lots of memory, and naming things ‘hdr’ and ‘ts’ help nobody, and make your code horrible for others to use. Seriously, expand them out.

Related posts:

1. Creating graphs in python
2. Python Commands Module
3. For those with edgy python problems

Show me pretty pictures

Unfortunately, I believe I hacked my CairoPlot.py file to make the dots and pulled some stuff from other repositories, so the code might not give the same results. You may want to try the trunk. But the general point still exists that it is really easy to use, and the library is really easy to hack. To make the graph above I just ran:


#!/usr/bin/python
colors = [ (0.2, .3, .65), (0.5, 0.7, .1), (.35, .2, .45), ]
graphData = {}
graphData['server1'] = [ 20, 12, 42, 14, 11, 35 ]
graphData['server2'] = [ 18, 23, 10, 17, 23, 25 ]
CairoPlot.dot_line_plot("./graphs/blog", graphData, 500, 500, axis = True, grid = True, dots = True, series_colors = colors)


Obviously this is really easy to script, so you can parse your syslog files, append them to the graphData dictionary corresponding to the server, and bam you have a full report of everything happening etc. What I did was use the datetime module to sort events into time buckets that were then used as graphs, giving a view of 24 hours or whatever the period entered was.

Parsing syslog
I will give you a hand here too. In python, to parse syslog, I used a module called pyparsing. It uses a parsing language which is pretty easy to understand if you give it 20 minutes or so. Ie to parse the syslog lines I was looking for, I did the following:


month = Word(string.uppercase, string.lowercase, exact=3)
integer = Word( nums )
ipAddress = delimitedList( integer, ".", combine=True )
serverDateTime = Combine( month + " " + integer + " " + integer + ":" + integer + ":" + integer )
hostname = Word( alphas + nums + "_" + "-" )
daemon = Word(alphas) + Suppress("[") + integer + Suppress("]:")
ip = Suppress("remote IP address ") + ipAddress
bnf = serverDateTime + ipAddress + daemon + ip

for line in syslogFile:
try:
fields = bnf.parseString(line)
...


Keep in mind I did this very quick, so I am sure it can be refactored a bunch, just an example.

Related posts:

1. Python Commands Module
2. Python-RPM
3. Creating a virtual machine using ubuntu-kvm-builder

The Python module ‘commands’ allows commands to be ran on the system. I frequently use this in combination with Django to launch processes.

Get a directory listing:

#!/usr/bin/python
import commands
status, output = commands.getstatusoutput('ls')
print "Status: " + str(status)
print "Output: " + str(output)


This program will print out the directory contents of where you ran it from. An important thing to note about programs in Linux in general is that on success they return 0, and anything not 0 is probably an error. I commonly check the status of the returned code:

Check if status is not 0:

if status != 0:
    print "An error occurred!"


This allows us to do some really fancy things with this module. For instance, I created an application that is a django app that provides the user a menu to launch programs. This runs on their Firefox browser, but will launch the programs on their system. Each system runs it’s own Cherokee webserver locally, which is extremely lightweight and fast. To install Cherokee to use django I followed this guide: http://www.cherokee-project.com/doc/cookbook_django.html

The users run as the user ‘autologin’, however the webserver runs under the user ‘www’. This is probably a bit hackish, but it works. What I do is copy autologin’s .Xauthority file to the www user’s home directory, then launch the program as autologin using sudo.

Launch a local X program from a django program:

def launchAutologinXProgram(commands):
    status, output = commands.getstatusoutput('sudo cp /home/autologin/.Xauthority /home/www/')
    os.spawnl(os.P_WAIT, "/usr/bin/sudo", "sudo", "-u", "autologin", "-H", "/bin/bash", "-c", 'export DISPLAY=":0.0"; . /etc/profile; nohup ' + str(commands) + ' &')


Obviously you want to check the status variable and make sure it completed. To do this you need to update your sudoers file to make the www user be able to run commands as autologin. This is for a kiosk type setup, so www and autologin don’t have sensitive data, I am sure there is a more secure way to do it. But it sure is fun.

Related posts:

1. Python threads
2. The Cherokee Webserver
3. Python and real time graphical analysis