My journal

You want to use pigz, really

2012-01-24T00:00:00+01:00

If you always looked for a faster gzip this blog post is for you.

Here is a quick test on a 1.5GB SQL dump.

21:22 (jerome~gimli) /tmp
% time gzip --best testfile.txt
gzip --best testfile.txt  123.48s user 1.35s system 98% cpu 2:06.19 total

21:25 (jerome~gimli) /tmp
% time gunzip testfile.txt.gz
gunzip testfile.txt.gz  8.47s user 1.73s system 62% cpu 16.410 total

21:25 (jerome~gimli) /tmp
% time pigz --best testfile.txt
pigz --best testfile.txt  132.76s user 1.83s system 188% cpu 1:11.40 total

21:27 (jerome~gimli) /tmp
% time unpigz testfile.txt.gz
unpigz testfile.txt.gz  5.94s user 2.46s system 66% cpu 12.695 total

So I believe it is time for you to :

apt-get|yum|port install pigz

Pigz official website : http://zlib.net/pigz/

Specialist or generalist ?

2012-01-11T00:00:00+01:00

I believe that it probably happened to you in the past where a potential customer tells you something like "You are not an expert with technology X then I am afraid I will not work with you".

That makes me think about the difference between being a specialist and a generalist. I had to work as a specialist (i.e an "expert") at my previous job as I worked for a software vendor so I was obviously expected to know the product deeply. The problem when you behave like this is that you only know and work with one technology. But this is what you are paid for after all.

After some time, you know the system so well that you have the feeling to get a giant hammer and suddendly everything looks like a nail. You stop thinking about where to put the nail in and how to put it. You just want to put a nail somewhere so your customer did not pay you for nothing and this is really bad.

This is a behaviour I already noticed with people who are experts in their field. When you start discussing about a problem there is alway a moment when you get a "Ho that is really simple, with technology X I could do it in 2 days" where technology X is obviously the one this person has a really a deep knowledge of.

In my opinion this is a bad idea to think like a specialist because the risk of overlooking the entire picture of the situation is too big. It is extremely important to get the big picture of a customer's situation tatooed into your mind first and then start discussing with your customer what you can do to help him. You also have to make it very clear that you work for your customer's customers (i.e final users) because if final users are happy with the system, your customer will be happy (everybody is someone's customer at some point).

Solving web performance issues is a perfect example of the generalist VS specialist behavior. Imagine your customer asks you to make his website faster because it is too slow. The specialist will most likely dive in his cumfort zone. If it's a frontend engineer he will tell you that your CSS are too heavy or that your JavaScript files should be loaded at the bottom of the page. If it is backend engineer if will probably dive into your PHP code or he will tell you that your website if obviously slow because it does not use NodeJS or MongoDB and you should rewrite the entire thing, because, you know, "MongoDB is web scale" ...

If you ask a generalist you should get more questions than answers first, like "is it a network problem (DNS, latency, anything else ?)", "is it a distribution problem ? (maybe the CDN used -if any- has problems), "is it a front-end problem (ie is the slowness feeling a pure perceived performance problem or a real one ?)", "is it a backend problem ? (maybe something is going wrong on the server or at the datastore leve?)". The key to these questions if to understand the problem first because when you do not understand a problem entirely chances are it will not be fixed, or worst il will be fixed at the wrong level of abstraction. Then he will ask you for metrics and logs, analyze this ton of informations and provide a couple of theories explaining why the website is slow. The next step is to prove this theories. If all of them are wrong then drain, wash and rince again an find what has been overlooked previously. Then you should get an answer that really fixes the problem(s).

There is one thing I noticed with generalists, they have a better capacity to dive into a manual or the source code of products they have to work with most likely because they do not live in a cumfort zone but are constantly switching between different problems during the day.

If I had to hire someone I think I would prefer a generalist to a specialist.

An alternative to lesscss: writing CSS with GNU m4

2012-01-10T00:00:00+01:00

I hate working with CSS so I decided to make it less of a pain by using GNU m4 with it so I can at least get variable support and basic math operations.

If you want variables and arithmetic support (and even more) you can use less CSS but GNU m4 provides basically the same thing and has been there for over a decade, furthermore it is available on all platforms. If you ever compiled a program there is a good chance you have already used GNU m4 without even knowing it so it is likely you already have it on your machine, so nothing more to install :)

Here if how you can use GNU m4 with your CSS:

testm4.css.m4 :

include(`/tmp/csscolors.m4')dnl
define(`GLOBAL_WIDTH', `600')dnl
define(`LESS_LARGE', `200')dnl
dnl
.text_red {
    color: COLOR_RED;
}
.text_white {
    color: COLOR_WHITE;
}
.text_black {
    color: COLOR_BLACK;
}
.text_green {
    color: COLOR_GREEN;
}
.text_yellow {
    color: COLOR_YELLOW;
}
.text_blue {
    color: COLOR_BLUE;
}

#div1 {
    /* '()' are required. m4 will not expand the macro otherwise */
    width: GLOBAL_WIDTH()px;
    border: 1px solid COLOR_RED;
}

#div2 {
    width: eval(GLOBAL_WIDTH - LESS_LARGE)px;
    border: 1px dotted COLOR_BLUE;
}

csscolors.m4 :

define(`COLOR_WHITE', `#FFFFF')dnl
define(`COLOR_BLACK', `#00000')dnl
define(`COLOR_RED', `#FF0000')dnl
define(`COLOR_GREEN', `#00CC00')dnl
define(`COLOR_YELLOW', `#FFFF00')dnl
define(`COLOR_BLUE', `#0000FF')dnl

testm4.html :

<html>
    <head>
        <link rel='stylesheet' href='testm4.css' type='text/css' media='all' />
    </head>
    <body>
        <span class="text_red">a</span>
        <span class="text_white">b</span>
        <span class="text_black">c</span>
        <span class="text_green">d</span>
        <span class="text_yellow">e</span>
        <span class="text_blue">f</span>
        <div id="div1">FOO</div>
        <div id="div2">BAR</div>
    </body>
</html>

Then you can build the final CSS file by running :

m4 testm4.css.m4 > testm4.css

Done :)

Apache not considered harmful

2011-11-30T00:00:00+01:00

There has been a bit of fuss recently about the (non) Git support at the ASF.

The ASF would be harmful because there is no Git support, but GitHub would be the new place to be right now. Really ?

Here is what the ASF does:

The Apache Software Foundation provides organizational, legal, and financial support for a broad range of open source software projects. The Foundation provides an established framework for intellectual property and financial contributions that simultaneously limits contributors potential legal exposure. Through a collaborative and meritocratic development process, Apache projects deliver enterprise-grade, freely available software products that attract large communities of users. The pragmatic Apache License makes it easy for all users, commercial and individual, to deploy Apache products.

To rephrase the statement above the ASF is about providing a framework so it is safe for developers to contribute to any project with no legal exposure. Where can I find a statement like this on GitHub ?

In my opinion the legal part of a project is extremely important as copyright issues can be a pain when you have to deal with them because some companies tend to think innovation as the amount of million dollars you can pay lawyers...

If I work on my small project nobody heard of and I get a minor pull request on Github I will accept it as the legal exposure is limited regarding IP (Intellectual Property).

If I work at a company, contributing to an open source project by providing patches then I should be asked to sign a CLA when I explicitly accept to transfer the copyright ownership to the ASF (or the entity hosting the project). This is what the ASF is about.

Yes there is (at the moment) no Git support at the ASF and yes it is sad and yes I hope the ASF will provide Git support (or any VCS that provides cheap branches) soon but that does not mean GitHub is the place to be for your open source project. GitHub is just a code repository providing Git support, that's what they do for a living and they do it really well, but GitHub will not protect you from any potential legal issue regarding IP.

Summary of what is provided at GitHub and the ASF:

Entity	VCS	Issue tracker	Wiki	Mailing-lists	Mailing list archives	Legal Support
GitHub	Git and SVN	Yes (kind of)	Yes	No	No	No
ASF	SVN, git mirror at GitHub	Yes, JIRA or Bugzilla	Yes	Yes	Yes	Yes

Interesting links:

Update

Julien informed me that Git is currently being tested for CouchDB, other projects might use it soon as well. :)

A crawler in 10 minutes

2011-10-18T00:00:00+02:00

Use case

You need to fetch a set of HTML webpages (no external resources like JS, CSS, images etc) either for data analysis or for forcing pages to be cached in your reverse proxy.

(You also have much more interesting things to do so you do not want to spend your life on that problem.)

Solution

(Well this is one solution among others)

Generate the list of URLs to crawl
Crawl 30 URLs in parallel

Tools

You need the following tools:

curl
cat
xargs
make

Putting it all together

Let's store the list of URLs to crawl in a plain text urls.txt.

Now we need to write the "crawler" crawl.sh :

#! /bin/bash
echo -n '.'
# Force the page to be cached
# We do not need to store the result
curl --silent --request GET -H "Accept-Encoding: gzip" $1|sed 's/&/\\&/g' > /dev/null

Let's make it usable and parallel now:

PARALELL_REQUESTS=30
SOURCE_FILE=url.txt

crawl:
    @echo "Crawling"
    @cat ${SOURCE_FILE} | xargs -n1 -P${PARALELL_REQUESTS} ./crawl.sh
    @echo "Done"

clean:
    @rm -f ${SOURCE_FILE}

Now run make crawl and you will get 30 pages fetched in parallel.

The key is to use the -P argument available in xargs. Here is what the xargs documentation says:

--max-procs=max-procs, -P max-procs
        Run up to max-procs processes at a time; the default is  1.   If
        max-procs is 0, xargs will run as many processes as possible at
        a time.  Use the -n option with -P; otherwise chances  are  that
        only one exec will be done.

Now you can spend your time on something more interesting like Nginx Developement Kit.

A better pager

2011-08-12T00:00:00+02:00

I was looking for an alternative to less and I found a pager called most which provides interesting features and colors out of the box.

Installing most is really trivial :

sudo apt-get|port|whatever install most
export PAGER=most

And try it with :

man ls

You should see a colorized man page :)

How to boot on the 64-bit version of the Mac Os X kernel

2011-08-06T00:00:00+02:00

Warning

What is explained below has been tested successfully on Mac Os X 10.6 (Snow Leopard). If you are a Lion user, you decide to use this tip at your own risk. I am not sure it is even needed as Lion normally provides 64bits support only.

Method

The method is the following :

Open /Library/Preferences/SystemConfiguration/com.apple.Boot.plist
Use the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Kernel</key>
    <string>mach_kernel</string>
    <key>Kernel Flags</key>
    <string>arch=x86_64</string>
</dict>
</plist>

Reboot the system.

In order to verify that the change has been applied, open a Terminal and run the following command:

uname -a

You should get something like this:

Darwin [...] root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64

Back from Denmark

2011-07-07T00:00:00+02:00

What to see ?

Legoland

View Larger Map

This is a must see! You can have a look at the pictures I took there so you can have an idea of what you will see. If you are a fan of Star Wars you will not resist to what you will see:

You should also go to the pirate splash battle where you have to attack visitors (also know as enemies during the time of the ride) with a water gun.

When you see the warning "you will get wet" at the beginning of the waiting queue you should understand "you will get soaked"!

I do not remember how much I paid for a ticket but you can order them online so you do not wait at the entrance of the park. I highly recommend you to buy tickets in advance. You will also have to pay 50 DKK (If I remember correctly) for the park.

Mariager

View Larger Map

Mariager is a small picturesque town. Just go down town an enjoy your time here by walking in the streets and admiring houses and the view on the Fjörd.

Do not forget to stop by the ice cream shop where they proudly prepare ice creams traditionally.

Here is a couple of pictures :

Skagen

View Larger Map

Skagen is at the very north of Denmark, there is not that much to see there but you can go to the very end of the beach so you can say "I did it, I am at the very north of Denmark". You can go downtown Skagen and get lunch at a restaurant on the port, the food is quite good, though a bit expensive.

Here is a couple of pictures :

Frederiksborg

Frederiksborg (Frederic's castle) is located in Hillerød.

View Larger Map

Frederiksborg is to Denmark what Versailles is to France, you should definitely visit it! Note that if you go there with a pushchair for your baby you will have to use one the provide in the castle. Once done with the visit there is not that much to see around but I recommend you to get a bagel downtown as they are excellent and you get a 10% discount if you order food to take away :)

Here is a couple of pictures :

Copenhagen

View Larger Map

I do not have much to say about Copenhangen since I visited it really quickly by walk and with the kids so I did not see half of what I wanted to see. What I would recommend is to spend a couple of days there, 3 would be OK I guess, and rent a bike. That way you can go anywhere you want really quickly. Parks are insanely expensive. It is best if you can avoid using a pushchair. You can rent bikes with two seats for children.

Roskilde

View Larger Map

Around 40 kms from Copenhagen. This is a very pleasant town where you can stay for a couple of days. I slept at Roskilde camping and I was really happy by the quality and the price. If you decide to stay there you can get the following view on the Fjörd :

When staying at Roskilde I highly recommend you visit the viking museum. You can see a couple of pictures I took there:

http://www.flickr.com/photos/qwix/sets/72157627138033098/

This is only when you discover how vikings lived and built their drakkars that you realize how smart they were.

Once you stay here you can go to Copenhagen by car in around 40 minutes (mind the traffic jam at around 9am !).

Food

In general food in Denmark is acceptable as long as you avoid eating at fast foods you will see everywhere. There you can get some very fat and poor quality food like hot dogs and french fries for almost nothing.

Danish people have a lot of different breads and they are all excellent. If you like fish you should definitely try herring which is prepared in different flavors, and of course salmon which tastes as good as herring. Cheese is good as well, there is quite a lot of different cheeses available I recommend you try all of them.

Do not bother trying (unless you really want to experiment) pastry, it is ugly. There is way too much sugar or cinnamon. You can try an ice cream with soft ice and you favorite toping but do not take too much ice creams during your trip unless you want to quickly see your blood turn into syrup. As an example, here is what you can buy :

For 40 DKK:

Or a Grand Danois for 200 DKK, only if you are brave enough:

You should definitely buy strawberries, you can buy some anywhere you want on the road as shown here http://www.flickr.com/photos/qwix/5912852371/in/photostream. You do not have to ask someone to buy them, just put 20 DKK (in average) in the box and take some strawberries.

People

People in Denmark are really nice though a bit shy (at least this is the impression I had) they speak a very good English but it is better if you speak some basic German. I met people (especially in the north of Denmark) who were much more comfortable with German and were only able to speak a couple of words in English. When you are in Copenhagen you can ask anything you want in English without any problem.

Money

Money used in Denmark is the Danish Crown. Overall, this country is not really expensive. Since the country is not in the Eurozone you should ask your bank how much you will be charged for each withdrawal and each payment with your credit card. If you need to change money during your trip do it only at banks as they seem to have the lowest rates.

Oil

If you have an LPG (car gas) car do not bother buying an adaptor (if you need to) for danish pumps I found no station where I could get LPG and I drove around 2000 kms in the country so I saw a lot of oil stations ...

Renting a house

I rented a house at Sol Og Strand as was very happy with ther services. You can find a lot of houses almost everywhere in Denmakr for acceptable prices. As an example, I rented a 80 m2 house for a week for around 500 Euros which is really cheap.

Living without google search

2011-06-13T00:00:00+02:00

Introduction

About a month ago I tried to experiment something different for searching on the Internet. I have always used google and was pretty happy with the search results but in my opinion Google is way too hegemonic (evil ?) and I do not like this. So I decided to try something else.

What did I used ?

Well this is quite simple, I have been using Ecosia for one month and the experience is quite interesting. Search results come from Bing and Yahoo and the relevance of returned results is good. I also tried Duck Duck Go but the search results between Ecosia and Duck Duck Go are pretty much the same so I decided to stay with Ecosia. For more "specialized" searches I use Firefox' search modules for sites I use every day, like the search module for Stack Overflow or the Python documentation.

What's next ?

Now that I can live google search free I decided to live Gmail free. I did not try any alternative yet, I think I will seriously look for an alternative once I am back from vacations. However, if you have any recommendation I would be happy to here from you :)

How to backup your Solr / Lucene index

2011-05-04T00:00:00+02:00

Replication

You can set up a replication system as described in the following page:

http://wiki.apache.org/solr/SolrReplication

What I appreciate about that system is the master is totally unaware of slaves so you can add as many slaves as you want transparently. But pay attention to performances when adding too much slaves as they will continously poll the master.

Simple copy

Solr already provides shell scripts for backup operations:

backup will create backups which name is the timestamp of when the backup was done. As you can imagine you can get a lot of backups, depending on the frequency you defined, you need a system to clean very old backups. This is what backupcleaner is made for. It will remove backups older that X days.

Those script (as well as others) are available here :

https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/src/scripts/

The problem with the backup script is that the backup will be local, if you want to get a distant backup you can use rsync and simply copy the index on another machine. As a Solr index is just a pile of files is it trivial to rsync them from a machine to another one.

Dumping the index to a file

This is equivalent to dumping a database. This can be useful for backups but also as a reference to know exactly what is available in your index or not, especially during development phases. In order to export the index to a file you need Luke :

download the lastest version (3.1.0).

run

java -cp lukeall-3.1.0.jar org.getopt.luke.XMLExporter /path/to/index /path/to/output.xml -gzip

After a moment (depending of course of the size of your index) you will get an XML file with the content of your index.

If you want to export ranges of documents, XMLExporter supports the -range X-Y syntax, that means that using -range 0-100 will only export 100 documents.

Run java -cp lukeall-3.1.0.jar org.getopt.luke.XMLExporter to get help.

A better cp with pv

2011-03-23T00:00:00+01:00

If you copy large files and hate not to know when the copy is going to be finished then this post will interest you.

Here is what you can do to fix that problem:

install pv (yum|port|apt-get install pv)

add the following function to you ~/.zsh|bash|*rc.

function pvcp()
{
    SOURCE=$1

    if [ -d $2 ]
    then
        DESTINATION=$2/`basename $SOURCE`
    else
        DESTINATION=$2
    fi

    pv ${SOURCE} | > ${DESTINATION}
}

source ~/.zsh|bash|*rc

You can now do:

pvcp <source> <destination>

And here is an example of what you should get:

pvcp /path/to/foo.avo /path/to/destination/
702MB 0:01:27 [8.01MB/s] [===============[...]================>] 100%

Installing Reconnoiter on Debian Squeeze

2011-03-14T00:00:00+01:00

If you want to know more about Reconnoiter, watch this demo.

This how-to is based on Marius Ducea's article HowTo install reconnoiter on Debian Lenny.

Requirements

Debian as in :

jerome@reconnoiter:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0 (squeeze)
Release:    6.0
Codename:   squeeze

non-free packages (for sun-java6-jdk) in /etc/apt/sources.list :

deb http://your.favorite.debian.mirror/debian/ squeeze non-free
deb-src http://your.favorite.debian.mirror/debian/ squeeze non-free

Do not forget to run sudo apt-get update once sources.list has been updated.

Installing required packages

The following packages are required:

sudo apt-get install autoconf build-essential zlib1g-dev uuid-dev \
libpcre3-dev libssl-dev libpq-dev libxslt1-dev libapr1-dev \
libaprutil1-dev xsltproc libncurses5-dev libssh2-1-dev \
libsnmp-dev sun-java6-jdk subversion postgresql-8.4 \
uuid-runtime apache2 libapache2-mod-php5 php5-pgsql \
libcurl4-gnutls-dev

Checking out and compiling

Reconnoiter must be built from code so let's use trunk:

cd
svn co https://labs.omniti.com/reconnoiter/trunk reconnoiter
cd reconnoiter
autoconf

Now you can configure the project, you may use the default ./configure or define an installation prefix by using the --prefix=/path/to/reconnoiter for a custom installation path. I used :

./configure --prefix=/opt/reconnoiter

Once it is configured you can start compiling, just run make. Before running make install you need to prepare a directory which will contain require java libraries :

sudo mkdir -p /path/to/prefix/java/lib/

Or if you did not specify any prefix for ./configure just run

sudo mkdir -p /usr/local/java/lib/

Once done, install the system by running make install.

Preparing Postgresql

In order to avoid any problem when running postgres operations, you have to update the file /etc/postgresql/8.4/main/pg_hba.conf on line 82. The line:

local   all         all                               ident

must become :

local   all         all                               trust

Or you can do it that way :

sudo sed -i "s/local   all         all                               ident/local   all         all                               trust/g" /etc/postgresql/8.4/main/pg_hba.conf

And restart postgres:

sudo /etc/init.d/postgresql restart

Switch to user "postgres" :

sudo su postgres
cd /path/to/svn/trunk/of/reconnoiter/sql

Open a postgres console:

psql

And import the DDL dump:

\i ./sql/reconnoiter_ddl_dump.sql
\q

Now return to your user's shell

exit

Go to /path/to/svn/trunk/of/reconnoiter/sql and run:

# fixes hard coded path to postgres
sed -i 's/\/opt\/psql835/usr/g' crontab

This will fix a hard coded path to postgres.

Now you can add the conjobs to the postgresql user:

sudo crontab -u postgres crontab

You can check it is OK by running :

sudo crontab -u postgres -l

Preparing SSL certificates

Ideally you need real certificates but for my testing purposes I used the ones that are available in the test/ directory. Just copy them to reconnoiter/installation/path/etc. If you did not configured any value for --prefix for ./configure then the following command should work :

cd /usr/local/src/reconnoiter/test
cp test-*.crt test-*.key /usr/local/etc/

In order to get completely ready we need to create a couple of symlinks in the etc/ directory. For example here is what I get in the end :

lrwxrwxrwx 1 root root     11 Mar 14 09:40 ca.crt -> test-ca.crt
-rw-r--r-- 1 root root   3037 Mar 14 09:32 config_templates.conf
-rw-r--r-- 1 root root 237476 Mar 14 09:32 default-ca-chain.crt
-rw-r--r-- 1 root root   6806 Mar 14 10:40 noit.conf
lrwxrwxrwx 1 root root     13 Mar 14 09:38 noit.crt -> test-noit.crt
lrwxrwxrwx 1 root root     13 Mar 14 09:39 noit.key -> test-noit.key
-rw-r--r-- 1 root root   7111 Mar 14 09:45 stratcon.conf
lrwxrwxrwx 1 root root     17 Mar 14 09:41 stratcon.crt -> test-stratcon.crt
lrwxrwxrwx 1 root root     17 Mar 14 09:40 stratcon.key -> test-stratcon.key
-rw-r--r-- 1 root root    615 Mar 14 09:37 test-ca.crt
-rw-r--r-- 1 root root    493 Mar 14 09:37 test-ca.key
-rw-r--r-- 1 root root   2489 Mar 14 09:37 test-noit.crt
-rw-r--r-- 1 root root    497 Mar 14 09:37 test-noit.key
-rw-r--r-- 1 root root   2484 Mar 14 09:37 test-stratcon.crt
-rw-r--r-- 1 root root    497 Mar 14 09:37 test-stratcon.key

I created the symlinks shown above because the paths to SSL configuration files available in noit.conf and stratcon.conf are not prefixed by test-. I believe you can rename the files as well. This is really a matter of preference.

Preparing configuration files

In the Reconnoiter's installation directory you will find noit.conf.sample and stratcon.conf.sample. Rename them to noit.conf and stratcon.conf.

noit.conf

The default noit.conf is ready to use so you should not need to update it right now. You will be able to change it later if you want, either by update the file directly or by using the CLI console (which I recommend you to use as it is really efficient and simple to use).

stratcon.conf

The default stratcon.conf needs the following change:

<dbconfig>
  <host>localhost</host>
  <dbname>reconnoiter</dbname>
  <user>stratcon</user>
  <password>unguessable</password>
</dbconfig>

must become

<dbconfig>
  <host>localhost</host>
  <dbname>reconnoiter</dbname>
  <user>stratcon</user>
  <password>stratcon</password>
</dbconfig>

The password is "stratcon" as defined by default in the DDL dump.

You also have to configure <hostname> and <document_domain> with appropriate values. As an example here is what I have:

<config>
    <hostname>localhost</hostname>
    <document_domain>localhost</document_domain>
</config>

Starting daemons

The noid and stratcond daemons must be run in the following order.

1. noitd

Start noitd in the foreground with debugging informations so you can spot any error message easily.

sudo /path/to/reconnoiter/sbin/noitd -c /path/to/reconnoiter/etc/noit.conf
-d -D

-d stands for "show debuggins informations" and -D stands for "run in the foreground".

If you did not use any --prefix when running ./configure then the command should be :

sudo /usr/local/sbin/noitd -c /usr/local/etc/noit.conf -d -D

If you want to run noitd as a normal process, just remove the -d and -D flags.

Note

If you run ps aux | grep "noitd" and notice two noitd processes, this is normal. This is because there is a management process for noitd.

2. stratcond

Start stratcond in the foreground with debugging informations so you can spot any error message easily.

sudo /path/to/reconnoiter/sbin/stratcond -c /path/to/reconnoiter/etc/stratcon.conf
-d -D

If you did not use any --prefix when running ./configure then the command should be :

sudo /usr/local/sbin/stratcond -c /usr/local/etc/stratcond.conf -d -D

As for noid remove the -d and -D flags to run stratcond as a normal process.

Now noitd and stratcond are running you should see quite a lot of output and everything should be OK. You can relaunch them, but in the background this time if you want.

Installing the web UI

Well that could not be any simpler.

Copy the ui directory in /var/www/.

Create the following VirtualHost, in /etc/apache2/sites-available/reconnoiter (adapt the path if needed) :

<VirtualHost *:80>
    DocumentRoot /var/www/ui/web/htdocs

    <Directory "/">
        Options None
        AllowOverride None
        Order allow,deny
        Deny from all
    </Directory>
    <FilesMatch "^\.ht">
        Order allow,deny
        Deny from all
        Satisfy All
    </FilesMatch>
    <Directory "/var/www/ui/web/htdocs/">
        php_value include_path /var/www/ui/web/lib
        php_value short_open_tag off
        Options FollowSymLinks Indexes
        AllowOverride All
        Order deny,allow
        Allow from all
    </Directory>

    LogLevel warn
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    ErrorLog /var/www/ui/web/logs/error_log
    CustomLog /var/www/ui/web/logs/access_log common

    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType application/x-httpd-php .php
    DefaultType text/plain

    RewriteEngine On
    RewriteRule ^(/data/.+)$ http://localhost:80$1 [P,L,QSA]
</VirtualHost>

Enable mod_rewrite and the reconnoiter VHost

sudo a2enmod rewrite
sudo a2ensite reconnoiter
sudo /etc/init.d/apache2 restart

Note

Just disable the default VHost for the moment by running sudo a2dissite default and restart Apache.

You should now get the Reconnoiter's web UI. Click on the small '+' button on the "worksheet control" line, you should see the "graph controls" panel open it and click on the "browse data" tab. Hopefully you should get results, it is now up to you to decide what you want to graph.

Troubleshooting

If you get the following error :

SSL read error: Input/output error

There is a problem with your certificates, check the path are correct in noit.conf and stratcon.conf. Maybe they are incorrect.

Hope that helps :)

Testing ESI without Varnish (or equivalent)

2011-02-22T00:00:00+01:00

Note :: If you are new to ESI and want to know a bit more about it you can read this previous blog post : About ESI

Use case

You decided to implement a feature that will use ESI but you do not want to install Varnish on your computer to avoid any cache during the development phase.

Solution

To use a bit of JavaScript to simulate the ESI processing. This is possible by using the Jay Kuri's ESI parser.

Please not that only <esi:include> and <esi:remove> are supported.

Using ESI parser is pretty straightforward:

download the tarball
extract it wherever you want in your project
include esiparser.js
include the following code in the page(s) you want to process:

$(document).ready( function () { do_esi_parsing(document); });

That should be it :)

Setting up a Solr sandbox for spotting problems

2011-01-27T00:00:00+01:00

Use case

You get problems with Apache Solr which sometimes does not return expected results. That can be anything, but when you have a very large index and when it takes a lot of time to (re)index data, spotting problems is very difficult and frustrating.

The methodology described below helped me a few times already so I thought I could share it. Any enhancements are of course much welcome :)

The method requires the following steps:

Creating a new Apache Solr sandbox
Exporting data from your storage and import them in the sandbox
Importing data and test anything you want

Creating the sandbox

You have to setup a sandbox where you can play with data safely. If you use a separate index you can do anything you want for testing and debugging without breaking anything in the real index.

In order to setup that sandbox I created a very trivial Makefile available below:

init: download extract

download:
    wget http://mir2.ovh.net/ftp.apache.org/dist//lucene/solr/1.4.1/apache-solr-1.4.1.tgz

extract:
    tar zxvf apache-solr-1.4.1.tgz

clean:
    rm -rf apache*

Just run make init and you will get a new Apache Solr instance.

Now the sandbox is installed you have to copy your schema.xml (and solrconfig.xml if applicable) in apache-solr-1.4.1/example/solr/conf/ so this new index has exactly the same structure as the real one.

Start Solr by running :

java -jar start.jar

The sandbox is now initialized and running.

Exporting data and importing them in this new index

Exporting your data depends on what you use, so I will not describe this part. However you have to export these data in XML in the following format:

<add>
    <doc>
        <field name="name_of_your_field_described_in_schema.xml">
            Anything you want here
        </field>
        <!-- [repeat for all fields you want to index] -->
    </doc>
    <!-- <doc>....</doc> -->
</add>

Note

You can run xmllint on the generated XML in order to be sure everything is OK.

Indexing data

Once you get an XML file containing a sample of your data you can index it in Apache Solr.

Go to apache-solr-1.4.1/example/exampledocs and run:

sh ./post.sh path/to/your/generated/xmlfile.xml

Done :)

Now you have a ready-to-debug index, if you need to reindex, just rerun the command shown above. You may also want to purge the index first. I created this shell alias, which will delete everything and commit the result :

alias solrpurgeindex='java -Ddata=args -jar post.jar "<delete><query>*:*</query></delete>" && java -jar post.jar'

'Hope that helps :)

Moving changesets from the default branch to a new branch in Mercurial

2011-01-21T00:00:00+01:00

Yesterday I needed to move a specific list of changesets from the default branch. Here is how I did it (there is surely another way to do it :)).

First you need the Mercurial Queues extension which you can activate by adding the following contents in your hgrc file

[extensions]
mq =

Once the extension is activated the sequence to apply is quite trivial :

convert the revisions you want to patches
```
hg qimport -r <START_REV>:<END_REV>
```
remove them from the history
```
hg qpop -a
```
start a new branch
```
hg branch <mynewbranch>
```
push the patches back in this newly created branch
```
hg qpush -A
```
finish the work
```
hg qfinish -a
```

Done :)

Demystifying telecommuting

2011-01-04T00:00:00+01:00

I have been telecommuting for a few years now and quite often I meet people -mostly team managers- who are immune to that kind of work arrangement.

I will try to answer the questions I already heard and post my answers below. Please note that this apply for my domain which is the IT industry I do not know if the same concepts can be applied to other domains.

"Telecommuting is bad for business"

This is something I heard a lot. But this is not true some people think telecommuting is generally a bad idea for business and that you can not become a successful company with that kind of organization. Let me tell you a little story. Once upon a time there was a small finnish company called MySQL AB, they created one of the most (if not the most) popular RDBMS on earth a few years after. A few years ago this same company was sold for $1 billion. Guess what ? Some engineers were (maybe they still are) telecommuters, I got that info from Monty when I had the chance (and the honor) to met him last year. I am sure there are other examples like MySQL, if you know some of them feel free to tell me and I will add them.

"Once at home, nobody will work"

That is indeed a risk. However nothing really forces someone to work when he does not want to even at the office. You can get people spending most of their time watching videos on YouTube or doing something completely unrelated to work. You could filter your network and block unwanted websites, but there are a lot of websites on earth and you can not block everything.

If your team writes code on a daily basis I believe you have at least an SCM and a bug tracker. Both of these tools send notifications to a list that is followed by everyone. If you see commits and notifications regularly from your colleagues then you can be sure they are working. If you see nothing then maybe they are not working or maybe there is something else. For example someone is sick or working on a really big issues that can not be fixed within one day or work.

This is also a matter of discipline. When you work from home, you have to control yourself not to take too much coffee breaks and avoid digression as much as possible. Some people are capable to apply this discipline to themselves, some of them are not. Some others refuse to work from home because they want a clear distinction between work and the rest. When you work from home this is sometimes really difficult to stop working, which might sounds delightful to a manager :)

If you are concerned about efficiency a recent belgian study showed that telecommuting allowed to save 5 hours per week ! The study is available here [fr]

"Scrum does not work with telecommuters"

That is a myth. I used to use Scrum on a daily basis at my previous company. We were a distributed team, people came from France, Belgium, Poland, Norway and Germany. We had a 15/20 minutes daily scrum meeting. All the rest was handled remotely. From sprint definitions to sprint demos and it worked just fine. I believe the company still works that way and they have a new release every 6 months and they are always on time. If that worked for me why would not that work for you ?

"It is hard to meet everyone in the same place for brainstorming"

Brainstorming is indeed not as easy as asking everyone to go to the meeting room as everything will have to be done online. For face to face (video) meetings I prefer using Skype even though I find the screen sharing feature a bit limited. For voice only meetings I use Skype as well, however when there is more than 10 people in the same meeting it is difficult to work efficiently. If I need to show slides I like to use GoToMeeting. I heard good things about WebEx but I never tried it so any experience feedback is welcome :)

"My team will loose the social thing"

That is true. This is why sometimes it is just good to meet everyone in the same place for a few days. That can be arranged easily. Some people can sleep at coworkers' or can go to the hotel. Honestly the extra cost of making everyone meet in the same place from times to times is often negligible compared to what telecommuters can give you in term of quality of work.

Conclusion.

Let's be honest, telecommuting is hard and requires a lot of discipline. But the benefit on the long term are really worth it. What about trying this new way of collaborating in 2011 ? It could be one of your new year's resolution ;)

PHP library release survey

2010-12-02T00:00:00+01:00

Note

This is a (almost) copy and paste version of Tobias' original post.

The Apache Zeta Components project is currently discussing how to adjust the old eZ Components release process to the ASF guidelines.

During this discussion, quite some ideas came up on how the process could be optimized.

This brought our interest to how users of PHP libraries would actually like to obtain library code for their projects. The results of the survey will be published later so all of the PHP libraries and frameworks can benefit.

So please take some minutes and fill out the form :

http://bit.ly/phplibrelsur

Thanks in advance :)

Flushing a cache-block manually in eZ Publish

2010-11-17T00:00:00+01:00

What is described below is extremely experimental !

Purging cache-blocks in eZ Publish is something you can hardly control because everything is done on the fly for each HTTP request.

Furthermore there is no API for that task and you have to inspect the guts of the template compiler (in lib/eztemplate/classes) in order to find what you need.

For the unit test suite I am working on I needed to flush a specific cache-block from the CLI SAPI so I needed a function to do that automatically.

I finally came up with the following code:

<?php

function expireCacheBlock( $template, $keyAttributeValue )
{
    $functionPlacement = fetchCacheBlockPlacement( $template );
    $placementKeyString = eZTemplateCacheBlock::placementString( $functionPlacement );

    $keys = array(
        // cache-block key attribute
        $keyAttributeValue,
        // cache-block placement in the template
        $placementKeyString,
        // siteaccess
        $GLOBALS['eZCurrentAccess']['name'],
    );

    // subtree_expiry is NOT supported
    $nodeID = false;

    $cachePath = eZTemplateCacheBlock::cachePath(
        eZTemplateCacheBlock::keyString( $keys ), $nodeID
    );

    // should use the cluster API here
    unlink( $cachePath );
}

function fetchCacheBlockPlacement( $template )
{
    $ini = eZINI::instance();
    $ini->setVariable( 'TemplateSettings', 'TemplateCompile', 'disabled' );

    $tpl = eZTemplate::factory();
    $resource = $tpl->loadURIRoot(
        "file:{$template}", $displayErrors = false, $extraParameters = array()
    );

    $tpl->process( $resource['root-node'], $text, "", "" );
    $children = $resource['root-node'][1];

    $functionPlacement = array();

    // searching for the {cache-block ...} function
    // and its placement
    foreach ( $children as $child)
    {
        $nodeType = $child[0];
        if( $nodeType == eZTemplate::NODE_FUNCTION )
        {
            $functionPlacement = $child[4];
        }
    }

    return $functionPlacement;
}

$template = 'extension/varnishcache/design/varnishcachetests/templates/testcacheblock.tpl';
$key = 'varnishtest';
expireCacheBlock( $template, $key );

?>

As you can see the cluster API is not used here but I do not need it so using a simple unlink is enough for my needs. Furthermore this script has a few limitations :

it support only one key ( equivalent to {cache-block .... key=foo ...} )
it does not support the design fallback. This is why you have to provide the full path of the template.

If you are interested in improving this code, feel free to fork it from its github repository

New blog!

2010-11-14T00:00:00+01:00

Enough wordpress and dynamic pages where the only thing I need is a simple page generator and ReST syntax. Welcome Pelican :)

The old RSS feed is now obsolete. It has been replaced by a new Atom feed available here :

http://jrenard.info/blog/feeds/all.atom.xml

About ESI

2010-11-08T00:00:00+01:00

What is ESI ?

ESI (Edge Side Includes) is a system which allows content assembly by a web accelerator (Akamai, F5, Varnish, Squid ...). If you are familiar with the concept of SSI with Apache then you will quickly become familiar with ESI as the concept is more or less the same.

What problem does ESI solve ?

The most common use case is the following (I will take the most trivial one). Imagine you have a page with an article, and at the top of the page you want to display 'Hello Mr John Smith' where John Smith is the name of the user who is visiting the page. When you want to service the page with your web accelerator you have a problem because the page is personnalized. So basically you can not cache the page in your web accelerator (I know you could use JavaScript for that trivial use case bla bla bla bla). So what can you do if you still want your web accelerator to service the page ? What if you could ask your web accelerator to still keep the article cached, but call a module dedicated to display 'Hello Mr John Smith' ? This is where ESI comes into play. The web accelerator will keep a cached copy of the article but by using a specific set of XML tags it will be able to call the 'hello xxxx' module and insert the result in the final page before servicing it to the user.

How does ESI work ?

Well, that is quite simple. By using a bunch of specific tags your web accelerator will process them, insert the result in the HTML page and will finally service it to the client. Let's take an example with the <esi:include [...]/> :

the webserver will return an HTML page with a tag <esi:include src="http://path/to/something" [...]/>
the web accelerator will parse the HTML, find the <esi:include [...]/> tag
the web accelerator will call the URL defined in the src attribute and insert the result in the HTML page where the <esi:include [...]/> tag has been defined of course
the web accelerator will service the final page to the client.

Where can I get ESI support ?

ESI is supported by vendors such as Akamai and F5 but the requirement here is to get ready to sign a (big) check ;). ESI is also supported by free open source softwares such as Varnish and Squid. However this is not because it is supported by these vendors/softwares that the specification is fully supported. The only vendor I know who support the specification completely is Akamai (F5 as well ? If you know other vendors who support ESI completely, please let me know). Squid and Varnish have only a very limited support of the language.

So what can I do with it ?

If you plan to use the <esi:include [...]/> tag only then you can use Varnish but you have to know it has a very limited support of this tag and will ignore errors silently. If you plan to use more than <esi:include [...]/> then you do not have much choice, you have to use vendors like Akamai and this costs money. Sometimes it makes sense to sign a check to get a quick solution "now" if "now" is definitely important to you. However you have to take the long run into account as well. Using a feature which is not completely supported by different vendors/softwares introduces the risk of being locked with a specific vendor/software. Conclusion ----------

I personally see ESI an numeric equivalent to cigarettes. So it is better for your health not to taste. Use ESI only if this is the very last solution you can use.

Useful links

ESI language specification
Introduction to ESI by F5
ESI support by Varnish (only esi:include and esi:remove so far)

'Hope that helps :)

mysqlslap and SQL syntax errors

2010-09-23T00:00:00+02:00

While I was playing with mysqlslap for my local benchmarks I got an error that puzzled me :

You have an error in your SQL syntax; ...

After analysing the query and running it with the CLI mysql client and in other tools I tried to write the query on one line. That was the solution, it seems mysqlslap only accepts one line queries :/ I tried with the following version

mysqlslap  Ver 1.0 Distrib 5.1.39, for pc-linux-gnu (i686)

On tools

2010-09-08T00:00:00+02:00

I just realized that a lot of people expect a bit too much from tools because they think they will magically do the job for them. Unfortunately that is wrong.

What tools are good at

Tools are good to help you to do the job and to deliver what you have to deliver. A lot of them are usually designed to do one thing, but to do it well. They are usually specialized.

What tools are not good at

Tools are not good at doing the job for you. No matter what you have to deliver to your customer, the final result is generally an aggregation of features. This means that you will have to use different tools in order to deliver the final package. And no tool can do that job for you.

The best tool you have is your brain

Do you imagine a chef in a restaurant became a chef just because he (she) uses the best tool available ? No. The chef uses his (her) brain and creativity first in order to find the best composition of flavours. Once the recipe is roughly written the chef starts to do the real cooking with all the tools he (she) needs in order to achieve the expected result.

Do not be the tool of your tool(s)

Having the best hammer available on the market does not mean you can use it for everything. Otherwise that means everything you are going to touch will look like a nail and the only thing you will try to do is to drive this nail without really knowing what you do and the result might look like this:

Learn and try new tools

You have to master the tools you use or plan to use. This will help you a lot to build the solution you have been asked to build, but also when you have to make a decision on whether or not to use tool X or Y to do a task. The are new tools released every year, it is impossible to know them all but it is important to at least know they exist. If possible try them in a sandbox, a Virtual Machine is perfect for that you create a new instance and 5 minutes later you can try a new tool in case of mistake you remove the Virtual Machine :)

One saying to have in mind

Henry David Thoreau :

Men have become the tool of their tools.

Do you really want to prove him right ? :)

Finding OmmWriter audio files

2010-09-06T00:00:00+02:00

I really appreciate working with OmmWriter when I need to get focused on something. However sometimes I just woud like to listen to the audio files without launching the app. OmmWriter's audio files are available in :

/Applications/OmmWriter.app/Contents/*.m4a

If you want to copy them here is what you can do in a Terminal :

find /Applications/OmmWriter.app/Contents \
-type 'f' -name '*.m4a' -exec cp {} /path/of/your/choice \;

Now you can listen to these audio files without launching OmmWriter. :)

Working with people living in different timezones.

2010-08-17T09:45:00+02:00

I just spent the last 10 days working for a customer in the USA (in the Alexa top 500 in the US actually) and I was "debriefing" so why not share the experience feedback ? :) The problem was how to work with people living in different timezones ? The following list of could help you :

http://www.timeanddate.com/ is priceless, especially the meeting planner.
Send emails for everything. That could be seen as over communication but it is preferable to send an email to say "Hey I get this problem" and then send an email 2 hours later to say "Hey I fixed this problem" rather that saying and let people you work with without informations that might be relevant later during the day (or the week). Especially if people need this information during your sleep hours.
Be available. That does not mean you have to wake up at 5am and got to sleep at 11pm, that just means that if your collaborator(s) need you for a couple of minutes then do not refuse a call unless you have a really good argument/excuse. You and your collaborators know that at some point you (or them) will have to either wake up (very) early or go to bed (very) late. People usually respect that and meetings tend to be quick and focused.
Think about the next day. The great thing when you work with different timezones is that production (almost) never stops. So you can go to bed after your day and when you wake up in the morning you see new stuff being done, new features implemented. That is a great productivity booster, but sometimes that can get really confusing. So when you work on monday, think about what you will have to do on tuesday and send an email in order to synchronize with the rest of the team in order to avoid any overlap.

Generating an execution trace for a PHP script with Dtrace

2010-08-12T00:00:00+02:00

Here is a small Dtrace script I used to generate the execution trace of a PHP script running in a terminal : (executiontrace.d)

#! /usr/bin/dtrace -s
#pragma D option quiet
#pragma D option switchrate=10

BEGIN
{
    printf("TRACE START [%Y]\n", walltimestamp);
    self->depth      = 0;
    self->time_last  = 0;
}

/**
 * program begins, so {main}
 */
php*:::function-entry
/copyinstr(arg0) == ""/
{
    printf(
        "%d\t -> {main}() %s:%d\n",
        /* well, just zero :-) */
        0,
        /* source file, you can use basename(copyinstr(arg1)) if needed */
        copyinstr(arg1),
        /* line number*/
        arg2
    );

    self->time_last = timestamp;
}

/**
 * start any other function that is not {main}
 */
php*:::function-entry
/copyinstr(arg0) != ""/
{
    self->depth += 2;

    printf(
        /* string format */
        "%d\t %*s %s%s%s() %s:%d\n",
        /* time delta (microseconds)*/
        (timestamp - self->time_last) / 1000,
        /* indentation   */
        self->depth, "->",
        /* class name, if available */
        copyinstr(arg3),
        /* scope operator, if available */
        copyinstr(arg4),
        /* function name, always available */
        copyinstr(arg0),
        /* source file, you can use basename(copyinstr(arg1)) if needed */
        copyinstr(arg1),
        /* line number*/
        arg2
    );

    self->time_last = timestamp;
}

/**
 * end of any other function that is not {main}
 */
php*:::function-return
/copyinstr(arg0) != ""/
{
    self->depth -= 2;
}

END
{
    self->depth = 0;
    printf("TRACE END [%Y]", walltimestamp);
}

It looks like an Xdebug trace (without the memory usage / delta) but since Dtrace and Xdebug do not operate at the same level the results are quite different. If you want to test this script you can use for example the following (nestedfunctions.php) :

<?php
function parent()
{
 new DirectoryIterator( __DIR__ );
 child();
}
function child()
{
 sleep( 1 );
}
function launch()
{
 parent( );
}
function foo(){}
launch(  );
foo();
sleep( 2 );
foo();
launch();
foo();
?>

And by running the following command :

sudo dtrace -s executiontrace.d -c "php nestedfunctions.php"

You should get a result like this (do not forget to hit <Ctrl-c> in order to exit from Dtrace).

TRACE START [2010 Aug 12 10:44:27]
0     -> {main}() [...]/testnestedfunctions.php:0
669     -> launch() [...]/testnestedfunctions.php:21
24       -> parent() [...]/testnestedfunctions.php:16
37         -> DirectoryIterator::__construct() [...]/testnestedfunctions.php:5
286         -> child() [...]/testnestedfunctions.php:6
21           -> sleep() [...]/testnestedfunctions.php:11
1000158     -> foo() [...]/testnestedfunctions.php:22
32     -> sleep() [...]/testnestedfunctions.php:23
2000097     -> foo() [...]/testnestedfunctions.php:24
32     -> launch() [...]/testnestedfunctions.php:25
21       -> parent() [...]/testnestedfunctions.php:16
32         -> DirectoryIterator::__construct() [...]/testnestedfunctions.php:5
141         -> child() [...]/testnestedfunctions.php:6
20           -> sleep() [...]/testnestedfunctions.php:11
1000111     -> foo() [...]/testnestedfunctions.php:26
^C
TRACE END [2010 Aug 12 10:44:34]

I tried to add as much comments as possible in order to make the script clearer. Please note that if you run this script on production you will get a lot of output because the script does not filter on any pid so Dtrace will aggregate results across multilple Apache processes. It is up to you to use the correct predicate to filter on the PID you want in order to limit the result set.

A comparison of the difference between the french and the US culture in IT.

2010-08-09T00:00:00+02:00

Disclaimer :: This blog post is based upon my experience, it does not pretend to generalize anything for one culture or another.

I have the chance to work for awesome customers in both France and the USA and that gives me the opportunity to work with people with different cultures and mindsets. So I thought that could be interesting to write down my experience with these two cultures and try to describe the differences.

There is one sure thing : American people care about IT, not only because it makes your life simpler or because this is serious business, but also because they simply invented it. If you have a look at the last decade you'll see that almost any major technological invention comes from the USA. If you have a look at the Alexa top 20, you'll see that almost all websites are from American companies.

American people are what I call superlative oriented, everything has to be great, awesome, excellent etc, etc. They are in constant need for heroes and rock stars. In my opinion this is a bit too much, but that is the way they are and I respect that.

On the project management side, one great thing is that most project managers (I have worked with, cf the disclaimer ;)) actually have a real technical background. I remember one day there was a problem on a big website and the editor in chief (yes, the editor in chief, they guy who care about contents, not tech) told me something like "I think there is an issue with mod_rewrite, most likely a regex, can you please have look at it ?". And guess what ? He was right. If I compare with my experience in France, there was a small problem with an import system that had to be fixed quickly and the project manager told me pedantically "There must be an infinite loop somewhere" ...

There is one main difference between the american and the french culture and I am not sure it will disappear in the next few years. American people have a deep respect for developpers (or engineers the word does not really matters here) whereas in France being a developer makes you be that weird guy who writes lines of code nobody else have a clue about. No matter what you say you are a "geek" so who would think you say interesting things anyway ? In France real men become project managers. It is acceptable to write code for a few years after being graduated (the lesser, the better) but at some point, you have to "grow up" and be a manager. I actually can not imagine the CTO of a big IT company going to the sysadmin's office and say them "you are useless". Think that can not happen in France ? It actually did I was there when that happened and that was not a a small company, that happened at a very big company I used to worked for from times to times.

American people (well, people living in an english speaking country) have a strong (who said unfair ? :)) advantage, every technical documentation / article / paper, well everything is written in english. If you have a look at how your program is built, it is all in english as well (if/then/else, while, for loop, function names, class names, etc, etc). So programming gets a bit more "natural" because they do not have to do any translation effort. And that gives them the opportunity to dive into any technology in record time.

On the contrary, French people have to make this effort all the time, it is no big deal when you to do this on a daily basis, but still this is not natural. This is why I personally think that the very first language any french developper should know is English, nothing else.

Everything is not of course black and white, and we have the chance to have pretty big websites in France as well like Meetic Dailymotion Netvibes (BTW : I would love to work you :)) for example. But I am not sure we will see (even though I'd love too) a lot of websites of that dimension emerging quickly in the future in France.

I know this post is a bit messy but this is all I can do for the moment given my spare time. Any feedback welcome :)

Finding syscalls generated by PHP functions with Dtrace

2010-08-04T00:00:00+02:00

I wanted to check a list of syscalls that are generated by PHP for a bunch of functions. Dtrace saved the day again, in 10 minutes I had all I needed. Here is a PHP script you can use for testing (syscalls.php) :

<?php
file_exists( __FILE__ );
file_get_contents( __FILE__ );
chdir( __DIR__ );
dir( __DIR__ );
getcwd();
opendir( __DIR__ );
scandir( __DIR__ );
new DirectoryIterator( __DIR__ );
stat( __DIR__ );
is_readable( __DIR__ );
is_writable( __DIR__ );
file_put_contents('/tmp/foo.txt', 'foo');
?>

And the small Dtrace (syscalls.d) script I used:

#!/usr/sbin/dtrace -s
#pragma D option quiet

php*:::function-entry
/pid == $target && arg0/
{
 printf( "%s%s%s\n", copyinstr(arg3), copyinstr(arg4), copyinstr(arg0) );
 self->follow++;
}

syscall:::entry
/self->follow > 0/
{
 printf("  ->%s\n", probefunc);
}

php*:::function-return
/pid == $target && arg0/
{
 self->follow -= self->follow == 0 ? 0 : 1;
 printf("\n");
}

You can test the script by running :

sudo dtrace -s syscalls.d -c "php syscalls.php"

And you should get a list like this (at least on Mac Os X)

file_exists
 ->access

file_get_contents
 ->lstat
 ->lstat
 ->open
 ->fstat
 ->lseek
 ->fstat
 ->read
 ->read
 ->read
 ->close

chdir
 ->chdir

dir
 ->open_nocancel
 ->fcntl_nocancel
 ->__sysctl
 ->fstatfs

getcwd
 ->open_nocancel
 ->fstat64
 ->fcntl_nocancel
 ->close_nocancel
 ->stat64

opendir
 ->open_nocancel
 ->fcntl_nocancel
 ->fstatfs
 ->close_nocancel

scandir
 ->open_nocancel
 ->fcntl_nocancel
 ->fstatfs
 ->getdirentries
 ->getdirentries
 ->close_nocancel

DirectoryIterator::__construct
 ->open_nocancel
 ->fcntl_nocancel
 ->fstatfs
 ->getdirentries

 ->close_nocancel
stat
 ->stat

is_readable
 ->access

is_writable
 ->access

file_put_contents
 ->lstat
 ->lstat
 ->readlink
 ->lstat
 ->lstat
 ->open
 ->fstat
 ->lseek
 ->write
 ->close

If I have time I'll post more Dtrace scripts useful for analysing what PHP does.

Edit : I fixed a small bug for the DirectoryOperator class.

Sphinx, documentation made easy

2010-08-01T00:00:00+02:00

I just wanted to promote a fantastic documentation tool called Sphinx. I have been using it for more than 6 months now and I am more than happy with it. What makes Sphinx so unique is that it makes actually you love writing documentation I use it for almost anything now, from writing specifications to documentations for deliverables I send to my customers. And the result is not just awesome, it just look professional (IMHO).

I use the Flask theme which I think provide a better readability than the default ones. As an example, here is what the Apache Zeta Components documentation looks like when generated with Sphinx using the Flask theme As a long time docutils user, learning the few Sphinx specific constructs was a piece of cake.

If you already know or worked with the RST syntax then using Sphinx should not be a problem at all. If you would like to convince your manager(s) to use Sphinx just say that the official Python documentation uses it, so does the official Symfony documentation.

Happy documenting :)

Come on Mr Arrington ...

2010-07-17T00:00:00+02:00

Dear Mr Arrington, I just read the following blog post on Techcrunch : Breaking: French Government Still Can’t Get France.fr live Even though you have a point about france.fr being down for days right after its launch, you did not need a blog post on Techcrunch for that, there is a simple word to explain this failure : incompetence. What is absolutely pointless in your article is this:

If the country’s experience with building aircraft carriers is any indication of their ability to build websites, we should see it limping along sometime in 2013.

Let's skip the aircraft carriers thingy because it is absolutely out of topic here. French people not able to do websites ? Do people behind Dailymotion, Netvibes, Skyblog, Seesmic, Deezer come from Mars ? You can love or hate the service but you can not say that french people can not make websites. By the way, showing the list of wars France lost is also useless, unless you want to talk about Vietnam and Irak ;) If ever someone read this blog post and uses the WW2 argument, I would recommend this person to read a history book first so he can understand why the USA entered into the conflict. Reading a bit on Lafayette is recommended as well. So please Mr Arrington, do what you do well i.e publishing news about tech and the Internet in general and stop the rest. Best Regards, PS : if ever this website goes down after a while due to the incoming traffic this is because it is hosted on a cheap shored host I do not quite care about not because it is French.

Collectd and Varnish user ? We need your feedback

2010-07-15T00:00:00+02:00

If you use collectd and would like to monitor your Varnish instance(s), this blog post is for you.

The new Varnish plugin for collectd is ready, however we would love to get feedback from you. The plugin has already been pushed on production on Camp To Camp and the feedback is really positive so far. However it is always good to get more feedback from different people using collectd and Varnish in different contexts.

How can you help ? Simply run the Varnish plugin on one of your production (or test) server(s) and provide feedback, and ideally some graphs :) Installing the Varnish plugin for is really trivial :

run git clone http://github.com/octo/collectd.git
compile collectd as you would normally do, do not forget the --enable-varnish flag
configure the Varnish plugin in collectd.conf, as explained in the configuration synopsis
start collectd For more information, feel free to read and/or comment the dedicated wiki page.

Feedback can be provided as you wish, either by sending an email to the mailing-list or directly to me <jerome.renard @ gmail.com>.

Thanks in advance for your feedback. :)

Easy switch from Mercurial to Git

2010-06-29T00:00:00+02:00

For a project I am currently working I needed to switch from Mercurial to Git. I used the method described below, that is really trivial and it works just fine. Download fast-export and extract it wherever you want. Run the following commands :

mkdir git.migration && cd !$ && git init
/path/to/fast-export/hg-fast-export.sh -r /path/to/your/mercurial/repository

And wait for a few minutes/seconds, depending on the size of your repository. In the end of the process you should get a result like :

git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:      10000
Total objects:         5426 (       566 duplicates                  )
 blobs  :         1990 (       453 duplicates       1454 deltas)
 trees  :         3027 (       113 duplicates       1762 deltas)
 commits:          409 (         0 duplicates          0 deltas)
 tags   :            0 (         0 duplicates          0 deltas)
Total branches:           1 (         1 loads     )
 marks:           1024 (       409 unique    )
 atoms:           1030
Memory total:          2540 KiB
 pools:          2149 KiB
 objects:           390 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =          1
pack_report: pack_mmap_calls          =          1
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =    7542871 /    7542871
---------------------------------------------------------------------

Now run :

git checkout

Done.

Fixing user infos in git commits

2010-06-29T00:00:00+02:00

After I easily switched from Mercurial to Git, I needed to update the committer's name and email adress. I found that using git filter-branch could help me. Here is what I did :

git filter-branch --env-filter \
'export GIT_AUTHOR_NAME="Foo" ; GIT_AUTHOR_EMAIL="foo.bar@example.com"'

And git updated the author's name and author's email for every commit.

Viewing the progress of a process through a pipe with pv

2010-06-14T00:00:00+02:00

What is pv ?

It is a tool that monitors the progress of data through a pipe.

How to install it ?

Well, that is quite simple, the package name is the same for all platforms : pv so you can use

sudo port|yum|apt-get install pv

How to use it ?

Let's take an example, you want to compress a big text file, you would normally run something like gzip <filename> and wait for the process to be finished without knowing exactly when. With pv you can know exactly what the status of the process is and when it is finished. Let's reuse the gzip example above and try the following (I tried on a 1.6GB text file) :

17:08 jerome@gimli/private/tmp% gzip dummy.txt
#and just wait for gzip to finish

Now let's try with pv:

17:10 jerome@gimli/private/tmp% pv dummy.txt | gzip > dummy.txt.gz
1.16GB 0:00:14 [81.9MB/s] [==============[...]===================>] 100%

Let's extract the file by using pv as well :

17:11 jerome@gimli/private/tmp% pv dummy.txt.gz | gunzip > dummy.txt.gunzip
3.45MB 0:00:22 [ 158kB/s] [==============[...]===================>] 100%

Cool isn't it ? Now we know exactly when the process is finished, this is really useful. For more information about pv, you can read the manual page

Avoiding warnings about autoconf 2.13 when building PHP

2010-06-09T00:00:00+02:00

If you want to build PHP from the SVN repository, you have to run buildconf [--force]. But depending on the version of autoconf you have on your machine you can get the very annoying warning message :

[...] buildconf: Your version of autoconf likely contains buggy
cache code. Running vcsclean for you. To avoid this, install
autoconf-2.13. [...]

If you do not have autoconf 2.13 installed, then you have to install it first. On RedHat running sudo yum install autoconf213 should work. On Debian, the autoconf2.13 package is available. Once autoconf2.13 install, you can instruct buildconf to use this new version of autoconf by runing

export PHP_AUTOCONF=`which autoconf-2.13`

You can check the value is correct by running

echo $PHP_AUTOCONF

Which should give you something like (depending on your system) /usr/bin/autoconf-2.13.

Now retry to run

./buildconf --force.

The annoying message is gone, joy :)

Getting varnish statistics in XML format

2010-05-27T00:00:00+02:00

I was searching something in the Varnish source code when I found a non documented (at least there is nothing in man varnishstat, for Varnish 2.1.2) feature. Varnishstat can return statistics in XML. How to get this format ? That's pretty simple, just add the -x argument to varnishstat, for example using :

varnishstat -x

will return something like :

<?xml version="1.0"?>
<varnishstat timestamp="2010-05-27T17:03:26">
<stat>
    <name>client_conn</name>
    <value>33036</value>
    <description>Client connections accepted</description>
</stat>
<stat>
    <name>client_drop</name>
    <value>0</value>
    <description>Connection dropped, no sess/wrk</description>
</stat>
<stat>
    <name>client_req</name>
    <value>34740</value>
    <description>Client requests received</description>
</stat>
<stat>
    <name>cache_hit</name>
    <value>33580</value>
    <description>Cache hits</description>
</stat>
[...]
</varnishstat>

So now, if you only want to get cache hits and cache_misses in XML so you can parse them easily on you can do :

varnishstat -f cache_hit,cache_miss -x

And you get something like

<?xml version="1.0"?>
<varnishstat timestamp="2010-05-27T17:05:41">
<stat>
    <name>cache_hit</name>
    <value>33580</value>
    <description>Cache hits</description>
</stat>
<stat>
    <name>cache_miss</name>
    <value>1160</value>
    <description>Cache misses</description>
</stat>
</varnishstat>

If you want to know more about that, have a look at the do_xml function in varnishstat.c

What's new in eZEnhancedDebug 1.2 ?

2010-04-26T00:00:00+02:00

I just released ezenhanceddebug 1.2, here are the new things I added :

Compatible with eZ Publish 4.3

If you read the changelog in the eZ Publish 4.3 release, you will notice :

Implemented enhancement #16216: Deprecate templateInit()

Which is documented by

kernel/common/template.php: templateInit() Function will be removed in a future version, use eZTemplate::factory() instead.

This change has been implemented.

KCacheGrind support

There is two new configuration directives in ezenhanceddebug.ini.append.php :

[ProfilerSettings]
Profiler=enabled
ProfilerOutputDir=/tmp/ezenhanceddebug

If [ProfilerSettings]/Profiler is set to enabled, then anytime you generate a page you will get a KCacheGrind compatible file in [ProfilerSettings]/ProfilerOutputDir/url_alias_of_the_page.out Here is what it looks like :

Each function follows the pattern : <template_name>::<template_function_or_operator> There is no callgraph available, this is currently not implemented. Due to the anarchic way of calling templates from eZTemplate, it is very difficult to deduct a valid callgraph, maybe I will add this in the future if I find exactly what I am looking for in the template engine. The full eZEnhancedDebug extension is now hosted at GitHub .

Tools I can not work without

2010-04-12T00:00:00+02:00

I wanted to share a bit of informations about the tools I use every day and which I could not work well without. Let's go.

Vim: I spend most of my time editing text (writing code, sending emails etc, etc) so having an efficient text editor is really important, I started to work with vim a few years ago and feel really cumfortable with it, furthermore I can learn one new feature / keyboard shortcut every day to make my life even easier.

Hudson: Every time I start working on something new I define a new job in hudson so all my projects are build and unit tested automatically on a regular basis. Coupling hudson and Ant is a piece of cake. I appreciate the feeling of being backed up by a tool that will ring the alarm bell when something is wrong (of course it can do much more :)).

OpenGrok: I discovered that only a few weeks ago and can not live without since then, this "wicked fast" source code browser is awesome. Basically you take Lucene + Excuberant Ctags + a fair amounf of java code and then you have it. Installing it on OpenSolaris is a piece of cake and takes around 5 minutes to get a working instance :) If you plan to use it on Mac Os X, get ready to fight a bit, first for installing Tomcat and then you'll have to patch the OpenGrok shell script in order to make it work, but after that, it works just fine. I really appreciate the way of browsing different code repositories using different SCM and different programming languages (like shown below) and to be able to search into one of them easily. Even if you are a grep wizard or an ack addict, using OpenGrok is really an interesting complement.

And you, what are the tools you can not work without ?

Fetching headers sent by PHP from the command line

2010-03-25T00:00:00+01:00

I needed a way to unit test some code that sends HTTP headers via the header function. However when doing this from the command line :

php -r "header('Foo : Bar'); print_r(headers_list());"

The only result you get is an empty array, which makes sense of course but for unit testing from the command line that is annoying. XDebug saves the day again by providing the xdebug_get_headers function. Let's try the same code, but with XDebug this time :

php -r "header('Foo:Bar'); print_r(xdebug_get_headers());"

Returns

Array
(
 [0] => Foo:Bar
)

Joy :)

A quick but working startup item for Hudson for Mac OS X

2010-02-22T00:00:00+01:00

Hudson is just a great tool but what really annoys me is having to run java -jar hudson.war all the time. Since I am lazy and stupid I like having my computer doing the work for me so I created a small plist file to run hudson as a service, here is the content of the file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>Hudson</string>
    <key>ProgramArguments</key>
    <array>
    <string>/usr/bin/java</string>
    <string>-jar</string>
    <string>/path/to/hudson.war</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

I saved it in

~/Library/LaunchAgents/hudson.ci.plist

and enabled it with :

sudo launchctl load -w ~/Library/LaunchAgents/hudson.ci.plist

Now I have hudson started automatically. No need to have that in mind in the future :) Note : This file can be of course enhanced. What I needed was a 5 minutes fix and that file did the trick, if you have any enhancement feel free to propose them :)

If you want to disable the startup item just run :

sudo launchctl unload -w ~/Library/LaunchAgents/hudson.ci.plist

Using Hudson in VirtualBox for multi OS testing

2010-02-08T00:00:00+01:00

For a small project I currently work on I wanted to run my unit test suites on different operating systems. Since I work on a Mac and wanted to also run tests on Linux, Virtual Box was exactly what I needed.

Preparing the guest

Just create a new virtual machine, for my project I created a CentOS guest. Install CentOS and configure it like you want. Install the guest additions as well, you will have to install quite a few extra packages in order to get these additions working but the install script is verbose enough to tell you what to install.

Preparing the shared folder

In your VM configuration, just create a new shared folder which points to the project you work on, ideally in read-only mode. You can now mount the shared folder on the guest, like this:

sudo mkdir /mnt/shared/folder

Add the following line in your /etc/fstab

shared-folder-name /mnt/shared/folder vboxsf defaults 0 0

You can now mount the shared folder by simply running:

sudo mount shared-folder-name

Installing Hudson

You should now have a working guest with CentOS. Just install hudson by using the native packages available.

Configuring Hudson

Create and configure a new job like you would normally do for any normal hudson job. Below is the configuration for my project, please note that I use the mercurial plugin and the clover plugin.

Project name : my project
SCM : mercurial, repository : /mnt/shared/folder with @hourly build
Run the following ant targets : rununittests, runcodingstandardcheck, generatecodecoveragereportclover
Publish code coverage report : checked in path ci/codecoverage with file 'clover'
Activate Chuck Norris : yes (what would be a hudson job without the Chuck Norris plugin ^_^)

By using such a guest VM I am able to test on 2 different systems, Mac OS ( the host ) and Linux ( the guest ). Next step, test on an Open Solaris guest :)

Injecting server variables in a PHP script from an Apache module

2010-01-21T00:00:00+01:00

In case it help someone, here is a function that makes it possible to inject server variables in a PHP script so it is available in the $_SERVER array. The function looks like this :

static void inject_server_variable(request_rec *r, const char *name, const char *value)
{
    apr_table_set(r->notes, name, value);
    apr_table_set(r->subprocess_env, name, value);
}

The line

apr_table_set(r->notes, name, value);

makes it possible to fetch the variable by using the apache_note function. The line

apr_table_set(r->subprocess_env, name, value);

stores the variable in the $_SERVER array so it can be fetched only by doing a simple print($_SERVER['xxxx']) :)

Useful links for Apache module development

2010-01-19T00:00:00+01:00

If ever you plan to create your own Apache modules, here is a list of helpful URLs :

APR API doc
APR-Util APR doc
Apache API doc
libapr programming (quite old but really useful)
example modules
How to Build Apache for Development
APR Memory Pools Rock
Developer Documentation for Apache 2.0
An introduction to APR

And last but not least the wonderful Apache Tutor website. If you have other interesting links, feel free to post a comment with it and I will update this post.

Some notes about Varnish

2010-01-13T00:00:00+01:00

A quick memo about useful informations I found on Varnish this afternoon after digging in the VCL system. I prefer writing them somewhere before forgetting... The list of variables available in VCL for each context is available in libvcl/vcc_obj.c which seems to be generated by lib/libvcl/vcc_gen_fixed_token.tcl. In vcc_gen_fixed_token.vcl the following code :

{ client.ip                IP
 RO
 all
 "const struct sess *"
}

Results in the following C code:

{ "client.ip", IP, 9,
 "VRT_r_client_ip(sp)",
 NULL,
 V_RO,        0,
 VCL_MET_RECV | VCL_MET_PIPE | VCL_MET_PASS | VCL_MET_HASH
 | VCL_MET_MISS | VCL_MET_HIT | VCL_MET_FETCH | VCL_MET_DELIVER
 | VCL_MET_ERROR
 },

"client.ip" is the directive available in a VCL file, "RO" means Read Only - "RW" (Read Write is available as well) - 'all' means it is available in all contexts in the VCL file :

VCL_MET_RECV | VCL_MET_PIPE | VCL_MET_PASS | VCL_MET_HASH | VCL_MET_MISS | VCL_MET_HIT | VCL_MET_FETCH | VCL_MET_DELIVER | VCL_MET_ERROR

The list of functions one can use directly (between C{ ... }C tags) in VCL seems is available in include/vrt_obj.h VRT_SetHdr and VRT_GetHdr are available in include/vrt.h (for the declaration) and in bin/varnishd/cache_vrt.c. It is possible to change the TTL of an item directly by using a few lines of C code, like for example :

if (obj.http.My-Header)
{
 C{
   char *ttl;
   ttl = VRT_GetHdr(sp, HDR_OBJ, "\012My-Header:");
   VRT_l_obj_ttl(sp, atoi(ttl));
 }C
   remove obj.http.My-Header;
 }
}

The "\012" prepended to "My-Header" is the octal version of the length of the "My-Header" string. The "sp" variable is the variable describing the current session. HDR_OBJ is a constant available in in include/vrt.h :

enum gethdr_e { HDR_REQ, HDR_RESP, HDR_OBJ, HDR_BEREQ, HDR_BERESP };

It is possible to define custom HTTP headers by using the VRT_SetHdr function available in bin/varnishd/cache_vrt.c. For example :

C{
 const char *contentMD5 = "12345";
 VRT_SetHdr(sp, HDR_RESP, "\014Content-MD5:", contentMD5, vrt_magic_string_end);
 }C

Reading the VCC code generated from a VCL file is available by running : varnishd -d -f path/to/file.vcl -C

Useful links:

Octal/Decimal converter (Yes, it is good to be lazy :p)
Configuring varnish to use custom http headers

Using Liquibase for database updates

2010-01-12T00:00:00+01:00

This is a post-it for me so I get a resource on how to install and run liquibase in case I forget.

What is Liquibase ?

LiquiBase is an open source (LGPL), database-independent library for tracking, managing and applying database changes. It is built on a simple premise:

All database changes are stored in a human readable yet trackable form and checked into source control.

Source :http://www.liquibase.org

There are few introductory videos available here :

Installation

Do not use the RC version because it does not seem to work well yet. Using 1.9.5 version is fine and works out of the box.

Download Liquibase here : http://www.liquibase.org/download

Store it in :

/opt/jerome/local/liquibase-1.9.5

Install the Java connector for mysql and postgresql :

sudo port install mysql-connector-java postgresql-jdbc

Now use the following XML changelog file for testing. Do not expect any relevancy in this XML file it is just there to test that Liquibase works :

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.7"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.7 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.7.xsd">
<changeSet id="1" author="me">
    <comment>bla bla bla bla</comment>
    <createTable tableName="person">
    <column name="id" type="int">
        <constraints primaryKey="true" nullable="false"/>
    </column>
    <column name="firstname" type="java.sql.Types.SMALLINT"/>
    <column name="lastname" type="java.sql.Types.REAL"/>
    <column name="username" type="java.sql.Types.LONGVARCHAR">
        <constraints unique="true" nullable="false"/>
    </column>
    <column name="testid" type="int"/>
    </createTable>
</changeSet>
</databaseChangeLog>

http://java.sun.com/j2se/1.3/docs/api/java/sql/Types.html Use also the following liquibase.properties file

classpath:/opt/local/share/java/mysql-connector-java-5.0.jar:/opt/local/share/java/postgresql.jar
changeLogFile=createtable.xml

# Mysql Settings
username=root
password=1234
url=jdbc:mysql://localhost/testsliquibase

# Postgres driver
#url=jdbc:postgresql://localhost/postgres
#username=postgres

logLevel=severe

Now try everything works fine:

liquibase updateSQL

Troubleshooting

In case Liquibase fails to connect to MySQL make sure the login credentials are correct and that it is possible to connect to mysql by using the mysql CLI client. If yes, but Liquibase still refuses to connect to MySQL then comment the following line in /etc/hosts :

::1             localhost

and retry the liquibase command. It should work now.

Backporting ReflectionMethod::setAccessible in PHP 5.3.1

2009-12-09T00:00:00+01:00

I needed the ReflectionMethod::setAccessible method for one of my script. But after reading the CHANGELOG I realized that it is available for PHP 5.3.2 only. Since PHP 5.3.2 is not released yet I was blocked :/ So I decided to add ReflectionMethod::setAccessible in my version of PHP so I can use it in my scripts. The backport is available here : reflection.diff.gz You can install it by following the method below:

download the patch above
download the PHP 5.3.1 tarball and extract it

apply the patch

patch -p0 --verbose < ./reflection.diff

run the standard ./configure [...] && make && make test && sudo make install commands. Alternatively, if you only want to test the Reflection extension you can do the following, after running "make":

export TEST_PHP_EXECUTABLE="./sapi/cli/php" && php ./run-tests.php ext/reflection/tests

export TEST_PHP_EXECUTABLE="./sapi/cli/php" && php ./run-tests.php ext/reflection/tests/ReflectionMethod_setAccessible.phpt

But in any case, you should get :

==========================================
Number of tests :  267               267
Tests skipped   :    0 (  0.0%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :  267 (100.0%) (100.0%)
------------------------------------------
Time taken      :    9 seconds
==========================================

All Reflection tests must pass. 'Hope that helps :)

Back from the PHP conference in Paris

2009-11-14T00:00:00+01:00

Thanks to my friend Arnaud I have been able to come to Paris in order to attend the last day of the PHP conference. That was great and the talks were interesting. I had a interesting discussion with Johannes Schlüter about DTrace (I was happy to meet another Dtrace addict :)) and he gave me the solution to run Skype on Open Solaris so I'll test it and if that works then I'll switch to Open Solaris fully and get rid of Mac OS X. We also had an interesting dicussion on PHP6, and it seems we have the same concers. I also met a lot of people in the community I had not met for a while: Julien, Jean-Marc, Perrick, Damien (sorry I did not test the Snickers pie yet), FreshDaz, Cyril... I was really happy to meet you all. I also had the chance (and the honnor) to discuss with Monty along with my colleague Patrick. And I have to say that he really opened my mind to something I was secretely looking for. Thanks a lot to all the team in charge of the organisation, I am sure you sweated a lot, but that was really worth it.

Apache .htaccess and disk access, a quick but deeper look

2009-10-30T00:00:00+01:00

I was discussing (well twitting is more appropriate) with my friend Arnaud about .htaccess files and why they should not be used. To be quick, using .htacess files is not good at all for disk performance because Apache will hit the disk for each HTTP request, so when the site is on production you can easily imagine this can quickly be an issue if you use those files. But let's have a look at how it actually work on the system. Remember, for each HTTP request Apache will look on your file system if it finds a .htaccess file. So if we create the following virtual web application :

webproject
|-- cache
|-- design
|   |-- css
|   |-- html
|   |-- images
|   `-- javascript
`-- src
 |-- php
 `-- python

With the following directive in your Apache configuration file (wether a VirtualHost or not) :

<Directory "/path/to/webproject/">
 AllowOverride All
 [....]
</Directory>

Now let's have a look at how Apache handle AllowOverride on the Operating System, what happens if I visit the 'webproject/src' directory ? Calling http://localhost/webproject/src generated the following system calls

1679/0x97c9998:  open("/Users/jerome/work/www/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/cache/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/design/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/src/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/src/python/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/src/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2
1679/0x97c9998:  open("/Users/jerome/work/www/webproject/src/php/.htaccess\0", 0x0, 0x1B6)         = -1 Err#2

As you can see Apache looked for the .htaccess file for each directory I had to browse in order to go to the webfolder/src directory. That is a simple example, but if you have a much more complex directory structure and a lot of traffic I guess you can imagine how much Apache is going to hit the disk in order to find a .htaccess file. So in order to avoid that it is always recommended to configure everything in VirtualHosts and to avoid .htaccess files.

'Hope that helped a bit :)

An alternative method to build eZ Publish configuration files with ANT and the C Preprocessor

2009-10-18T12:47:00+02:00

eZ Publish configuration files have a major problem : they are not context aware. This means that you can not build them especially for production or development servers. I thought about a solution to sort of workaround this problem by using the C preprocessor and macros in configuration files. Let's take a simple example. When you work with override.ini directives, you have to write a few configuration directives in order to load the correct template for the correct context, for example, a nodeID. But if this nodeID change between servers, you could have for example this on your development machine :

[foo_bar]
....
Match[node]=123

this on your staging machine :

[foo_bar]
....
Match[node]=456

this on your production machine :

[foo_bar]
....
Match[node]=789

When it comes to industrialize the process of creating correct configuration files with correct nodeIDs (or any other configuration directives) things become more and more complex. It would be interesting to be able to include the correct values depending on the context in which you want to use your configuration files. This is where the C preprocessor comes into play, by using specific macros like #include or #ifdef we can generate context aware configuration files without any problem, simply by including the correct configuration file and integrate it into the real eZ Publish one. For example for override.ini.append.php, this gives the following override.ini.append.php file :

<?php /* #?ini charset="utf-8"?

[full_folder]
Source=node/view/full.tpl
MatchFile=full/folder.tpl
Subdir=templates
Match[class_identifier]=folder

[line_folder]
Source=node/view/line.tpl
MatchFile=line/folder.tpl
Subdir=templates
Match[class_identifier]=folder

// some comments
// in the ini file
// they will automatically
// transformed into correct
// eZINI comments

#ifdef CONTEXT_DEV
#include "override.ini.dev"
#endif

#ifdef CONTEXT_STAGING
#include "override.ini.staging"
#endif

#ifdef CONTEXT_PROD
#include "override.ini.prod"
#endif

*/ ?>

With the following included files override.ini.dev

// global overrides
[my_dev_override]
Source=path/to/template.tpl
MatchFile=path/to/matchfile.tpl
Subdir=templates
Match[node]=11111

override.ini.staging

// global overrides
[my_staging_override]
Source=path/to/template.tpl
MatchFile=path/to/matchfile.tpl
Subdir=templates
Match[node]=22222

override.ini.prod

// global overrides
[my_prod_override]
Source=path/to/template.tpl
MatchFile=path/to/matchfile.tpl
Subdir=templates
Match[node]=33333

So when we run (note this is not that simple, but I took a few shortcuts in order to be quick)

cpp -P -DCONTEXT_PROD=1 override.ini.append.php

We get the following result :

<?php /* #?ini charset="utf-8"?

[full_folder]
Source=node/view/full.tpl
MatchFile=full/folder.tpl
Subdir=templates
Match[class_identifier]=folder

[line_folder]
Source=node/view/line.tpl
MatchFile=line/folder.tpl
Subdir=templates
Match[class_identifier]=folder

# some comments
# in the ini file
# they will automatically
# transformed into correct
# eZINI comments
# global overrides

[my_prod_override]
Source=path/to/template.tpl
MatchFile=path/to/matchfile.tpl
Subdir=templates
Match[node]=33333

*/ ?>

As we can see, override.ini.append.php has been created with the correct production nodeID. Since calling cpp is quite annoying with this ini syntax (mostly due to the presence of PHP comments) I decided to automate the process by wrapping everything into an Ant build script. With the script, building all configuration files should be as easy as running

ant clean && ant dev

ant clean && ant staging

ant clean && ant prod

You will find attached a tarball with a fake eZ Publish instance plus settings and context files. By running of of the commands above, you can generate your configuration files in no time. Here is an explanation of the ant script's behaviour. First you have to have the following directory structure :

.
|-- build.xml
|-- common.xml
`-- fakeezinstance
    |-- build.xml
    |-- extension
    |   `-- myextension
    |       |-- build.xml
    |       `-- settings.orig
    |           |-- dbsettings.dev
    |           |-- dbsettings.prod
    |           |-- dbsettings.staging
    |           |-- site.ini.append.php
    |           `-- siteaccess
    |               `-- site.com
    |                   |-- override.ini.append.php
    |                   |-- override.ini.dev
    |                   |-- override.ini.prod
    |                   `-- override.ini.staging
    `-- settings.orig
        |-- override
        |   |-- override.ini.append.php
        |   |-- override.ini.dev
        |   |-- override.ini.prod
        |   `-- override.ini.staging
        `-- siteaccess
            `-- site.com
                |-- override.ini.append.php
                |-- override.ini.dev
                |-- override.ini.prod
                `-- override.ini.staging

As you can see any original settings are located into settings.orig directories. Each of them contains configuration files, I just created an example with override.ini and site.ini. But you can use any configuration file. For each .ini file I created a context file, dbsettings.dev/staging/prod for site.ini for example. Once the configuration files are ready, we can simply run :

ant clean && ant prod

And you will see that all settings folder will be created with the correct values. If you want to add another extension in the build, the only thing you have to do is to copy the build.xml file available in extension/myextension and create a settings.orig directory structure. If you do not want your extension(s) to be processed by the build script, just do not copy the build.xml file and create the settings directory structure as you would normally do. So one can say that there is a lot of blank lines in the files onces they have been preprocessed. That is true, but remember that those files are supposed to be read by eZ Publish only. If fakeezproject.taryou need to change anything, just update the file you want in settings.orig and rerun the build process, it takes less than 2 seconds. Download the fakeezproject.tar archive, go in fakeproject ant run ant prod for example. 'Hope that helps.

Linting an eZ Publish template file in Vim

2009-10-03T00:00:00+02:00

Sometimes when I do eZ Publish templating I want to check if my template is syntactically correct without refreshing the page I am working on. And what I like with Vim is to run :make and get everything checked (and compiled if needed). So I updated my configuration file for eZ Publish templates. I defined all .tpl files as Smarty, so in my ~/.vim/filetype.vim I have the following content :

au BufNewFile,BufRead *.tpl setf smarty

And in ~/.vim/ftplugin/smarty.vim I have :

set makeprg=php\ \.\/bin\/php\/eztemplatecheck\.php\ --debug=error\ --no-colors\ %

So now for each eZ Publish template I edit I can run :make and see if I did syntax errors or not. This requires that you edit all your template from the top directory of your eZ Publish instance, otherwise it will not work.

How-to find the Unicode value for a character in Vim

2009-08-26T00:00:00+02:00

I was looking for a way to easily find an Unicode value for a character in Vim. The solution is actually quite trivial, move the cursor on a char and write 'g8' you should see the UTF-8 value for this char. For example with a left curly bracket '{' Vim displays '7b' which results in U+007B. It is also possible to use 'ga' to get the decimal, hexadecimal and octal value of a string. How to find the unicode value of a character ?

Unicode characters are commonly referred to using a notation like U+nnnn where nnnn is a four-digit hexadecimal (base 16) number specifying the code position of the character in Unicode. For example, the space character has the same code number in Unicode as in ISO 8859-1, namely 32 decimal, 20 hexadecimal; thus, it can be denoted as U+0020. Generally, a notation like U+nnnn is needed for referring to characters uniquely in contexts where one cannot reliably present the character itself.

Source : http://www.cs.tut.fi/~jkorpela/latin1/all.html

Useful links

Diffing two directories

2009-08-25T00:00:00+02:00

Finding which files differ :

diff -Naurq dir1 dir2

Outputing all the differences in VI

diff -Naur dir1 dir2 | vi -

Or in colordiff

diff -Naur dir1 dir2 | colordiff

Now I have no excuse to forget this all the time ...

Parsing XML files with APR

2009-08-17T00:00:00+02:00

I wanted to write an XML parser for one of my personal projects, and I needed to use Apache APR for this. Since I did not find any example about doing this on the web I thought it might be a good idea to write a short tutorial about this. Maybe it could be helpful to someone else. Let's prepare the tutorial, we need a XML file, for example this one example.xml :

<?xml version="1.0" encoding="utf-8"?>
<document>
<a someattribute="attribute of A">
    <b>the value of B</b>
</a>
</document>

And a Makefile, for example this one :

APR_CONFIG=$(shell which apr-1-config)
CC=gcc
CXX=g++
CFLAGS=$(shell ${APR_CONFIG} --cflags --cppflags --includes) -Wall
CXXFLAGS=$(shell ${APR_CONFIG} --cflags --cppflags --includes) -Wall
LDFLAGS=$(shell ${APR_CONFIG} --ldflags)
LDLIBS=$(shell ${APR_CONFIG} --libs --link-ld) -laprutil-1

Now the parser, parse.c:

#ifdef HAVE_CONFIG_H
#include
#endif

#include
#include

#include
#include

/**
* Dumps an XML document
*/
static void dump_xml( apr_xml_elem *e, apr_pool_t *pool )
{
    apr_xml_elem *child_element;
    const char *text;

    if ( e->attr )
    {
        printf( "This tag has an attribute : <%s %s=%s>\n", e->name, e->attr->name, e->attr->value );
    }

    if (e->first_child)
    {
        child_element = e->first_child;

        while (child_element)
        {
            printf( "<%s> --> ", child_element->name );
            if( !APR_XML_ELEM_IS_EMPTY( child_element ) )
            {
                apr_xml_to_text( pool, child_element, APR_XML_X2T_INNER, NULL, NULL, &text, NULL);
                printf( "%s\n", text );
            }
            dump_xml( child_element, pool );
            child_element = child_element->next;
        }
    }
}

/**
* Process an XML file
* @return APR_SUCCESS if success, otherwise -1
*/
static int process_xml_file( const char *filename, apr_pool_t *pool )
{
    apr_file_t *fp;
    apr_status_t rv;
    apr_xml_parser *parser = 0;
    apr_xml_doc *doc;
    char errbuf[255];

    if ( ( rv = apr_file_open( &fp, filename, APR_FOPEN_READ, APR_OS_DEFAULT, pool ) ) != APR_SUCCESS )
    {
        return -1;
    }

    rv = apr_xml_parse_file( pool, &parser, &doc, fp, 2000 );

    if( rv != APR_SUCCESS )
    {
        fprintf( stderr, "APR Error %s\n", apr_strerror( rv, errbuf, 255 ) );

        if(parser != 0)
        {
            fprintf(stderr, "XML Error : %s", apr_xml_parser_geterror( parser, errbuf, 255 ) );
        }

        return -1;
    }

    dump_xml( doc->root, pool );

    return APR_SUCCESS;
}

int main(int argc, char **argv)
{
    apr_pool_t *pool;
    const char *filename;

    if ( argc == 2 )
    {
        filename = argv[1];
    }
    else
    {
        filename = "example.xml";
    }

    apr_initialize();
    apr_pool_create( &pool, NULL );

    process_xml_file( filename, pool );

    apr_terminate();
    return 0;
}

Now store example.xml and parse.c in the same directory and build parse.c

make parse && ./parse filename.xml

You should get the following result:

<a> --> <b>the value of B</b>
This tag has an attribute : <a someattribute=attribute of A>
<b> --> the value of B

Some useful documentation :

http://apr.apache.org/docs/apr-util/1.3/apr__xml_8h.html

Edit: Many thanks to Riccardo Campisano for his feedback.

Overriding kernel files in eZ Publish

2009-08-12T00:00:00+02:00

Here is a 30 seconds tutorial on how to override kernel files in eZ Publish. The file used for this example is ezcontentobjecttreenode.php but you can use exactly the same method for any class of the kernel/ or lib/ directory.

cd path/to/ezpublish mkdir -p extension/myextension/classes/kerneloverride
cp kernel/classes/ezcontentobjecttreenode.php extension/myextension/classes/kerneloverride
echo "define( 'EZP_AUTOLOAD_ALLOW_KERNEL_OVERRIDE', true );" > config.php php
./bin/php/ezpgenerateautoloads.php -o

That is it.

Creating private targets with Ant

2009-07-30T00:00:00+02:00

I have been using Ant for quite a few weeks now, trying to get rid of the eZ Publish' 6000 lines shell script build system and then using Ant almost exclusively. I was looking for a way to create private targets, i.e targets which could not be called by the user. I finally found the solution, and it is quite trivial, just prefix your target's name with a dash. Let's do a quick "before/after" comparison.

Before

<?xml version="1.0"?>
<project name="MyProject" basedir="." default="build">
<target name="public">
    <echo message="Calling public target"/>
</target>
<target name="private">
    <echo message="Calling private target"/>
</target>
</project>

and calling ant public private returns :

public: [echo] Calling public target
private: [echo] Calling private target

After

<?xml version="1.0"?>
<project name="MyProject" basedir="." default="build">
<target name="public">
    <echo message="Calling public target"/>
</target>
<target name="-private">
    <echo message="Calling private target"/>
</target>
</project>

and then call the build file again, which returns :

public: [echo] Calling public target
BUILD FAILED
Target "private" does not exist in the project "MyProject".

By prefixing the "private" target with a "-" I made it private, well invisible would be more appropriate because there is no real concept of private targets in Ant. But at least it is not callable from the end-user which is exactly what I was looking for.

Backward search in your history with ZSH

2009-07-29T00:00:00+02:00

Bash has a cool native feature, backward search in your history. Zsh has the same feature but you need enable it. Here is how to do it :

# Bacward search in the shell history with <C-r>
bindkey ^r  history-incremental-search-backward
setopt hist_ignore_all_dups

Just add the lines above in your ~/.zshrc and then source the file again :

source ~/.zshrc

Now you should be able to search in your history with CTRL+r exactly like you would do with bash.

Compiling PHP 5.3 on Mac OS X

2009-07-18T00:00:00+02:00

A few people find it hard to compile PHP on Mac OS X and prefer using tools like Mamp or MacPorts. Even though MacPorts is useful I prefer compiling PHP by myself, here is a step by step tutorial.

Note

This tutorial has been written for Mac OS X 10.4 but I am sure it is the same for more recent versions.

Note

With the configuration directives described below PHP is compiled in CGI/FastCGI mode if you want to use it as an Apache module this only thing you have is to do is to use the following extra argument : --with-apxs2=/path/to/apache2/apxs

Downloading PHP 5.3 wget http://fr3.php.net/get/php-5.3.0.tar.gz/from/fr.php.net/mirror
Extracting the tarball tar zxvf php-5.3.0.tar.gz

Configuring PHP

cd php-5.3.0
./configure \\ --prefix=/opt/php5
\\ --enable-exif \\ --with-gd=/opt/local \\
--with-freetype-dir=/opt/local \\ --with-jpeg-dir=/opt/local \\
--enable-gd-native-ttf \\ --with-png-dir=/opt/local \\
--with-t1lib=/opt/local \\ --with-zlib=/opt/local \\
--with-mysql=/usr/local/mysql \\ --with-mysqli \\ --with-ldap \\
--enable-mbstring \\ --with-mcrypt=/opt/local \\
--with-bz2=/usr/bin \\ --enable-ctype \\ --enable-dom \\
--enable-libxml \\ --with-iconv \\ --enable-pdo \\
--with-pdo-sqlite \\ --enable-posix \\ --enable-simplexml \\
--enable-xml \\ --with-zlib \\ --with-curl \\ --enable-filter \\
--with-pdo-mysql=/usr/local/mysql \\ --with-openssl \\
--with-xmlrpc \\ --with-xsl \\ --enable-soap \\ --enable-ftp \\
--enable-pcntl \\ --enable-shmop

Compiling and installing
```
make && sudo make install
```

You can go to the kitchen and get a tea or a coffe while it is compiling. If you do not like tea or coffee you can also try other fun activities like the following : Source : http://xkcd.com/303/

Back from the kitchen, you are done. PHP 5.3 is now compiled and installed on your machine. You can of course use a few extra CFLAGS or options to the configure script if you want to, but the most basic steps are here and you now know the required steps Now a small tip, let us say I want to compile with the --with-mysql=/path/to/wrong/mysql/directory argument but unfortunately the configure script fails with such an error message :

configure: error: Cannot find MySQL header files under /path/to/wrong/directory. Note that the MySQL client library is not bundled anymore!

How to find the correct path and fix the problem ? I know that the mysql extension is available in the ext/mysql directory

The configuration related to this module is in config.m4. By opening ext/mysql/config.m4 I discover (line 77) that the mysql.h header is required and it is searched in the following directories (still on line 77) :

/usr/local
/usr/

The only thing I have to do is searching for mysql.h. Since I use the official MySQL binary I know it is installed in /usr/local/mysql/ so if I look in /usr/local/mysql/include/ I find the required mysql.h file. The config.m4 file searches for /include/mysql.h (line 81) so the only thing I have to do is to correct my --with-mysql argument and use --with-mysql=/usr/local/mysql.

By relaunching the configure script it worked and we have been able to compile PHP with MySQL successfully :) Since you now have compiled PHP by yourself the only limit for you now is your imagination :)

Displaying the byte sequence of a string in PHP

2009-06-26T00:00:00+02:00

Since I am never able to remember this, I decided to write it down here. The code is the following :

<?php
function showByteSequence( $string )
{
    for ( $i = 0; $i < strlen( $string ); $i++ )
    {
        echo '0x', dechex( ord( $string[ $i ] ) ), ' ';
    }

    echo "\n";
}
?>

For example writing

<?php
showByteSequence( 'foo-bar' );
?>

should display

0x66 0x6f 0x6f 0x2d 0x62 0x61 0x72

About me

2008-11-13T00:00:00+01:00

Want to hire me ? My resume is available at the following URL:

http://www.jrenard.info/cv/

You can also follow me on Twitter:

https://twitter.com/jeromerenard