There is an intentional parse error which generates a message on line 76. comment that out.
I altered test.php to remove the print statements and to execute all steps in a loop of 1000 iterations.
I'm running Ubuntu 13.10 64 bit on a cheap several years old Dell with an Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz and 4 gigs of ram.
Stock Ubuntu 13.10 configuration using mod_php and Apache.
Configuration per my previous article. The build in question is not a debugging build and is one I compiled myself. (takes forever)
LALR(1) Test Results
I first ran the test on the command line using PHP:
yml@humility:expressionLanguage$ php ./test.php Time elapsed: 5.0141038894653
then followed by HHVM:
yml@humility:expressionLanguage$ hhvm test.php Time elapsed: 5.5141990184784
At this point I was a little disheartened. I had expected HHVM to be faster in all cases. I had been told by the HHVM developers that hhvm first profiles the execution before invoking the JIT. I believe they said it happens after 10 or so requests. Figuring that maybe this is not invoked when run as a command line interpreter, I repeated the experiment in the browser running the test 15 times. This should give the JIT compiler a chance to get involved.
Time elapsed: 5.0715029239655 Time elapsed: 5.067526102066 Time elapsed: 5.0686640739441 Time elapsed: 5.1444919109344 Time elapsed: 5.0798120498657 Time elapsed: 5.0674719810486 Time elapsed: 5.0758039951324 Time elapsed: 5.0680429935455 Time elapsed: 5.0671808719635 Time elapsed: 5.1225011348724 Time elapsed: 5.0637631416321 Time elapsed: 5.0711719989777 Time elapsed: 5.0798349380493 Time elapsed: 5.0700869560242 Time elapsed: 5.0699179172516
Time elapsed: 5.9072468280792 Time elapsed: 5.9045979976654 Time elapsed: 5.8999648094177 Time elapsed: 5.8969769477844 Time elapsed: 5.8965940475464 Time elapsed: 5.8854668140411 Time elapsed: 5.8934369087219 Time elapsed: 5.8918490409851 Time elapsed: 5.8934841156006 Time elapsed: 5.8858242034912 Time elapsed: 5.8903369903564 Time elapsed: 1.5902738571167 Time elapsed: 1.5128128528595 Time elapsed: 1.5166771411896 Time elapsed: 1.5109701156616
And one can clearly see where the JIT was invoked yielding, in this case, a 70% reduction in execution time. Not as dramatic as 0.6 but still impressive.
There are many other aspects of the system that could be tested to paint a more detailed picture of the performance differences between PHP and HipHopVM. One can lose a lifetime trying to benchmark every feature of a given platform and emerge from the exercise no wiser.
HHVM vm-perf Tests
HipHopVM includes a few intense benchmarks that exercise narrower feature sets. These can be found in the github repository.
Out of curiosity, I decided to experiment with a few of these.
I used the same configuration as above except for these I used wget in a loop on the local machine and stopped it after 15 requests as in:
while true; do time wget -q http://localhost/vm-perf/fibr.php; done
The tests run long enough that the overhead of processing the tests and returning the result represents a negligible percentage.
This test computes the n'th Fibonacci number recursively up to 35.
This ran pretty consistently around:
Before the JIT kicked in it was slower than PHP.
But once the JIT kicked in:
This test is designed to be used as a benchmark and contains a number of different sorts and calculations.
This is the one test case I've been able to find where PHP outperforms HHVM.
I found this test interesting in that the JIT had a profoundly negative impact in this one scenario. I suspect this is probably due to some issue that will get resolved. The HHVM team is constantly pushing the envelope of performance and I expect they will find many more ways of improving this platform as time continues.
As a general rule, benchmarks don't tell you much about how a given technology is going to perform in your particular use case. They just make an argument that it might bear investigation. Is PHP slow? Slow is relative. It is clearly much slower than HHVM in a wide range of use cases. Of course we would expect this since PHP is an interpreter and HHVM is a native compiler.
In my particular use case, which is likely representative of a wide range of web application platforms, HHVM represents a vast performance improvement over PHP without requiring me to change my code.
Because of this constantly improving speed and the multi-threaded nature of the hhvm server, I suspect that the range of problems that PHP can be used to solve will expand.
Personally, I would love to see more multi-threaded features exposed in PHP. Yes, there is already some basic multithreading support added available. But it would be awesome if one could launch long lived services in separate threads. For particularly large and heavy frameworks like my own, this would be a boon. Build up your application object trees once and then re-use across requests, you know, like real server software.
Such support would also open a host of interesting possibilities. AMQP/STOMP in PHP? XMPP? NodePHP?