8-hour ordeal. 17-minute interview

Yesterday, I went to my interview for the Paratransit (door to door ADA service).

I set it up while I could still "walk" down my seven front stairs in 1/2 an hour.

Wheelchair-bound now.

Several days of big snow.

Shoveled by "the love of my life" who's going on weeks of 15-minute catnaps 24/7 to care of me.

Three big burly guys in two shifts to haul my wheelchair down and up stairs:

• Mike DeWhite
• Duli
• Robert Owens

My sister drove me to-and-fro.
Actually, Mike DeWhite with help from my sis (torn rotator cuff) and my SO (6 pins holding a steel plate in her broken collar bone) deserve special mention: Mike DeWhite man-handled me upstairs, with guidance from sis and SO.

Eight-hour ordeal. 17-minute Interview. Including snapping a photo for my ID badge.

My front lawn is way too short for a ramp.  Now I "just" need a $5,000 to $10,000 chair lift...

End Game

Well we knew it had to happen in the end. 

The brain cancer has won.

You know how they say in the old movies, "Get your affairs in order, you have 3 to 6 months to live?"

Well they phrase the first part differently, the last part is the same.

I have been given 3 to 6 months to live by the doctors. 

Well I don't think the doctors know everything, and I'm trying this funny-looking headgear that makes me look like an Irish Rabbi. Things look bad.

I have granted powers of attorney, made out my will, blah blah blah.

I'm just gonna let them foreclose the house I love. 

I quit my job because I can't type, and a programmer who can't type is not a programmer. 
[I'm dictating this to my daughter, who's taken such good care of me.]

The love of my life hasn't slept for days taking care of me, and my family has had to bear the brunt of my care. I thank them for standing by me during this period.

I hate to beg for money, but I am in dire financial strates. There's a PayPal link in the top corner. Any financial help would be a blessing.

PS
I am maintaining a THANK YOU page, but forgive if I don't update frequently...

Facebook Scrollbars

For reasons beyond my ken, Facebook has a fluid-height IFRAME that seems to "not work" for a lot of people.

Probably people like me who know just enough CSS to be dangerous.

You can Google for various wordings of the problem, and find half a dozen equally good solutions.

Except the ones that use FB.Canvas.setAutoSize, because that's been replaced by the seemingly the same behavior FB.Canvas.setAutoGrow.  So you have to change your code for a (meaningless?) backwards incompatible change. Thanks Facebook. Again. But I digress.
(I welcome comments that explain why this function was renamed.)

You can Google for the FB.Canvas.setAutoGrow solution, but here's a TIP:

If you attempted to get your application to "fill" the page by adding height: 100%; to your CSS, it sort of worked sometimes, but not others, and so you turn to the real solution above...

When you do that, it does NOT work. The 100% "fights with" the FB code to produce, shall we say, less than desirable results, like making your page roughly a mile high.

TIP Rip out those "height: 100%;" CSS attributes.

Using Visioneer 7300 USB scanner with Mac OSX

Using Visioneer 7300 USB scanner with Mac OSX

So I have this Visioneer 7300 USB scanner that was acquired from a firing from a former job, since the owner didn't even want it back.

It's kind of a nice scanner, lightweight, with five programmable buttons on the front, such as scan and print with one click.

The UI for the button programming is about as bad as I've seen. I refuse to provide instructions because I basically click around semi-randomly when I try to program a button until it works. Kind of like Windows.

Anyway, I wanted to hook it up to my MacBook Pro, but it never seemed to work.

Then an upgrade to Windows made it not work there either because there was no driver allegedly, but I found that gimp could use something to make it work.

Actually, I think the driver for the PaperPort software was at fault, but I like gimp interface better anyway. Though gimp does take a long time to start up. Oh well.

So then I tried gimp on the Mac. With a bit of effort, I made it work.

1. Starting on the sane homepage I found a supported devices link.
2. I clicked through to the vendor-ordered devices and found my device.
3. I then did the usual "sudo port search" to find sane stuff.
4. I then did my usual "install everything that looks useful":
• sudo port install sane-backends sane-frontends twain-sane xsane
• You may not be that indiscriminate however. Good luck.
5. I tried gimp in a terminal, and it wouldn't start.
6. So I did "sudo port install gimp" to re-install that.
7. Again typing "gimp" in a terminal seemed to work...
8. However, while I had an Xsane dialog... in my gimp "Create" menu...
• It couldn't find my device.
• The console log in the terminal said:
• Couldn't open firmware file (`/opt/local/share/sane/gt68xx/Cis3r5b1.fw'): No such file or directory
• So I went on a hunt for Cis3r5b1.fw
• I found a download on a Polish (?) site: Despite my complete lack of understanding of Polish, the download button was rather obvious.
• Anybody remember when the internet was almost all English, and the Russian site 'demos" opened up and everybody and their brother sent them "welcome" emails and wanted to know what kind of demos they were working on and it was "Demos" the moon of Mars... I think they were astronomers or something. Sorry. I digress. 
9. I created the gt86xx director in /opt/local/share/sane/ and downloaded the file there.
10. A restart of X11 and gimp and Bob's your uncle.
11. I made a test scan, and it took forever, and the white/black/grey scale was all whack.
12. But the next scan seemed to go faster, and I assume I can tweak settings more or less at random until the colors/grays come out right.

Anyway, that's how I did it, and it might work for you, or not. And it even might be helpful for a different scanner. Or not.

Or even a different platform that can use sane. Or not.

I may have left out a minor step here or there. I'm horrible at taking notes because I usually figure I won't get it to work anyway.

The Cost of Connectivity

synopsis: The US Sucks

Public interest and consumer advocates have cited the U.S.’s recent decline in global competitiveness as evidence of the need for policy reforms that will spur greater competition and investment from the nation’s Internet providers. Meanwhile, opponents of this view have dismissed these international rankings as deeply flawed and akin to comparing apples and oranges. They cite the United States' significantly lower population density and larger geographic size relative to other nations at the top of the rankings,which tend to have smaller land areas and greater population densities. emphasis mine

Are the opponents of fair competition seriously suggesting that a survey of major cities around the world is biased to a great effect by population density?

Yes, USA has a serious "urban sprawl" effect. One has to only look at L.A., New York Metro and Chicago to see that.

But, really, is it any easier in Hong Kong, Tokyo, Seoul, Paris, Berlin, London or Amsterdam to tear up streets and install fiber?

Actually, if you think about it, the "urban sprawl" makes it easier

Sure, buying the fiber and running it a longer distance is more expensive.

But they've got more "elbow" room in those cities.

How in the world did they find any room in Tokyo to run fiber? The whole city is a spire with tiny apartments/condos crammed on top of each other like a Jenga game.

Have you ever seen the traffic in Paris or London?

Hong Kong? You can't even breathe without invading somebody's personal space.

The US doesn't even have a single city in the top 50. You have to scan down to top 90 to hit Los Angeles.

Does that prove their point? Or does that population density make it much more expensive to run fiber?

In fact, if you Google for prices of laying fiber, you will find the same answer everywhere: Population density is the #1 cost factor. They're running fiber inside sewers just to avoid dealing with tearing up streets!

[Bleep]y Error Messages

[Bleep]y Error Messages.
(My daughters might read this blog, so I need to keep it G audience.)

I encountered yet another useless error message today.

Not that I've blogged before about this, but any developer, any user has experienced this, so I mean yet another useless error message from the corpus of all useless error messages inflicted upon any user anywhere, any time.

And they are legion.

So I won't point out the specific vendor/software/whatever.  I'll just rant about what makes a GOOD error message.

Are you smarter than a 5^th grader?

Maybe it was just my school, but we learned the basic Journalism/Essay rules in 5^th grade:

• Who
• What
• Where
• When
• Why

5^th Grade. Five Ws.

Let's apply this to Error Messages:

WHO

Exactly WHO is throwing this error message at me?
For some kind of device, like coffee maker, router, car, ok, this should be obvious.
Maybe.
Some routers have mini-web-servers built in, so maybe not...
Maybe the coffee-maker has different components.
Certainly today's cars are complicated enough.
At any rate, all the software/hardware involved should identify WHO they are in the error message:

• OS Error: Permission Denied.
• MySQL  Error: Permission Denied.
• PHP Error: Permission Denied.
• XYZ (application) Error: Permission Denied.

Every one of those would require radically different approaches to diagnose and correct.

If you, the software/hardware developer are bubbling up or passing on error messages, or to be kosher to an end user, suppressing the error message in favor of a "Something Went Wrong" error message, and logging the real message somewhere for Developers to look at, you still can at least tell the end user WHO is throwing the error message.

Even if you want to hide the inner implementation (E.g. particular Database vendor), at least tell me it's a database error that went wrong!

As a user, I can make an educated guess what course of action to take, or whom to contact within your/my/our organization.

And if the error message is for a Developer or needn't be masked for Security, I need to know who is throwing the error message.

Today's systems are too complex and too many error messages are being passed through, with all context lost.

 WHAT

What exactly went wrong? I need context. I'm not knee-deep into your code, hardware, architecture. I need enough context that I can figure out what actually happened, from Square One of knowledge.

All too often you find an error message that makes zero sense to anybody not intimately familiar with the software.

WHERE

I don't need just filename, I need the line number.
I don't need just filename, I need the full path.
In all probability, whatever you choose for a filename is going to be used by somebody else as well.
Let me know exactly where the error originates.
Is it configuration?  Business logic? Resources?

WHEN

I can read the calendar and a clock, so if the error is in real-time, I don't mean that.
Even for an asynchronous error  generated/read, I still need...
When did this error occur relative to what the software was trying to do.
Was it reading the disk, writing to RAM, reading an RSS feed, or trying to feed the pigeons.
I can't begin to diagnose what went wrong with whatever you were doing if I don't know when it happened, relative to your workflow.

WHY

Why are we here? Not in a philosophical sense,  but why is this an error in the first place? Why do you need whatever you need to carry on what you are doing? Maybe your idea of the one true source of what you need doesn't match mine. Perhaps I can provide an alternate source.

And a bonus one:

WHAT NEXT?

If you're going to tell me something went wrong, you probably have a pretty good idea what I should do to correct it. So why not tell me what to do next, or at least list some actions that will probably be useful.
Whether I have to sacrifice a rubber chicken or poke a voodoo doll or tighten the wing nut, let me know.  Odds are you already know what I need to do.  Tell me. Don't make me spend an hour googling and researching all about your software/hardware/whatever to find out I only have to turn the amp up to 11.
Give me very specific instructions. If it's really long and complicated, give me the references I need to get the instructions.  Preferably multiple references.  I may not have the user manual.  I may not have Internet access. Don't tell me of only one of the two resources. List both.

Maybe I'm just getting too old and turning into a grumpy old man. But I'm very weary of spending my time researching your error messages because you can't communicate. For the same effort, word-count, whatever, except possible thought on your part, you can give me a better error message.

Maybe you think I'm being lazy.
Maybe you think I should just shut up and take it, just like everybody else.

Maybe you think I'm asking to be spoon-fed everything.

After 20+ years of software development, I've served my time reading [bleep]y error messages and trying to make some sense of them.

What we have here is a failure to communicate.

It's not me.

PHP Caches (APC et al)

I occasionally still see people referencing ancient misinformation that various PHP bytecode caches are giving huge performance boosts by caching the bytecode so the PHP bytecode parser/compiler doesn't have to do the monumental task of reading PHP source and converting it into bytecode to be run by the Zend Engine.

Nothing could be further from the truth.

Okay, there is one tiny bit of truth in there.

The php process/program/runtime, without a bytecode cache, is in fact a JIT parser/compiler to a bytecode, which is then run by the Zend Engine.


Amd. okay, the various caches all cache the bytecode as part of the process of gaining huge performance boosts for most PHP web applications.

So that's two tiny bits of truth in a monumentally flawed statement.


But the significant boost is not from bypassing the parser/compiler.

APC and other caching mechanisms save a great deal of time by not hitting the hard disk to load the script, but keeping it in RAM, if possible. Hard disks are slow.  RAM is fast.

The bytecode is saved in cache instead of source, which does bypass the PHP parser/compiler.

But that's just "gravy"

Compare two following psuedo code samples that describe the difference the APC (or other cache) makes:

Code Listing #1

//save hitting the hard disk
if ( $source_code = in_cache($path) ){
  //got the source code from cache, do nothing
}
else{
  //file_get_contents = super-duper slow!!!
  $source_code = file_get_contents($path);
}
$bytecode = zend_parse($source_code);
zend_execute($bytecode);

//Code Listing #2

//save hitting the hard disk
//and a small bonus, cache the bytecode, not source:
if ( $bytecode = in_cache($path) ){
  //got the bytecode from cache, do nothing
}
else{
  //file_get_contents = super-duper slow!!!
  $source_code = file_get_contents($path);
  $bytecode = zend_parse($source_code);
}
zend_execute($bytecode);

As you can see, both code listings bypass the hard drive access which is super-duper slow.

But the second one also bypasses the PHP parser/compiler, simply because it's such a trivial difference, just moving one single line of code into the conditional, to cache $bytecode versus $source_code.

The savings from parsing is chump change compared to disk I/O.

It's also trivial chump change to implement.

But every ounce counts, so all the "bytecode caching" (sic) mechanisms do it this way.

They should all really be called "RAM caching, with bytecode gravy" or even "RAM caching" and just ignore the minimal difference between source versus bytecode.

I'm sure this post is going to make all the caching implementers run out and change their documentation etc. :-)

Well, at least you now understand what "bytecode cache" really means. I'll call it a "win".

Rebuild theme registry on every page.

Rebuild theme registry on every page.

During theme development, it can be very useful to continuously rebuild the theme registry. WARNING: this is a huge performance penalty and must be turned off on production websites.

The last couple weeks, I have found out the hard way exactly what this is talking about... :-(

You probably won't really notice it on page load, unless your site is high traffic.

But even a LOW traffic site, with MySQL replication set up, your MASTER hard drive is going to fill up very quickly.

We were generating 4.4G per day on a low-traffic site.

This is because re-building the theme registry on every page deletes all the {variables} and the whole theme registry and then re-INSERTs them all on every page hit.

Just flipping that off on ONE low-traffic site brought disk usage down to 400M per day.

When keeping several days' worth of mysql binary logs, that difference adds up quickly.

Note that only SOME themes even have such a checkbox in their "Configuration" panel. The most notable one (that I know of) is Zen or any theme built on Zen.

Drupal 6 Performance, Part 3

After weeks of no problems in STAGING and loadtests looking great, it turns out there's just one teensy little flaw in caching Drupal pages longer than a couple days...

Drupal nukes the CSS/JS optimized/consolidated temp files every couple days.

So the monitoring sees a nice valid HTML from an HTTP 200, but humans see a theme-less site.

Worse, the thing corrects itself somehow, or our guys jump on it and flush the cache, and it's all good...

For a couple days.

Rinse, repeat.

Digging into the Drupal caching source code even deeper, and I'm convinced: This things wasn't architected; It just grew.

It's a spaghetti code mess.

I defy any Drupal core dev to correctly describe the caching "architecture" of D6 pages, CSS, JS, blocks, views, etc in any coherent way.

I officially give up now.

When the VP of Marketing complains the site is slow, I'll just say "Yes, it is. That's Drupal."

We'll have to spider the silly thing after every cron job, actually, instead of doing that. Which sucks, but there it is.

Drupal 6 Performance, Part 2

First, I was wrong.

I was wrong on two points.

Number 1 Mistake:

I thought that there would be a slow query in the mysql slow query log that, with an index, would magically make Drupal 6 faster.

There wasn't.

There were a few slow queries, but they had perfectly good indexes, and they were slow, as far as I can tell from the symptoms, due to thread contention, as described here:
Facebook Doubles MySQL Throughput with PMP

Unfortunately, there are three problems with the solution there.
1) It would take me forever to figure out what those guys are really doing
2) We'd never be able to run un-approved MySQL patches like that
3) There's no way my VP of Marketing is pounding our server hard enough for this to be the problem I'm trying to solve.

So as much as I'd love to dive in and follow in Domas' and Mark's footsteps, that's not going to happen.

Number 2 Mistake:

In my previous article, I stated that I had attempted to eliminate some obvious candidates, and one of those was:
"It only fails after a cache flush? No."

I was wrong.

So very very wrong.

I had eliminated the manual cache flushes done by our team as a source of the problem.

What I didn't know was that Drupal 6 (and 7) has a brain-dead hard-wired notion that a "Page" in cache_page should be flushed every time cron runs.

Yes, there is a setting for cache_lifetime on your admin "Performance" page, but page_set_cache ignores that silly little administrative option.
(What actually honors that setting is perhaps an even more interesting architectural decision than flushing the Page cache every cron run.)

This is apparently a known issue as described here:
Drupals cron kills you

Note that a couple high-profile Drupal community members who should know better make blatantly wrong statements in the Comments.

I have broken the rules and patched Drupal core, and probably broken polls (we don't use them).

In your includes/common.inc file, you can apply a patch like this:

cat common.inc.patch_cache_page_honor_cache_lifetime 
--- common.inc.original 2011-09-13 09:37:30.000000000 -0500
+++ common.inc  2011-09-13 09:39:04.000000000 -0500
@@ -2701,7 +2701,7 @@
         $data = gzencode($data, 9, FORCE_GZIP);
       }
       ob_end_flush();
-      cache_set($base_root . request_uri(), $data, 'cache_page', CACHE_TEMPORARY, drupal_get_headers());
+      cache_set($base_root . request_uri(), $data, 'cache_page', variable_get('cache_lifetime', CACHE_TEMPORARY), drupal_get_headers());
     }
   }
 }

Our Drupal pages now are cached honoring the cache_lifetime setting in Performance (as I expected for the past 18+ months) and we got consistent 100 requests per second across 15 multisites in loadtests last night, as we should.

I probably should have run more nights of loadtests before posting this, but I wanted to get this typed up while it was fresh, and, really, now that I know what's going on, I'm pretty confident of this solution.

Actually, I might just change that again to CACHE_PERMANENT, as caching a page only for a day when they change very seldom is just silly. But that's probably just us. Most Drupal users probably expect the behaviour of the patch above, based on the UI.

Here's one site's zoomed in graph comparing the 13th (with Drupal pages at 0 for failing to respond at all to 90% of my ab -n 1000 -c 100) but on the 14th, similar pages get ~100 requests per second.

Note that the highest peaks in the above graph are static images and JS/CSS files, not Drupal pages.

I'm expecting my loadtest graphs to get much smoother now, with no 0s littered everywhere.

Drupal 6 Performance

There are a lot of things I like about Drupal.

Performance, however, is not one of them.

Yes, I turned caching on.
Yes, CSS and JS Optimization are on.
Yes, I used YSlow.
Yes, the GA code is at the bottom.
Yes, the images are small.
Yes, our custom HTML templates validate.
Yes, I have fought off the business unit's demands to install modules recommended by SEO "experts" (cough, cough) which are completely inapplicable to our situation. (Example: When all your content is input by staff, you really don't need that spam detector in the cloud service module. If our staff start putting in Viagra ads, we'll just fire them.)

And, yes, I have a problem.

Well, several problems, actually, but if the preceding example didn't already clue you in that the business process is quite disfunctional, and that was only the tip of the iceberg, you haven't worked for enough large corporations, and there is nowhere near enough room in this blog post to pull that iceberg into view. I'll get back on focus now.

Here's the thing: Most of the time, Drupal is hitting its built-in cache with an indexed query of the entire HTML page for an anonymous user.
This is, naturally, quite acceptable, as it simply bootstraps a few too many PHP files (minor annoyance) and runs a single indexed query on a URL->HTML relationship and spits out the page and exits.

Unfortunately, my performance "pre-emptive monitoring. quasi-loadtest, and let's make sure none of these sites will make PRODUCTION fall over, and let's catch when the Drupal guy didn't turn caching on" ad hoc process, indicates that occasionally, some URL is performing abysmally [viz].

I am not an expert in testing, nor Drupal, nor mysql. But I have spent 25+ years specializing in PHP applications, so know a little bit about all those things. If my methodology is a bit "off", I am pretty confident the results are still "close enough".

Specifics:

I have 15 multisites same-codebase different-database D6 sites on a STAGING box, where business units QA (sort of) and approve (or not, or shouldn't have but did) before launch to PRODUCTION. Staging is a 2-tier setup, with 1 www node and 1 mysql node. We're running stock RHEL plus EPEL builds for the most part. PHP 5.3, MySQL 5.1. If you've done Drupal multisite, I haven't done anything particularly weird or interesting here.

There are also a couple "baseline" sites, with just static HTML pages, "phpinfo();", and a "mysql_connect(); mysql_query('select version');" for comparison purposes with the real sites, just so I know that there's nothing inherently wrong with Apache, PHP, nor PHP-to-mysql.

At night, while real people sleep, I spider each site on a configurable schedule, mostly weekly, and store all the URLs in a flat file per site. In addition, every 5 minutes, an ab loadtest fires for the "next" site in a queue for a randomly-selected URL from that site:

ab -n 1000 -c 100 $random_url_here


Some of the URLs are images. Some are CSS/JS. Some are actual Drupal nodes.

I record the ab result files, prepending a line to tell me which URL was requested.

I also log the requests per second in a database table, counting a big fat 0 for any errors I can detect. For example, if there were any non-2xx HTTP response codes, it's counted a complete failure in my book. If any of the 1000 requests never came back, again, big fat 0. (This is where testing experts are probably crying in their coffee. Sorry.)

I have some kludgy-hand made PHP|GD graphs.
(Yes, I'm old-school and can visualize and draw a halfway decent graph in PHP code faster than I can fight your fancy new-fangled tools to get a really nice graph. Sorry.)

Here is what a sample site graph looks like:

(The density on the left is because I used to run the loadtests all day every day, but scaled back to night-time in the middle of this process. Yes, testing experts everywhere are crying in their coffee. Sorry.)

First thing to look at is the bottom center stats min/avg/max: 0/265/1010
That means this site is getting a very acceptable "average" 265 requests per second across all the ab runs you see in the graph.
And that 1010 means that one URL excelled and had 1,010 requests per second in an ab run.
Alas, there is a 0 for that min, and that's the problem.

In fact, looking at the green lines of the graph itself, you see a fair number of 0 data points.

I won't bore you with hundreds of ab text outputs, nor the tedious process of matching them up with the dots on the graph, but here's the basic trend:
Static CSS/JS and even images come in first place with the peaks in the hundreds of requests per second.
Most Drupal URLs come in acceptable levels around 100 requests per second.
Occasional Drupal URLs just plain bomb out with, for example, only 8 successful requests, and 992 failures and get the big fat 0.

I have also attempted to eliminate the following:

• The same URL fails all the time? No.
• It only fails after a cache flush? No.
• It only fails when a loadtest runs so long it concurs with another? No, I added code to insure only one loadtest at a time. (Testing experts crying over coffee. Sorry.)
• Only certain sites? No.
• MySQL hasn't been tuned? No. We tuned it.
• APC isn't working? No. It's fine.
• Uncached Drupal performance isn't that bad. Yes. It is. I measured it with the same (sloppy) metrics above. A simple minimal setup with stock D6 and a few pages / modules falls over after 8 requests in heavy load, without caching.

So my theory, and I admit it's not really proven, since I don't have the mad Drupal skillz to log the info to prove it, is that the setting in Performance "Minimum cache lifetime: <none>" does not, in fact, keep URLs in the cache indefinitely, and something in Drupal purges them, for reasons beyond my ken.

NOTE:
These are brochureware sites with a hundred nodes each, and a couple webforms. There are the usual Drupal modules one would expect for that. Plus all the modules I was forced to add that they never actually figure out how to configure and use that the SEO "experts" (cough, cough) told them they need.

A lot of Drupal experts have recommended PressFlow and varnish and so on. We are considering that. But even a drop-in replacement isn't that easy to manage when we are rapid-fire launching a new multisite each week. We simply don't have the bandwidth for that right now.

And, ultimately, I find it offensive that uncached Drupal falls over after 8 requests on a 2-tier php/mysql 6G RAM setup. (Assuming my theory is correct.)

It's like somebody bolted a billboard to the back of our Maserati.

Then tossed some kind of broadcast instant replay device named "cache" in front of the billboard to mask the problem.

And are now recommeding another broadcast replay device in front of the "cache" called "varnish".

So am I just going to whine about it? No. I'm going to start analyzing queries in log_slow_queries from MySQL and whatever else I can find, and try to see if I can add some indexes to make uncached Drupal reasonably performant.

Of course, real Drupal developers are all madly coding Drupal 8, since Drupal 7 was released a few months ago, and Drupal 6, to them, is just not worth patching, so my efforts will probably be largely irrelevant to the Drupal community at large. Sorry.

You'll have to wait for the next blog post to see if that succeeds or not. I may even have to eat some crow if I can't make any improvement at all.

Again, I do like a lot of things about Drupal. Performance, however, and the architecture to make it performant, is not one of them.

Xdebug and RHEL 5.7

Much ado and misinformation about how to really profile or performance analyze PHP is all over the Internet.

Rebuffs also exist to much of that misinformation all over the Internet.

How to actually really set yourself up to properly profile PHP, not so much...

So, here's an attempt to provide a step-by-step process, at least for RHEL 5.7

If you run something else, you'll have to improvise a bit.

It's not like I really picked RHEL, so that probably deserves some background...

I'm mostly an old school compile from source guy for Apache, MySQL and PHP to get what I want.

I used to use RedHat/Fedora for the base Linux / desktop, and then download source for Apache, PHP, MySQL.

I switched to gentoo a few years ago, and admit I pretty much fail to update very often, but whatever.

I only have to maintain a handful of boxes, and update when there is a real need.

In my workplace, the sysadmin is a stock RHEL guy, but then he has to maintain hundreds of servers with puppet and monitor them with Nagios and...

Well, let's just say I completely understand that need.

And he HAS opened up to Dag's repos and a UIC or ULC or somesuch to get something I needed.

Alas, when you want to do serious debugging, profiling, or performance analysis, that's not good enough.

So I recently embarked on a task to build a "debug" desktop with the following goals:

• have debug builds of php, apache, and mysql
• have xdebug
• have kCacheGrind / valgrind / callgrind to visualize bottlenecks
• this box is used for debug build, nothing else, no other users but me
• stray no further from our stock build than I have to

Of course, just installing the GUI is a pretty far straying, and I let RHEL push me into gnome rather than KDE. Not sure that was wise, but there it is.

Still, I tried to have minimal impact on the build.

Testing Experts/Pundits:
Note that I'll exit X windows once I'm ready to really test, and then fire it up again to examine the results. If having the X binaries on the box screws up my tests, so be it. I'm already replacing the REAL binaries of the code with these debug-enhanced ones anyway, so at that level, it's not "the same" anyway. Having a separate desktop with all these tools on it just didn't seem practical in my case. Your needs may vary.

I first attempted to get the debug builds of at least PHP, and hopefully Apache and MySQL, just in case.

Everything seemed to work with yum, but I spun my wheels for hours checking phpinfo() configure line for --disable-debug and trusting its output about debug build.

It turns out that the RHEL debug builds are actually not built into the same binary files that ./configure --enable-debug would do. Instead, the debugging symbols are installed in separate binaries in /usr/lib/debug, and tools like gdb and strace and the all-important (to me) "httpd -X" just "know" to look in there and load those debug symbols in on top of the non-debug binaries in the usual places.

That kind of blew my mind, but whatever, you know?

And why the httpd startup scripts and PHP don't know to look there is beyond me, but I suppose if I really need to force them to do it, I could find a way... I doubt I'll need that, as you'll see shortly.

Anyway, in order to actually get these debug builds, you have to do something like this:

yum install --enable-repo=rhel-debuginfo httpd-debuginfo mysql-debuginfo php53-debuginfo

Note that this installs only the debugging symbol binaries into /usr/lib/debug. If you actually want to install the software itself, you still need to do these if you didn't already:

yum install httpd mysql php53


You also want to install valgrind, which basically allows you to run your binaries under debugging conditions with a whole suite of tools. We'll look at just one tool, callgrind, because a) it's the only one I know so far and b) it's the one that makes pretty pictures that make it blindingly obvious where your bottlenecks are in your code.

yum install valgrind


Just to be sure you installed Apache and PHP, set up your usual PHP script for testing purposes:

echo "<?php phpinfo();?>" > /var/www/html/index.php

(Or edit /var/www/html/index.php with your favorite editor.)

Surf to http://localhost/ and see the PHP status page with lots of blue tables telling you everything you ever needed to know about your PHP install.

Now to prove you can generate a callgrind script, to generate a visual representation of where your code spends all its time:

/etc/init.d/httpd stop
valgrind --tool=callgrind --dump-instr=yes -v /usr/sbin/httpd -X

(Someday, RHEL will catch up to the rest of the world and that httpd will change to apache or apache2...)

Note that you are now running a single httpd instance/child/thread with debugging on full blast, instead of your usual high-performance httpd process.

You definitely do not want to do this on a production box, or even a shared dev box.

You need to be doing this on a sandbox all your own.
Really.
If you can't figure out why, stop reading now and go do something else.

Now, reload that phpinfo page in your browser.

Finally, kill that httpd process. Open another shell and do:

killall -9 httpd
callgrind_control -k

If you kill the process, and it's built up a HUGE file of data, it just dies before it writes the data. Don't do that.

Back in the shell where you did that valgrind command, it will have exited. It will also have dumped a file whose name starts with callgrind.out. and ends with the PID (process ID) of the httpd process that was started.

You can just safely assume it's a random number, if you are not familiar with process IDs. Actually, if you're not familiar with process IDs, this article is way too advanced for you. Oh well.

You can open that file up in an editor if you like.

It won't make much sense, really, but it's kind of cool to see it.

Don't change anything in that file, for goodness' sake. We've worked very hard to build it.

Since you can't read the silly thing, let's install some software that can: kCacheGrind:

yum install kdesdk

Don't ask me how you're supposed to figure out that kCacheGrind comes as a part of the KDE SDK tool-suite. I used Google. A lot.

Now you can run that tool to see how phpinfo() spends all its time:

kcachegrind callgrind.out.*

You have to do this in the same shell/directory where you did the valgrind, just in case you didn't figure that out on your own.

You ought to see something not unlike this:

Whoa!
What's all that, you ask?

Focus on the big splotches of color. The bigger the rectangle, the more time PHP spent in that function.

As you can see, PHP spent a lot of time calling various _dl_* functions, presumably loading up code binaries, or double-secret internal functions you should ignore for now.

But you can also see a big chunk for good old strlen

Now, obviously, you are not likely to jump in and edit PHP's strlen function and make it faster, but the point is that you now know where PHP is spending all its time in your PHP script.

Of course, with phpinfo() as the only function call, that's pretty boring.

Install your favorite framework or large application project (Drupal, WordPress, Zend Framework) or your own home-brew MVC, and load a few test pages.

Heck, just copy a whole site that you think is "too slow" onto this box in /var/www/html/ and fire it up and find out!

But wait, you say, I'm just a PHP developer. I'm not going to be re-writing PHP internals!

Well, guess what?

This genius Derick Rethans has a tool called Xdebug that you can use to do this same thing for your PHP code.

You'll have to compile it from source, because there are no packages for Xdebug yet.

Or use PECL, if you've managed to install that.

I have not installed PECL yet, so I just downloaded the "source" and did the usual:

./configure
make
make install


Oh yeah. Something should be said here about libedit if you want readline-like abilities in the debugclient.

I pretty much said "Meh." and went on with life, once I realized that I'd have to fight RHEL for another hour or two to get that.

If you need that feature in our PHP debugger, you're on your own. Sorry.

Now, add a few settings anywhere you like in /etc/php.ini:

; Added xdebug settings
zend_extension="/root/installers/xdebug-2.1.2/modules/xdebug.so"

; Disallow usage of @ to suppress errors
xdebug.scream = 1

; Turn on profiling
xdebug.profiler_enable = 1

; Dump the enormous files here
xdebug.profiler_output_dir = "/tmp/xdebug"


And, so apache can dump the files there a couple shell commands:

mkdir /tmp/xdebug
chown apache /tmp/xdebug
chmod 775 /tmp/xdebug


You have to restart apache for the php.ini changes to take effect:

/etc/init.d/httpd restart

Oh. It wasn't started because we killed it and ran it with valgrind before.
Oh well.

Anyway, reload your phpinfo() page (or other) from http://localhost/index.php

Now go look in your /tmp/xdebug directory for some callgrind.out.* files, and load them up into kCacheGrind.

Voila! You now know where your PHP code is spending all its time, just by looking for the "big" rectangles.

kCacheGrind has a zillion options and tools to slice and dice your data, and different ways to show it off.

It also can export the pretty pictures, if you have graphviz and kghostview.

GraphViz is pretty easy. Snag the repo definitions from GraphViz RHEL Repo and put it in /etc/yum.repos.d/graphviz-rhel.repo file.


yum install graphviz


Finding kGhostView was quite frustrating. It seems like RHEL is intent on stamping out KDE, and kGhostView in particular since 2002 or so. I found all kinds of rpms from that era, and not much after.

So we turn to our friends at EPEL, and use their rpm which installs and maintains their repo definitions into /etc/yum.repos.d. Did you follow that?

Just download the appropriate rpm from here:
EPEL rpm to expand yum

Now at this point, with EPEL installed, I pretty much opened myself up to straying very far from RHEL, but all I want is kghostview so kCacheGrind can export pretty images:

yum install kghostview


Woof.


I'm pondering if I should now uninstall the EPEL stuff, just to stay on the straight and narrow, and let kghostview lag, or try to remember to only use EPEL in dire circumstances...

Well, anyway, you probably don't care about that bit.

You now should have a very powerful tool suite to really find the bottlenecks in your PHP code, instead of trying some Voodoo Analysis to "optimize" things like changing all "" to '' or using a single (un-indexed) query instead of two (indexed) queries, because running more queries is slower, or...

I know, I promised not to go there in the first paragraph, so I'll shut up now.

Brain Surgery

On August 31, 2010, I went into brain surgery.

This is a "delayed" blog post about that.

It got rather long, I'm afraid, which is why it took so long to get up here...

Plus I've been kind of busy until now, where I have a 2-week lull between post-op and Radiation.

But people seem more interested in the actual surgery than anything else, so here is a play-by-play of what I remember from it.


The day started early, with a 5:45am arrival for a brain-mapping GPS MRI scan
and 7:30 surgery. I had been warned I might lose my hands for a week, as they were probably going to "nudge" my motor strip in that area when they cut out the tumor.

After they put all those circular markers on my head, they scanned me, and then I guess dumped a map of my brain to the surgeon's computer.

I was then wheeled into Pre-op which is a fancy word for hallway with curtains.
Felt kind of like being in the wings of a particularly clean Theatre stage for any
Old Troopers. I guess it makes sense for resource management, but it sure doesn't seem very nice. I was lucky not to be too back-logged I guess. I imagine on a bad day, you could end up waiting there for hours and hours.

Dr James P. Chandler

Then started the introductions. I had already met the surgeon Dr Chandler, but was introduced to at least three anesthesiologist including Dr. Koht and his two assistants, and the Chief Resident, Ryan Halpin, who turned out was my "coach" for the surgery (viz) and he seemed to be the one watching the heart and other squiggly lines monitor that I could also see. Though I daresay there were many others monitoring all kinds of stuff I never even saw.

I think I got re-introduced to some doctors/nurses, and I'm told surgery started 2 hours late, but I didn't notice, so maybe that was their plan... Just keep me trying to remember names and faces and distract me :-)

Anyway, they wheeled me in, hooked up an IV or two, and conked me out. They apparently locked down my head and connected some plumbing then, for which I'm quite thankful I was not awake. That probably seems normal enough for any surgery. Then they woke me up, still on the table!

They had me move my fingers around and monitored stuff on their computers to make sure things looked right. They cut open my scalp and chopped a hole in my skull. Did I mention I was still awake?!

So then I presume he moved the protective water balloon brain sac aside, and went in through the vein above that red arrow, or moved some brain bits aside...

Then they had me touch each finger to my thumb and said they were getting signals on that.

Then he hooked up a wire to my brain and made a nerve in my right wrist  twitch at 60 Hz and asked "Can you feel that?" Surreal!

If you've ever gotten a light tremor from house current running through you for a few seconds, it was just like that.  Only direct from my brain. And on purpose.

They also confirmed on some med computer that they were seeing the same signal.

Repeated the process, for my right ankle. Then the left ankle. Each time they asked me "Can you feel that?", and confirmed signal with each other.

They didn't do my left wrist, which was disconcerting, but I'm not sure I was able to voice that concern well.

Then they warned me that they were going to do a few more that I wouldn't feel.

There's not supposed to be any nerves inside the brain, but I WAS able to sense each probe in turn as a sort of cool prick. I told them all about it, apparently in more detail than they required: I believe this was the first time they politely told me to be quiet and focus on taking nice deep breaths.

That would be Dr. Ryan Halpin, the "coach" and I have to say he has the patience of a saint. He managed to put up with me and keep me on task while I was halfway zonked out, which is a miracle.

They ran through all the signals again, confirming I could feel the nerve twitching at 60Hz and seeing the signals on their computers. Apparently "measure twice, cut once" applies to brain surgery as well as lumber.

Come to think of it, counting up MRIs (regular, functional, spectroscopic, and mapping) as well as CT scans and the probes, it was more like "measure 10X, cut once." Same principle at least, but I suppose a factor of 10 for Brain Surgery makes sense.  I mean, tossing a 2x4 out is one thing... :-)

Then Dr Chandler cut out the actual tumor. I think this may have been another time I was politely told to shut up and breathe. It seemed like the actual tumor removal only took 10 minutes, but my time sense wasn't exactly perfect, eh?

Finally they started putting things back together. There's a chunk of skull held in by titanium plates and screws (Looking for a pic of these). No MRI or airport restrictions on those, which is nice.

There is apparently some glue in there holding something together as well.  I wonder if they have a $10,000 hot glue gun just like my $20 one at home, or if it's some kind of packet like a condiment they use.

Then the scalp was stapled shut, as you can see to the right:
I kind of expected some stitches, but if I have any, I sure can't find them, even now that the staples are out.


At some point the doc asked for a 1x3, but the nurse only had a 2x2. They went back and forth a few times on that, which was disconcerting. I mean, who cares about the shape of the scar or whatever, but the wrong size screws or plate inside?... I didn't know exactly what stage of re-assembly we were in, so I tried to make a joke about lumber, but it fell flat. Oh well. Tough crowd I guess.

(Weeks later I found out it was just a gauze swab to wipe out some blood or something. Whew!)

They also didn't like when I broke into a song lyric with a trigger word/phrase somebody said; which anybody who knows me is a habit of mine. Guess I don't sing any better while in "twilight" anesthesia. But I only did that once, so that's not so bad! A musician friend of mine tells me she had the whole operating room staff singing with her. That was definitely one of the times I was politely told to shut up and breathe deeply.

Then they wheeled me out to post-op (read: hallway curtains) where I spent 2 hours needing to be there, and six more hanging around waiting for a room, thanks to construction... I've got a few pithy comments about post-op and a certain nurse, but will save those as I've already made this quite long.

Anyway, that's all the bits I can remember from nearly 3 hours of brain surgery. I apologize if it's out of order, or I've managed to completely mess up what they did. All I know is, it was surreal, but they got the tumor out, and over-delivered as my hands worked just fine, albeit with a bit of tingling and numbness now and then. I'm calling it a "win"

Honestly, the only things I could have asked for:

• Advance notice that my main job would be deep breathing.  They had described the finger / nerve twitching process in advance. But perhaps the breathing thing is case-by-case. I probably could have kept more quiet and done more breathing if they had.
• Check that left wrist, or tell me we don't need to.
• Let me know what is going on when the doc asks for a 1x3 and they ain't got it, so I'm not worried about it.

Considering everything else going on, these seem awfully petty!

Brain Surgery Staples Removed

I got my staples removed yesterday, so here's what it looks like today...

So, what I've got going for me:
I'm relatively young, in good health other than this cancer in my brain, and a wild guess estimate is 90% of it was in the tumor they already took out.
I've got just about the most supportive family, friends, and workplace anybody could ask for.

On the downside, if you lump me in with males up to age 90 with brain cancer, my "live expectancy" is a couple years...

 But, hey, let's Do The Math:

A guy 90 years old, already has not much more on "life expectancy"...  There's no numbers I can find breaking it down by decade or anything,  so let's assume it's some kind of curve, and I'm on the right end of the curve.

It blew up fast, over the course of a couple months or even weeks, so there was not much more that could have been done for "early detection".

The docs have caught it in time to get 90% out already, and will radiate and chemo the other 10%.  It will try to grow back, but we'll MRI it every couple months and catch it and chemo it some more, beating it back into submission.  So we have a game plan to keep this under control indefinitely.

Do I expect to make it to 70+, like my dad, as I once did?

No, not really.  I'll give it my best shot though!

Maybe I'll only hit 65, like my Mom.  Still not a bad run.

So what's my "life expectancy"?  I expect to get as much out of it as I can for as long as I can, same as anybody else!

Brain Tumor Pathology

Tumor w/ Surrounding "Bad Stuff" in active tissue

Brain Tumor Pathology
(translated from Medical to English)

Malignant.
Grade 4 (worst)
The surgeon got the main tumor out.
But it's guaranteed to have a splatter effect of Bad Stuff mixed in surrounding brain tissue.
Aggressive radiation/chemo (6wks/6months respectively) will maybe beat it back.
Then, ongoing maintenance of MRI scans every couple months, and chemo when more tumors appear, for life.

I guess that's not as bad as the old school movie "Go home and die" answer, but I think these days it's about as bad as it gets folks...

On the other hand, the oncologist said that the odds of hearing benign were in the realm of "case study in a journal".  I do wish they had just told me that weeks ago. I've been living with a false hope of hearing "benign" and on the hook of not knowing for weeks... Oh well.

I'm staying as positive as I can, but I'm in for a rough ride, and that's the breaks.

Thanks for all your prayers, support, and donations.

I'll try to post more on Friday 9/10. Tomorrow 9/9 is a CT scan with some kind of plastic fencing mask to map where the tumor used to be married with the old MRIs to program the linear accelerator for the radiation treatment.  Or, at least, that's my layman's understanding of what the doctor said...

I think I get my staples out of my scalp tomorrow too.  That's good, as the changing Autumn weather is making them hot/cold expand/contract in an annoying way. That probably sounds silly/petty in the grand scheme of things, but there it is.

Anyway, expect silence Thursday 9/9 from me, almost for sure.  I don't think they let me use my laptop from inside a CT scanner :-)

I also want to thank my employer, Rasmussen who have bent over backwards throughout this ordeal to keep me "employed" and "current" in insurance coverage. I don't know what I'd do without them. If you are looking for a good place to work in the online education industry, check out the openings and tell them I sent you:
Rasmussen Employment Opportunities
They've done right by me, that's for sure!

SEO For Beginners

You're probably even more tired of my posts about Brain Tumor/Surgery than I am, so I'll go off on a "rant" on another topic :-)

I still need to draw your attention to that Donate button over there... Okay, done.

A Beginner's Guide To SEO:
#1: CONTENT IS KING
Put content on your site that people want, and make it easy to find/browse in your own navigation.
If you have content people WANT, and people can find it, and search engines can read it (don't bury it in Flash) then you're all set.
If not, you have no "game" and all the SEO in the world won't do squat for you.

#2
Follow some common-sense basics like using HTML/CSS properly, with valid markup, and a good TITLE tag matching an H1 tag relating to what's on the page.
Put your important content first in the text, and use CSS to position it where you like.
Try to use a decent domain name that you can shout to a roomful of drunk people that they'll remember, maybe, the next day.
Even if your target audience is NOT rooms full of drunken people, it's still a good metric for the domain name.

#3
A META Description is good, if it matches the page/site content, and provides a consistent "brand" (theme/mission/image) you want to promote. You'd have to ask some Marketing drone whether it's a "brand" or an "image".
META Keywords haven't been used by any major search engine in years: Anybody who says different is stupid or out-dated. Or both.

#4
Don't try to "game" the system with techniques that misrepresent your content. As soon as the search engine makers figure out what you and all the other idiots are doing, they DEDUCT points for it, and you lose, Game Over.
Unless they just ban your site completely from the results, in which case you've just wasted your oh-so-precious domain name. GAME OVER!!!

#5, #6, #7, #8, #9: See #1

#10
After you have done that, worry about tweaking the so-called SEO junk to improve your site that last 1%.
Image ALT keywords, re-positioning

PS
I believe most SEO "experts" are charlatans, and I refuse to label myself an SEO expert.
Make of that what you want...

Brain Surgery Survivor

I made it!
Even have fully functional hands.
I feel about like you'd expect when somebody cuts a hole in one's head... :-)
Thanks so much for the prayers of any flavor and donations.
Followup appointments next week are still in pencil due to doctor schedule conflicts, but at lest one of them should tell me something some time next week, regarding malignant/benign and grade/scale or whatever.
Had a screamer on the same floor last night, so didn't sleep much...
Very tired, so I'll end this post here.
Just know I'm alive, and recovering, and will know the Real Story next week...

[EDIT: 2010-09-11]
I added this photo from my sister's phone.  Those white circles were some kind of GPS markers for an MRI mapping that lets the surgeon match up the brain images of the tumor with what's actually in my head.
Had I only known in advance, we probably could have gotten my brain on Google maps! :-)

Brain Tumor

Here is a side-view of the actual tumor.
That darker gray splotch in the top center, shaped somewhat like a guitar pick, only not.
I added a big red arrow, as I realized the the vein above could be mistaken for the tumor.

Brain Tumor

Quick Update:
The neurosurgeon says the Functional MRI and MR Spectroscopy point towards a tumor rather than infection.
Surgery on Tuesday.
Don't know what time until Monday night, as they need to keep their schedule flexible until the day before.
A biopsy for sure, and removal of as much as seems prudent, hopefully all of it.
He can't say for sure until he's in there exactly what can come out and has to stay.
Apparently I'll be conscious while he's operating, so he can double-check various brain bits aren't tied to anything important based on what twitches when he touches it or something.
I'd rather just be knocked out and wake up healed, but I'm sure not gonna argue brain surgery.
I'll probably/maybe have paralyzed hands for a week or so.
Won't have biopsy results for 72 hours after surgery. Guess I'll have a race with the Doc to see if I can walk out of the hospital before he has answers. :-)
Don't be surprised if this blog goes silent from 8/31 through 9/7, give or take...
Thanks for your generous donations!
I'll be swamped in bills soon, so it's much appreciated.

Brain Lesion

On Thursday August 12, I suffered a seizure.
My daughters found me, got Mama, and they all saved my life.
I was rushed to the hospital, where I suffered 3 convulsions before being stabilized in ICU.
I regained consciousness on Saturday August 14th.
After more blood/brain/body tests and scans than I want to list here, they found a lesion in my brain in the frontal(?) posterior lobe.
Apparently, it's not called a tumor until they figure out if it's malignant or not.
I've got a functional MRI, and an MRS (MRI Spectroscopy) in upcoming weeks.
My employer is being incredibly gracious about discretionary rules, and bending over backwards to keep my insurance current.
That said, there are deductibles and living expenses and possibly even a gap coming, depending on how much pressure I can exert on hospitals and labs to expedite things.
I don't really like it, but I've added a "Donate" button over on the right; I'd appreciate any help you can give, whatever you can afford.
I'm not a tax lawyer, but as far as I can figure, an individual like me can't be a non-profit, so it would have to fall under the generic "gift" category on your tax forms...
I'll be in and out of labs and hospitals for a few weeks; I'm not allowed to drive for 6 months (or longer, if I suffer another seizure) and that's just the beginnings of how this complicates things.
Thanks for all your prayers (of any flavor) and support! I've had a lot of help already from friends and family, and it is much appreciated!!!