Server Meltdown…literally

So i was serving rentmappr.com off of my acer laptop running ubuntu 7.10.  It was doing some hardcore multi-threaded processing of craigslist data every 30 minutes so I was using a lot of cpu (like 75% on a dual core machine).  So the acer got kinda hot.  And then my roommate in his infinite knowledge of computers (which is limited to let’s build a windows machine to play counter strike on) decided that my laptop was too hot for not “running any programs because i opened it up and there weren’t any windows open so I just shut it off and unplugged it and put it in your room.”  I did stash the thing in his room because he insists he has a hard line to the router/cable modem for his gaming, and I didn’t want to host a website on wireless.

ANYWAY, I had it fired up for a few days in my room on wireless and then for some reason I unplugged it or I don’t know but I wouldn’t turn back on.  My other roommate about the same time was complaining about a power surge b/c her alarm didn’t go off so I thought maybe my power adapter got fried (wouldn’t be the first time).  So I tried the cord in several different outlets and was sure it was fried so i started to splice the connector onto another laptop power supply and then realized that the internal center pin that the ac adapter plugs into was bent.

I attacked the pin with a screw drive to see if I could bend it into place but it turns out the whole rig just crumbled and fell into the case.  So now it looks like (after trying to get the f-ing case off the acer, and yes I even used an electric jigsaw at one point no dice) that the inner piece that fell off would have to be soldered/epoxy’d back in place to get the damn thing to turn on.

Well there goes 8000 lines of my rentmappr database.

Guess I could pony up for a cheap tower and maybe get a sata external laptop hardrive case to port my old data over oh well guess I shouldn’t be doing hardcore multi-threaded site scraping on a laptop stashed behind my roommates desk.

Ubuntu sucks…get a Mac.

Alright alright, I’m not saying Linux sucks – Linux is awesome for what is does and what it does well – 1. be cheap and 2. be fast – which makes it a great choice for a server – not desktop users.

I got real sick real fast of Vista – for all the reasons I don’t need to explain – and being low on funds I tried out some Ubuntu goodness (not greatness) – you can see some of my endless frustration in previous posts about getting wireless working (not only just working, like turning on, but connecting to WEP and WPA encrypted networks), getting ANY hardware to function well has been a pain in the ass – I don’t want to go hacking system files to get my mouse or monitor work the right (I’m not saying I can’t do it but my mom isn’t a l337 hax0r and if Ubuntu is Linux for people this shit has got to change).

Ok so the point of this is get a Mac. Why? Because for the most part, shit works when you plug it in – monitors, mice, mobile broadband cards, printers, networking, vpn, wireless – all of it – and the OS is fast as shit.  Ok its expensive but Apple has provided the first serious Unix environment intended for real stupid people to use.  the company i work for will only buy macs for the development crew – why – because shit works – you don’t dick around all day with hardware or software issues.  So just save up the cash and get one (mac mini…maybe not…mac book pro…yeah definetly.  Oh and get an external hard drive – we’ve had a couple disk failures but TimeMachine or SuperDuper! will restore your machine in about an hour to the last backup point (apps and all).

Apache on Windows x64 as Load Balancer for Mongrel

Its been awhile since I’ve posted any rails stuff on here but the drudgery with Windows and its 64 bits continues.

Cool things I’ve done with Rails is the past month:

• PDF to HTML converter with multiple file upload (javascript/flash/ruby working together)
• HTML to CSV converter (only works for one report format but its from one of the largest call center operators in the US so…you know whatever)
• FTP downloader (not quite working yet)
• cygwin sort of working (this gets me access to several useful Unix commands from Rails like

  cat

  and

  pdftohtml

• Google maps with route making ability (javascript and ruby working together)
• Google maps routing on custom routes

Ok so I’m awesome for those things lets continue.

Since I needed Unix commands cygwin had to work….BUT Pen, the load balancer I’ve been using, also needs a cygwin dll file in its directory work. I don’t like Pen b/c I can’t get it to start up when or if my server reboots – won’t install as a service in x64 land so its not a good option.

I decided to take the dive into Apache. Once you get past some of the scary config options (not that they are really scary its just daunting looking at config files you’ve never seen before and trying to get the damn thing to work). I followed a great tutorial on getting load balancing working in Ubuntu from UrbanPuddle – and then applied that to windows. The name-based virtual host thing is pretty easy to set up – tell Apache where to listen – port 80 – and then give it a list of possible names that could come in, then redirect that name to the right mongrel processes. We can also tell it to serve static stuff (images, stylesheets, and javascripts) and not bother mongrel for them.

Another problem I’ve had is long running processes. Like – we need to read a CSV file that is 26,000 lines long and do processing on each line and assign sales data. Ok, sure, well the whole thing takes like 8 minutes to run (I have it setup so it will tell you what row your on in the CSV file while it loads and gives you the average load time for each file based on previous runs). I had to do all this b/c the you get an hourglass for 8 minutes – thanks to load balancing other users can still access other parts of the app (you can even pop open a new tab to complete some other task if you want). But the real rub is that Spawn, from Tom Anderson (fellow Denver rubyist) for some reason won’t work on my x64 machine – works awesome in Unix, but no go on windows. It looks like I can get the long running process to fork but any ActiveRecord calls don’t get called after spawn renders my “waiting for data to load” page. If I ask it to wait, then it will wait for the process to finish before rendering a page but that really defeats the purpose b/c I still get 8 minutes of hourglass. Tom added Threading support, which requires I set concurrency to true in my environment file but when I tried this again this morning all I got was unrendered HTML output (looks like apache didn’t like the new settings) and if I hit a mongrel port directly everything renders but I have to reload each page I navigate to for the rendering to take place. Shitty.

I revisited spawn b/c I was getting:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /call_data/load_data.

Reason: Error reading from remote server

Nice…WTF I say. Turnsout this is a timeout problem. Apache defaults to 300 second timeout – awesome for a general website but my shit is an internal app that can take up to 15 minutes to load stuff. So I said, cool lets try Spawn again and see if I can get around not setting the timeout to like 20 minutes (so not safe omg). And whalaa…Spawny no worky.

So I had to set the timeout directive in apache to 1200 seconds. You can do this (at least in my setup) by uncommenting (remove the #)
Include conf/extra/httpd-default.conf
at the bottom of http.conf and then set
Timeout 1200
in httpd-default.conf (in the extra directory) then restart apache – worked for me

At least apache works and it will boot up when the server restarts – I tested it – so I have a persistent ruby on rails application that does not require anyone to administer the server on a crash (like how it should be) only took 10 months to figure out.

Protected: BR BP Questions.

This content is password protected. To view it please enter your password below:

Password:

Capistrano on Windows…totally not possible…damn it!

After a few hours of not finishing this retarded project (we’re into month 4 of two scheduled months) I’ve discovered some things about my server…my windows x64 server 2003.

So I’m not using the acclaimed Capistrano or SVN to manage my projects here – its just me, I’m the only one working on it and there’s one server – no need to spend the learning time upfront for something that I won’t use.  Instead, I like to roll totally unsecure, that’s right…Remote Desktop.  Ok so I was coding in Dreamweaver before I made the switch on my dev box to Ubuntu…I just mapped the drive where my app lives on the server and I edit them directly in DW.  Now I use a TextMate-ified gedit to do the same thing on Ubuntu.  Well I have this process that occasionally gives me Errno::EINVAL and I have to restart the server…this requires remote desktop login, open services, and click restart on the mongrel service (see previous post on how to get this work in x64land) .

General waste of time.  So then I think – it would be nice to use cap restart and wha-la I’m back in business.  Well this requires, for Windows, Cygwin, the sshd module for Cygwin, SVN, and the Capistrano gem ($ gem install -y capistrano).  Because Cap basically logs into to each server you list using S(ecure)SH(ell), you have to have a ssh server running on all your servers. Cake if your in *nix and Windows x32.  I found a somewhat confusing tutorial on setting up Cygwin for ssh from Colorado School of Mines.  It got me as far as getting ssh service installed.  However, sshd in Cygwin does not have x64 support.  So basically you can’t run a shh server from Cygwin, and Cap can’t run its unix commands on your windows x64 system(update code from repository, restart webservers, etc).

This is really unfortunate.  I really wanted to be able to gracefully update and restart my server from afar.  I’m disapointed I can’t use tools like Capote, by Fernand Galiana (a fellow member of Derailed, the Denver Ruby on Rails UG he founded, and also he is the genius behind ZiYa Charting for Rails) Capote is a web-based Capistrano manager – He took the command line techniques of Cap and turned it into a graceful web app that can manage all your Cap recipes…but for me too bad.  I did get a commercial SSH for free (I think? SSH Tectia) and I can now at least get a secure command line from afar.  I guess I could still run Cap, but I’m spent with this Windows tomfoolery

Silly rabbit, auto complete in Outlook isn’t a safe way to store contact info.

We had a new sales guy start today and he needed his “contacts” transferred to his office machine from his laptop.  Turns out his contacts are actually just the output from Outlook’s auto complete feature.  Turns out this info is stored in a .nk2 file locally.

After poking around some forums I found a program that will parse this file into a CSV so you upload them to Outlook as real Contacts.   I downloaded it and it worked great.  It puts a shameless plug into each contact you export/upload but I didn’t have to pay for the software and it worked well and fast.

Load Balancing Mongrel with Pen on Windows 2003 server x64 64bit 64 bit

Alright – so there was a stupid software company. And they write in ASP.net and c# for web based apps. Ew.

So they said – we need a windows server that has 64 bits.

This server is also the server my company wants me to deploy my Ruby on Rails app onto, which is not fun.

First off its windows…and windows doesn’t always place nice unlike its friendly penguin counterpart…oh and this other company wanted everything running out of Microsoft SQL instead of MySQL…ew again.

So I complied and got my app switched to MsSQL (that’s a whole other post). And its nearly impossible to run a rails app behind another app that’s using IIS on a 64 bit machine (it requires $100 software and that’s another post too.)

Hokay so, turns out the other company’s program didn’t work and I got to remove it. Yay! Now I can use Mongrel instead of IIS…Yay! Well here’s the rub – turns out Mongrel wasn’t able to run as a service on 64 bit machine until one of the Mongrel guys figured it out about 6 weeks ago.

To get mongrel running you need to install the service with mongrel:service and but you have to include the FULL PATH to ruby.exe – then it works!

Ok great now I have some mongrels that run as service and startup when the server restarts. I decide to go with Pen for load balancing: easy and lightweight sounds great. So I try to install Pen as a service so I don’t have to login and run it everytime the windows server crashes and restarts (hopefully not ever but the gods are against me and I probably won’t be at this company to start the webservers up for them in the future).

No go with installing Pen as a service with the usual SrVany.exe route – maybe I just don’t know how to edit the registry correctly or setup SrVany on a 64 bit machine – but the usual route was giving me “service would not start promptly” error immediately when I press “start” under the services control panel.

Today I figured out a workaround – I used Scheduled Tasks to add a command prompt task every time the server starts. Then I changed the program from c:\system\someDir\cmd.exe under advanced options to a .bat file that contained my startup script: d:\pen\pen.bat

echo Start Pen for Mongrel Balancing
"d:\path\to\pen\pen.exe" -H 80 localhost:3000 localhost:3001 localhost:3002 localhost:3003
echo end of Batch file
pause

I tested it and it works!

Configuration Clusterfuck.

So after the “fiasco,” or rather, “disappointment” of signing over my software today all hell broke loose.

The company that shares office space with the company I’m contracting for moved out today and they wanted me here to make sure they didn’t take our server down.

That went smooth so I decided to disconnect all the unused ports from the empty offices so I could add two switches. We for some reason, have a 24 port switch for over 50 data ports in the office. So the <omitted> company that was moving had installed their own switches to add computers to offices. So I wanted to add the switches to the locked telco room instead of having them floating around under desks.

I disconnected several of the ports, careful not to take down any of my company’s offices. So then I come back to my linux machine and I can’t get internet. At the same time no one can print. And the fax machine ran out of ink.

So I could not figure out why I couldn’t get an IP address on any of the connections that had been unplugged and plugged back in. Turns out the DHCP server was on one of the servers that got moved. Awesome. So I fire up the <omitted> beast and get DHCP working after an hour of trying everything else.

Great, now I can get a connection, cool. I hook up an old fax and the old lady trying to use it keeps getting an error. I decide to ignore her, usually best to leave frustrated women the hell alone, and I focus on the printer. After another hour of tinkering turns out the printer can’t get an IP. I end up having to set it statically and no config of any other machines on the network was needed.

Ok, awesome, internet and printing, but no fax. I decide to fax my dad a test sheet. He’s not at his office at 7:00 fucking-o’clock on Friday but it went through without error. Old lady musta had her numbers wrong…I’ll deal with her on Monday.

What a Friday…t<omitted> deadline. I think I’ll get schwasted tonight.

Uninstalled BackTrack2, removed dual boot, back to Vista.

So my wireless card was supposed to work right off the bat with BackTrack so I could start breaking WEP128 encryption on wireless networks. But as it turns out BackTrack doesn’t recognize the wireless switch on the outside of my laptop.

So for the uninstall.

1. I didn’t want to just delete the partitions because I wanted to remove the bootloader LILO first.

2. I ran the command

lilo -u /dev/sda

at a terminal window in Linux.

3. It told me my disk was busy and it couldn’t be removed :(

4. A tutorial told me to boot up with the live cd and unmount all the drives and it should remove it

5. I mounted and unmounted the drives several times and ran the same command above on all the different partitions sda1-sda7 but it now tells me it can’t find LILO signature on any drive…so maybe the first command worked after all?

6. I rebooted and LILO was gone and Windows Vista started almost normally – it froze after the loading screen but I rebooted and I got to the login screen. YAY I didn’t fry my laptop.

7. Now to try either for Mac OS X install or maybe switch to