Posts Tagged ‘administration’

Processing large data volumes


2009
12.15

So a compressed text file that ends up being 72GB sounds like a lot, right? Especially if you have to data-churn it with per-line processing. Fine. Wow. A lot.

That pales in comparison to the (reported) volumes of data processed by Google and Facebook:

In December 2007 (!) Google was processing 400 PB (petabytes) per month, with an average job size of 180GB.

Facebook’s volumes have been steadily increasing, too: From March 2008′s 200GB of daily new data, they’ve moved up to 2TB per day in April 2009, to steady off to 4TB per day in October 2009.

Most of which are, without a doubt, LolCat pictures :)

nginx on Windows Vista / Windows 7 with php


2009
11.22

You can get the latest stable version of a Windows compile of nginx (0.7.64) here.

Setup is quite straight forward; you’ll need the php-cgi.exe downloaded (copy the libmysql*.dll to the Windows system directory – quickest but dirtiest solution) and tied in; I assume that you can download and run the mysql application yourself?

You may need to run the startup as a batch file (f: below is the drive installed on, php has been extracted into the php under nginx as you can see). PHP is set to listen on port 9000, which nginx interfaces requests to below. (more…)

Zombie postings with Captcha automation


2009
11.14

Just got another comment posting request relating to a version 5.0 release of automated forum/blog spamming software which is guaranteed to get your customers ‘closer to your products’ by improving product and site visibility. It can also do in-forum PMs ‘for a more personal touch’ — ie violate any terms and conditions of a user environment to spam the web for cheap cross-links (at $540 for the app — including automatic Captcha recognition). (more…)

And we’re on 2.8.6


2009
11.13

Another quick upgrade to make it safe — 2.8.6 fixes two security problems that can be exploited by registered, logged in users who have posting privileges. Not really pertinent here but good to keep it closed :) The patching deals mainly with untrusted user issues, but “upgrading to 2.8.6 is recommended”.

But you knew that already :)

Create a new certificate for Ubuntu the easy way (but you knew that already)


2009
06.14

For the self-signed certificate:

sudo a2enmod ssl

wget http://librarian.launchpad.net/7477840/apache2-ssl.tar.gz

tar -zxvf apache2-ssl.tar.gz

sudo cp apache2-ssl-certificate /usr/sbin/

sudo cp ssleay.cnf /usr/share/apache2/

sudo mkdir /etc/apache2/ssl

cd /etc/apache2/ssl

sudo apache2-ssl-certificate -days 365

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/ssl

sudo ln -s /etc/apache2/sites-available/ssl /etc/apache2/sites-enabled/ssl

Modify the ssl site-available:

NameVirtualHost *:443

<virtualhost *:443>

ServerAdmin webmaster@localhost

SSLEngine On

# the following lines are links to the certs

SSLCertificateFile /etc/apache2/ssl/apache.pem

DocumentRoot /var/www/

Everything else stays the same.

Now all that remains is to restart your server:

sudo /etc/init.d/apache2 restart

All done!

Found this online :)

Copying files from one server directory to another (without sapping resources)


2009
06.03

cp just saps resources when transferring large volumes of data between directories. Especially on old machine memory usage spikes when the process is started and the machine becomes almost unresponsive. That’s on large volumes of data (and old hardware).

Enter rsync stage left. OK, none of this is new (you know all this already) — it’s really just to have in one spot:

No mess no fuss transfer from one server to the other (straight transfer via SSH):
rsync -av -e ssh username@oldhost:/home/old_home/ /home/new_home/

Synchronising your local to a remote directory:
rsync -r -a -v -e "ssh -l username" --delete webhost:/var/www /local/var/www

Synchronizing your remote directory to the (current) local directory:
rsync -r -a -v -e "ssh -l username" --delete /local/var/www webhost:/var/www

-r recurses, -z compresses, -v is verbose and -e is obvious

With large volumes of data (DVD isos etc) your processor is still going to be taxed (twice) — via ssh and rsync – but not quite as intensively as on the cp.

To copy files between two local directories, rsync does the job, too:
rsync -vur --delete --exclude=*.EXT1--exclude=*.EXT2 /folder1 /folder2

Quick, simple, gracious.

But you knew that already… :)

Shrinking LDF files (SQL Server 2000 etc)


2008
12.17

Some really nifty tips on huuuuuge LDF files I encountered (56G a pop) and came down to 212MB as a result of this process:

In SQL Server 2000 Query Analyzer run the following command (doesn’t matter what db you are in):
backup log [database name] with truncate_only
(don’t include the square brackets)

Open Enterprise Manager and navigate to your database:
–Right click on the database
–Highlight ‘All Tasks’
–Choose ‘Shrink Database…’
–On the bottom of the window on the right, click the ‘Files’ button
–In the drop down at the top make sure you choose the ‘_Log’ option. By default it is the ‘_Data’ option
–Leave the shrink action as: ‘Compress……’
–Click ‘OK’

More verbose information at these links (though just expounding on the above snippet)…: