Posts Tagged ‘server’

Processing large data volumes


2009
12.15

So a compressed text file that ends up being 72GB sounds like a lot, right? Especially if you have to data-churn it with per-line processing. Fine. Wow. A lot.

That pales in comparison to the (reported) volumes of data processed by Google and Facebook:

In December 2007 (!) Google was processing 400 PB (petabytes) per month, with an average job size of 180GB.

Facebook’s volumes have been steadily increasing, too: From March 2008′s 200GB of daily new data, they’ve moved up to 2TB per day in April 2009, to steady off to 4TB per day in October 2009.

Most of which are, without a doubt, LolCat pictures :)

MTN Business goes down – Network Outage over


2009
12.02

Seems as though systems are being restarted / links restored, 1hr 10 min later… Whole of South Africa was affected by the network outage.

Running commentary here.

To quote the WebAfrica forum “Is this “network issue” anything to do with the power failure (and generator backup failure) that occurred at MTN moments ago bringing down the entire Hetzner hosting facilities?”

mybroadband.co.za is still struggling to come back online, and has been taken offline again…

Update:  “According to MTN Business’ Edwin Thompson an Eskom power failure at 14:30 meant that the data centre relied on generator power this afternoon.  Unfortunately the main generator ‘blew itself up’ which impacted the electricity infrastructure at the facility.  While the power and generator system for the facility is fully redundant, Thompson explained that this it is not really possible to prepare for these types of unexpected events.

“The Gallo Manor data centre is a key hosting facility in South Africa with numerous large clients like iBurst and Hetzner.  The downtime therefore does not only affect websites like MyBroadband, but also hosted services like authentication & billing for iBurst and hosted PBX solutions.” – mybroadband.co.za is back up…

MTN confirmed that it is experiencing network problems.  “MTN is experiencing network problems in Pretoria and surrounding areas, which are impacting on all services. We are currently investigating the matter and we will shortly confirm how soon the problem is expected to be resolved,” said Sameer Dave CTO of MTN SA.

Cloud Computing 2009, The Forum, Bryanston


2009
11.24

With the usual eats (ok, the brownies are good – chewey, but not gooey) welcoming those who chose to attend, the presentations at the Cloud Computing Conference 2009 promise three potentially interesting presentations: a case study by iBurst, one by the University of the Witwatersrand, and a presentation about the potential security risks that cloud computing inherently presents. (more…)

Create a new certificate for Ubuntu the easy way (but you knew that already)


2009
06.14

For the self-signed certificate:

sudo a2enmod ssl

wget http://librarian.launchpad.net/7477840/apache2-ssl.tar.gz

tar -zxvf apache2-ssl.tar.gz

sudo cp apache2-ssl-certificate /usr/sbin/

sudo cp ssleay.cnf /usr/share/apache2/

sudo mkdir /etc/apache2/ssl

cd /etc/apache2/ssl

sudo apache2-ssl-certificate -days 365

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/ssl

sudo ln -s /etc/apache2/sites-available/ssl /etc/apache2/sites-enabled/ssl

Modify the ssl site-available:

NameVirtualHost *:443

<virtualhost *:443>

ServerAdmin webmaster@localhost

SSLEngine On

# the following lines are links to the certs

SSLCertificateFile /etc/apache2/ssl/apache.pem

DocumentRoot /var/www/

Everything else stays the same.

Now all that remains is to restart your server:

sudo /etc/init.d/apache2 restart

All done!

Found this online :)

Copying files from one server directory to another (without sapping resources)


2009
06.03

cp just saps resources when transferring large volumes of data between directories. Especially on old machine memory usage spikes when the process is started and the machine becomes almost unresponsive. That’s on large volumes of data (and old hardware).

Enter rsync stage left. OK, none of this is new (you know all this already) — it’s really just to have in one spot:

No mess no fuss transfer from one server to the other (straight transfer via SSH):
rsync -av -e ssh username@oldhost:/home/old_home/ /home/new_home/

Synchronising your local to a remote directory:
rsync -r -a -v -e "ssh -l username" --delete webhost:/var/www /local/var/www

Synchronizing your remote directory to the (current) local directory:
rsync -r -a -v -e "ssh -l username" --delete /local/var/www webhost:/var/www

-r recurses, -z compresses, -v is verbose and -e is obvious

With large volumes of data (DVD isos etc) your processor is still going to be taxed (twice) — via ssh and rsync – but not quite as intensively as on the cp.

To copy files between two local directories, rsync does the job, too:
rsync -vur --delete --exclude=*.EXT1--exclude=*.EXT2 /folder1 /folder2

Quick, simple, gracious.

But you knew that already… :)

Worldwide DNS patch roll-out (and the slow, lagging one)…


2008
07.13

So the DNS patch rolled out on 8 July. But you knew that already. Vendor after vendor started popping out the woodwork, and Ubuntu mirrors slowly started updating worldwide to rollout updates to bind9, the glibc stub resolver etc… The original CERT announcement didn’t had many ‘no response’s from a wide variety of providers (having been alerted only a few days earlier)… When asked how the issue was addressed in his application, dnsmasq’s author Simon Kelly, for example, had this reaction:

Good question.
I wasn't contacted in advance about this, and no patch for dnsmasq has
been released. Since the exact nature of the new vulnerability has not
(as far as I know) been announced, I don't know if dnsmasq is vulnerable.
My current plan is to implement query-port randomization, and I'm
working on that right now. If all goes well, it will go into 2.43, and
be released ASAP. To help with this, I'd like to gather as many testers
as possible. The changes are quite intrusive, and to be confident about
releasing them quickly, I'd like to get as many people as I can testing.
Since query-port randomisation is potentially quite resource-heavy (it
needs a socket per query), and will break many firewall configs, the
current plan is to make it optional, and not the default behaviour.
Cheers,
Simon.

Microsoft also came to the party with MS08-37, their Windows patch of the problem. However, other vendors are going nuts now, having to issue repatched versions (think ZoneAlarm – ok, that is, if you’re using it) of their own proprietary software. Depending on predictable port? Mal-implemented patch(es) by the upstream provider?

The fix (not the attack vectors) are described as such, eg in DSA-1603-1:

“Dan Kaminsky discovered that properties inherent to the DNS protocol
lead to practical DNS cache poisoning attacks. Among other things,
successful attacks can lead to misdirected web traffic and email
rerouting.

This update changes Debian’s BIND 9 packages to implement the
recommended countermeasure: UDP query source port randomization. This
change increases the size of the space from which an attacker has to
guess values in a backwards-compatible fashion and makes successful
attacks significantly more difficult.” – Ubuntueque

Dan Kaminsky went on at length about the need and success of CERT in the roll-out of the patches across vendors. In South Africa, DNS went down to a crawl as SAIX, Verizon, IS etc patched their servers (OK, Verizon also switched its ATM link to SAIX to GigEthernet, so after veeeery slow, thing started to fly…) SAIX’s DNS servers took 2 days after release to go live and randomise and not publish the port.

Bind9, dnsmasq all are “affected” by this…

The problem with all this is the lack of information and uncertainty. OK, argue that this was in the interest of security. Don’t make the weakness know to protect the web. But *no-one* knew what the technical details were (and verify the patch, peer-reviewing the process) until he privately finally disclosed to Ptacek and Zovi… who agreed that all was above board

I guess that leaves three concurrent steps to take:

  1. Wait for BlackHat 2008
  2. Patch whatever servers are out there in anticipation
  3. Trust.

Kaminsky put it nicely in his blog:

So here’s the bottom line.  I think people don’t have enough information right now, to determine whether there indeed exists any context in which a huge press rush should occur with so few deep technical details.  When everything is on the table, I leave it to the community to judge whether we have gained or lost credibility through this effort.

But it’s clear that, in lieu of details, to not even have respected and completely independent members of the community vouching for your work cannot stand, no matter how respected you are in the community, no matter how many vendors are behind you, no matter what.  OK.  So that’s a fairly big lesson learned, in a process I’ve sort of been making up as I’ve gone along.  Thanks to Dino and Thomas for setting me straight.

Let’s see where this takes us…

Single-line key-less SSH remote login (but you already knew that)


2008
07.11
ssh-keygen -t dsa
cat ~/.ssh/id_dsa.pub | ssh youruser@example.com 'cat - >> ~/.ssh/authorized_keys2'

Choose no password. And you’re done…

Routing – but you knew that already


2008
07.09

From the RTFM

In this example, your client machine is connected to a firewalled LAN through ethernet device eth0. Its IP address is 12.34.56.78; its network is 12.34.56.0/24; its router is 12.34.56.1.

Your network administrator may have told you to use 12.34.56.1 as default router, but you shouldn’t. You should only use it as a route to the client side of the firewall.

Let’s suppose the client side of your firewall is made of networks 12.34.0.0/16 and 12.13.0.0/16, and of host 11.22.33.44. To make them accessible through your client router, add these routes to your global network startup script:

route add -net 12.34.0.0 netmask 255.255.0.0 gw 12.34.56.1
route add -net 12.13.0.0 netmask 255.255.0.0 gw 12.34.56.1
route add -host 11.22.33.44 gw 12.34.56.1

You must also keep the route to the client’s local network, necessary for linux kernel 2.0 and earlier, but but unnecessary for linux kernel 2.2 and later (that implicitly adds it during the ifconfig):

route add -net 12.34.56.0 netmask 255.255.255.0 dev eth0

On the other hand, you must remove any default route from your scripts. Delete or comment away a line like:

route add default gw 12.34.56.1

Note that it is also possible to remove the route from the running kernel configuration without rebooting, by the following command:

route del default gw 12.34.56.1

Just so that it’s all in one place :)

Treason uncloaked


2008
05.21

This kind of error message has been popping up in some of the servers’ log files (mainly in the US):

1
 X Time(s): TCP: Treason uncloaked! Peer 195.166.XXX.YYY:18549/80 shrinks window 94463210:94463211. Repaired.

Doing any kind of google.com search (or yahoo.com for that matter) doesn’t give much of a hint as to what that’s all about.

There’s a mix of opinions.

RedHat Magazine puts this down to an “informative message. It is not a problem with the system.” which is rightly chastised by the comments following it.

Other scenarios are provided, such as that it’s a low-level DoS attack (to simply keep sockets open that will use up resouces)

This is backed up by the comment (hurray for comments!!) in the tcp_timer.c code in the ipv4 net code:

/* Receiver dastardly shrinks window. Our retransmits
* become zero probes, but we should not timeout this
* connection. If the socket is an orphan, time it out,
* we cannot allow such beasts to hang infinitely.
*/

So, it’s highly likely that it’s a tar-pit somewhere that’s trying to eek out resources.

A blunt approach would

1
iptables -A INPUT -s 195.166.XXX.YYY -j DROP

quite a definite defense mechanism (should they not be spoofing, which they most probably are…) — in this case, they claim to be from Nicaragua…

Two alternative explanations present themselves:

  • a seriously old TCP receiver stack OR
  • a mobile device that purposefully starts with small windows to start off with…

Not highly likely…

Will keep monitoring :)