personal web space web hosting services: News about <b>hosting company</b><BR>

hosting company

Warning to all: 'latest' RHEL4 OpenVZ Kernel has a root exploit!

Wed, 11 Oct 2006 13:24:01 -0400

Background:
OpenVZ is a (stripped down) free, open-source version of Virtuozzo linux virtualization software. The modified OpenVZ kernel allows server operators to partition their servers into multiple Virtual Environments running a different Linux distribution.

Outdated Kernel:
As of now (Oct 10th, 2006) the latest Kernel listed on the RHEL4 download page (version 2.6.9-023stab016.2) is vulnerable to a root exploit that was first reported in July of 2006. That means that OpenVZ has had the vulnerable kernel available for download for around 3 months!

Response from OpenVZ: (*UPDATE*)
The response from OpenVZ was quick & effective - we contacted them at around 10PM on Oct 10th and by 6AM on October 11th (~ 8 hours) they released an updated version (2.6.9-023stab030.1). This does not negate the fact that a vulnerable kernel was left available for download for ~3 months, but I am quite pleased with their response.

update 2: OpenVZ sent an email to their list today (October 11th) at around 1PM EST saying "Everybody using 023 kernel is advised to upgrade." - perhaps they should have mentioned the root exploit in the email as a reason to drive people to upgrade.

Damage:
This only effected the OpenVZ kernels, not the Virtuozzo kernels. Our paid Virtuozzo installations were in the 2.6.8 branch which was not affected. A handful of our OpenVZ servers running 2.6.9 were vulnerable - we've updated them immediately. Unfortunately we became aware of this because one of the servers was actually exploited.

Server Security & Incident Tracking:
It goes without saying that if an attacker manages to get root access to a server, somewhere a sysadmin will forgo a night of sleep trying to recover.

'root' access to a server is absolute - root is the ultimate Unix user. Once an attacker gains root access, he/she can do anything. Cleaning a box that has had a root exploit is a nightmare, and many will argue not even possible. Because the 'root' user has the ability to modify anything on the system, any system binary can be replaced with a trojan'd version. Any configuration file can be changed to allow an attacker access through an unexpected port, ssh keys can be added to let an attacker in and cronjobs can be put in place to ensure that their exploits will stick around even if a sysadmin deletes them. An attacker can add a new user to /etc/passwd with uid '0' (root). The list goes on (and I don't want to give malicious people any more ideas!)

Having a malicious entity gain 'root' access to a server is a worst-case scenario for any system administrator.

How do you know if you were rooted?
There are many obvious signs:

log files disappear
suspicious processes are running on the server
programs with names like 'sendmail' are running on a non-standard port
files will be modified

Many system administrators will just know when something does not feel right.

What can you do?
Arguably the most important thing that must be done after an attack is finding the source of the exploit; what php script was exploited? what kernel bug was exploited? etc If you don't close the security hole, the hackers will just jump back in.

There are many ways that you can diagnose your system for changes and unusual activity:

Check the logs (assuming they weren't deleted)
use the unix 'find' command to search for files that have been modified or created in the last X days
use RPM --verify (if you are running an RPM-based distribution) to verify that binary files are not replaced malicious ones
Use 'netstat -apn' to look at incoming and outgoing sockets and inspect the output for unusual items.
hire someone who has experience in these situations

Most of the time attackers don't clean up after themselves - while they will delete the server logs to cover their tracks, they will leave behind the scripts that they use - these will be invaluable tools to discover how they exploited your system. Time stamps are also keys to finding out what was changed or added to your system.

"Why did the hacker choose me!"
This is a common question that we get from shared hosting customers who have vulnerable PHP scripts or forums. The answer is, these low-lifes have automated tools that search the internet for vulnerable scripts & forums - and then they notify the attacker of the vulnerabilities so that attacker can proceed.

Most of the time (especially in mass-defacing situations) attacker doesn't have a grudge against your personal website and they are not targeting your website for any reason other than it is vulnerable.

Most of the attackers that we have dealt with have 1 goal: replace all website files with their own political or religious messages.... and to gloat to their underground, hacker friends.

What is even worse is that you have websites with archives of hacks and records of what hacker defaced what website in the form of a competition - which hackers have defaced the most websites today? Websites shouldn't be encouraging hackers to increase their hack count!

Hacking in a Hosting Environment
In the context of a web hosting situation, there are 2 important types of exploits:
* 'localized' Exploits
* Server-Wide Exploits

An example of a "'localized' exploit" would be when a customer who is running an outdated PHP script gets attacked. The customer then gains access to the customers username and overwrites their files, can read their emails & confidential files, etc. For a web hosting company, this is expected and of 'minor' significance. For a customer, this may be the end of the world - files are gone, data is missing or modified and they feel victimized.

What scares system administrators is the server-wide exploits. This can be a direct attack (perhaps an SSH deamon has a vulnerability?) or this can be the result of an attacker who used a 'localized' exploit to escalate his/her privileges to 'root' level. A server-wide exploit is terrifying for web hosts. While web hosting companies will always tell customers that it is the customers responsibility to backup their files, the web hosting company has a job to do: keeping customer files online & accessible 24x7.

Backups
When the worst possible scenario becomes a reality, the web hosting company will usually turn to its backups. Backups come in many shapes and forms - local harddrives to store backups, remote backups and RAID (though that's not really a backup method... it's a redundancy method to protect against drive failure) are just 3 examples. Many hosts employ combinations of local & remote backups.

The problem is: If you store backups on a local server, an attacker can delete them. But, the cost of storing backups on a remote server is measured in additional administrative time & coordination, the cost of more bandwidth and the cost of the external storage space - this can add up to be an expensive proposition, especially if you are backing up to a remote datacenter at fast speeds - the bandwidth toll is expensive. In a web hosting environment, backing up dozens of servers with data retention spans of 1-3 months can require many TB of storage.

Another important decision is the backup schedule: will you backup everything each night or backup important things each night etc. Backing up an entire server each night would increase the CPU load, require much more storage and more bandwidth. Another option is backing up website files (the bulk of the data) once a month and everything else each night. This will help reduce the storage, bandwidth & CPU requirements, but the result will be that you may have to settle for a 1 month old backup if your files are removed.

The moral of the story:

Customers: Keep your scripts updated! Help provide a first line of defense for the server that your website is on. ALWAYS keep backups of your website data on your computer.

Web Hosting Companies: Keep your servers updated! Make sure that you update nightly & that you have good practices in place to help detect, quarantine and recover from an attack.

Upload your short video now and share your web hosting growing pains, funny dealings in running your online business or troubles in getting it off the ground. If your video is selected, you will win the $10,000 Grand Prize! Plus, your video will be used as online advertising for PEER 1. So what are you waiting for? Wow us with your creativity and originality.

The Peer 1 people sent this one my way to share with you, and I promise you it is well worth sharing. Remember the Growing Pains video they released a while back? They have now done a sequel that is just as good as the first.

In Case You've Read Otherwise, SmugMug Still Loves S3

Mon, 12 Mar 2007 17:01:00 -0400

Last Thursday night, I came across this SearchStorage.com article via the Storagezilla blog. Beth Pariseau wrote that Amazon's Simple Storage Service (S3) has had "performance and reliability issues serious enough" to prompt second thoughts among early adopters. In particular, SmugMug CEO Don MacAskill recently decided to move hot storage back in-house.

The instant I finishing reading the article, my RSS reader lit up with Don's response. He still loves Amazon, even if S3 hasn't solved the "speed of light problem". It takes at least 60-80ms for bytes of data to travel the distance between SmugMug's west coast location and Amazon's east coast data center. There's no getting around that. He moved hot storage closer to his web servers NOT to solve Amazon's performance problems, but to reduce those thousands of miles to inches. Don also tracked down the Storagezilla post and added a comment there.

Fast forward to this morning, when someone sent me a snippet from a Tier 1 Research news brief in which Dan Golding wrote about Amazon's disillusioned users. I gave Dan a hard time for basing his article on the same two customers Beth interviewed without giving her credit. Dan argued that attribution isn't customary in the analyst world. Besides, we shouldn't even be having this conversation. As a non-subscriber, I should have deleted any T1R content that came my way upon receipt.

Ironically, during his HostingCon presentation last year, T1R founder Andy Schoepfer's key message was "don't be an island". It's important for web hosting providers to connect customers to external ecosystems like eBay and Amazon, because no e-business can thrive in isolation. Given T1R's Hosting 2.0 advocacy, Dan's reaction seemed... Analyst 1.0-ish. But towards the end of our conversation, he did promise that an upgrade is on the way. As a point of reference, Burton Group, Dan's former employer, has a great blog that links to external sources. Same goes for Forrester. And at least 220 other research firms, including T1R parent company The 451 Group. Raven Zachary, who leads 451's open source practice, is even on Twitter!

Anyway, I'll get off my soapbox now and back to Amazon. I think every web hosting exec needs to read Don's blog post - along with Robert Cichon's post on customer satisfaction metrics. Robert said a hosting provider has done a good job if (a) the company gets written testimonials, (b) customers refer other customers because they're happy with service quality, and (c) customers defend the company against negative remarks. Amazon gets three points based on Don's reaction. What's your score?

Accounts Payable Associate

Wed, 28 May 2008 16:13:25 EST

Responsibilities:
* Maintain accounts payable records
* Reviewing invoices for accuracy
* Routing invoices to proper personnel for approval for payment
* Placing calls to vendors or other departments regarding any questions related to invoice payments
* Ensuring invoices are posted to the proper department and account
* Complete check printing and prepare check packets for signature and mailing
* Research vendor statements
* Auditing and processing employee expense reimbursement
* Assist with accounts payable month-end close process
* Reconcile credit card charge accounts
* Prepare weekly cash requirement downloads
* Answer all vendor inquiries
* Inform management of payment issues
* Handle incoming check requests
* Create and maintain vendor files
* Record manual checks
* Perform filing and copying
* Other duties as assigned

Job Requirements:

Required Skills and Talents:
* A minimum of 2 years accounts payable experience
* Must be computer literate and proficient in MS Excel
* Excellent verbal and written skills
* Exceptional organizational skills
* HSD diploma w/associates degree preferred
* Ability to thrive in a challenging, demanding and rapidly changing environment
* Experience in Soloman a plus