Archive for the 'Uncategorized' Category
How to Fight a Joe Job Spam Attack
After receiving spam complaints and lawsuit threats at regular intervals for over a year and after many arduous hours of research, I finally found some methods to help combat what has become known as a “JoeJob” spam attack.
If you are the victim of a “Joe Job Spam Attack”, let me save you some time and tell you how to configure a few items to help thwart the spammers:
Before we setup the SPF record, let’s modify the default email (catch-all) address to :fail: or discard. Log in to cpanel for the domain and click on ‘Default Address’, configure it so that it looks like the following (substitute your domain and put whatever message you want):
Now let’s setup SPF.
First, go to the SPF setup wizard (open site in new window)
Then, Connect via SSH and perform the following steps manually:
> pico /etc/named.conf
*if pico doesn’t work, try edit
Find the line containing your domain and note the location of the zone file. Edit the zone file and add a line such as (last line in screenshot above):
domain.tld. 14400 IN SPF “v=spf1 a mx -all”
*Notice the period after the domain and tld. Here’s a screenshot of what my record looks like after adding the TXT record:
Command to check SPF record from SSH:
> dig -t TXT domain.tld +short
If you don’t have SSH or if you feel more comfortable using WHM/cPanel, here’s how it should go:
Log into WHM, click on ‘Edit DNS Zone’ and choose your zone. Click ‘Edit’
You should add an entry that looks like this:
Restart BIND and you should be all set!
Restart BIND via SSH with:
> service named restart
Alternatively, you can use WHM’s restart BIND option.
I sure hope this helps you and me. I just configured this tonight, but from what I’ve read in my research, I am very hopeful. I guess we will see once the spammer launch their next attack. Or, if it works really well, the spammers will have a hell of a headache on their hands when they are discovered and I guess I won’t know after all. Lol.
Diplomacy was never my strong suit
Although I can be very diplomatic and cordial on behalf of my clients in their problems, I tend to take spammers and hackers personally –especially when they are costing me money.
Here’s the transcript of my incident with ThePlanet today. They ended up wasting hours of my time today because of some stupid spammer (see related story). Maybe someone will get a kick out of this. 
*** Names have been changed to protect the innocent ***
XXXXXX - Tuesday December 18th, 2007; 12:09 AM CST
Dear Customer,
The Planet or its upstream providers has received a complaint related to a probable violation of the Acceptable Use Policy (AUP). We are forwarding the report, requesting that you take appropriate measures to address the issue.
It is very important that you take action on this matter and respond to this ticket within the deadline specified. Failures to investigate, address the issue, and update this ticket with the root cause and actions taken to resolve the problem may result in service interruption. Consider this your only notification. If there is no productive reply, or if the abuse does not cease, Policy Enforcement will be forced to interrupt and/or terminate your service to protect the integrity of the network.
For general reference regarding The Planet’s stance on abuse, refer to:
http://www.theplanet.com/about_us/legal.asp
Please direct any questions regarding this specific issue directly in this ticket, or open a new ticket if you are unable to update this ticket due to being logged in on a different user account.
The server in question has been identified as the origin of spam messages. This issue needs to be addressed quickly to prevent blacklisting, for which fees may be assessed for removal.
Please investigate, resolve the issue, and update this ticket with your actions, referencing the attached reports for details. Failure to perform these actions within 24 hours will result in service interruption. Thank you in advance for your time, efforts, and cooperation.
Regards,
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.
Hi, This is an on-going issue with bitcorp.net. Please view the headers and see that the spam did not originate from our server or domain.
Thank you.
XXXXXX - Tuesday December 18th, 2007; 12:44 AM CST
Dear Customer,
Thank you for your response. Per the headers below, you will see that the emails are originating from 209.62.57.163. We do show that bitcorp.net is 209.62.57.170, which is also tied to your account. We have attached an additional sample complaint, for your convenience. Please keep us updated.
Received: from 209.62.57.163 (HELO bitcorp.net)
by 59diner.com with esmtp (MTPUVMIMNJY NWHDOX)
Regards,
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.
Hmm… I was looking at the x-originating IP.
I do see it says it was received from our main shared IP after closer inspection.
I see a pop3 attack on the server and have banned the primary offending IP. However, I do not see how anyone would be sending spam off this machine –especially not from the bitcorp domain (i.e. no configured email or MX records).
I tried:
> tail /var/log/messages - nothing special
I see email attempts failing in the messages log, so it appears that someone may be spamming off the server and spoofing the addresses/IPs, but how can I tell who is the culprit?
I then tried:
> tail /var/log/maillog
I saw a lot of pop3 failed logins, so I added the offending IP to iptables to DROP
I then tried:
> tail /var/log/exim_mainlog
I see the following but I cannot determine who is sending the emails. Any help would be appreciated.
2007-12-18 02:34:31 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] Warning: Sender rate 281.2 / 1h
2007-12-18 02:34:31 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] Warning: Sender rate 282.2 / 1h
2007-12-18 02:34:31 H=webshieldin02.smed.net (webshieldin01.smed.net) [199.21.28.117] F=<> rejected RCPT <jraleigh@bitcorp.net>: webshieldin02.smed.net (webshieldin01.smed.net) [199.21.28.117] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] F=<> rejected RCPT <jraleigh@bitcorp.net>: aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=mail01.ikarisuper.co.jp (MAIL01) [219.122.47.218] F=<> rejected RCPT <jraleigh@bitcorp.net>: mail01.ikarisuper.co.jp (MAIL01) [219.122.47.218] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=mozart.eds-progical.fr (proxy.edspro.fr) [212.155.167.50] Warning: Sender rate 5.5 / 1h
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] Warning: Sender rate 283.1 / 1h
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] F=<> rejected RCPT <jraleigh@bitcorp.net>: aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] F=<> rejected RCPT <jraleigh@bitcorp.net>: aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:33 H=mozart.eds-progical.fr (proxy.edspro.fr) [212.155.167.50] F=<> rejected RCPT <jraleigh@bitcorp.net>: mozart.eds-progical.fr (proxy.edspro.fr) [212.155.167.50] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
After further inspection, the messages showing in exim_mainlog appear to be bounced emails trying to deliver to the bitcorp emails but are failing because there is no route -since MX is invalid.
Is it possible that the spoofed emails are bouncing back here and that is what we’re seeing? If not, I need a little more help in figuring out which account is sending the email.
Please keep in mind that this has been an off and on again issue for months. In each case, there was no evidence of any spam actually originating from our server.
-Me
XXXXXX - Tuesday December 18th, 2007; 1:50 AM CST
Dear Customer,
We have received your request for assistance in tracking down this abuse matter. We can suggest the following:
A) Please search the forums at theplanet.com for help with your abuse issue.
B) Consider searching google or an alternative search engine
C) Consider hiring a third party administrator
Please note that as the system administrator, it is expected that you investigate and resolve all abusive issues. Should you require further assistance, please contact our professional services department.
–
Regards
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.
This is the last thing I need right now.
I haven’t used any administrative hours in quite a while. Why don’t you use my administrative hours and get professional services to look into this?
Believe me, I have searched the forum, requested help, and search all kinds of search engines. It seems that there are no qualified persons who know the answer to “How do I know which user account is sending spam?” It is the ultimate question apparently.
XXXXXX - Tuesday December 18th, 2007; 2:01 AM CST
Dear Customer,
Thank you for your response. If you would like professional services to investigate this issue with your administrative time, you will need to open a new trouble ticket requesting that they do so. Please keep us updated.
Regards,
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.
This spammer has been spoofing emails with our domain for over a year and I am inclined to believe I am wasting my last two hours troubleshooting this problem -as I feel it is still the same issue.
I see no outgoing emails and I see in the logs only incoming emails and “refused: too many connections” from bounces trying to be delivered.
I am not 100% convinced that these emails are originating from our server. However, exim is having a lot of trouble dealing with the influx of spam bounces and I cannot fix that. Here is a related article that sounds exactly like what I’ve been experiencing and what I currently see in my exim logs:
http://www.webhostingtalk.com/archive/index.php/t-347496.html
After nearly 3 hours of investigation, I am convinced the headers were spoofed and you have once again wasted my time.
Shall I send YOU an invoice for $150/hour?
I decided to begin accepting email bounces for jraleigh@bitcorp.net and see what the actual messages really were. I will paste it below, and you can see that the IP address from which the email was received (in all cases) by the bouncer was NOT my server IP address. The fact that the email trail shows my server IP only indicates the high probability that the IP was spoofed. This is further corroborated by the fact that my exim mail queues are clean and we have no outbound connections. I suggest you investigate server at IP: 66.101.198.234. Please do not open any new tickets about spam on this server unless you are able to prove it to me.
Return-Path: <jraleigh@bitcorp.net>
Received: from ns1.hostsedona.com ([66.101.198.234])
by 66-101-196-128.accesssedona.net (8.11.6/8.11.6) with ESMTP id lBEFbB829195
for <president@phonesexcoalition.org>; Fri, 14 Dec 2007 08:37:11 -0700
Received: from co.clark.nv.us (dsl88-247-12379.ttnet.net.tr [88.247.48.91] (may be forged))
by [66.101.198.236] (8.11.6/8.11.6) with SMTP id lBEFb6X27532
for <prez@phonesexcoalition.com>; Fri, 14 Dec 2007 08:37:08 -0700
Received: from 209.62.57.163 (HELO bitcorp.net)
by phonesexcoalition.com with esmtp (TBWJNXUQOE CCDQWA)
id HMbClH-7iz5ZC-F7
for prez@phonesexcoalition.com; Fri, 14 Dec 2007 17:37:06 +0200
Message-ID: <114701c83e67$2e793c70$c0a80202@Marva>
From: “Marva T. Blackman”<Marva@bitcorp.net>
To: “Jerry N. Denny”<prez@phonesexcoalition.com>
Subject: Promote your little soldier of love in a new year!
Date: Fri, 14 Dec 2007 17:37:06 +0200
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary=”—-=_NextPart_4421_11AF_01C83E77.F2020C70″
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1441
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441
This is a multi-part message in MIME format.
YYYYYY - Tuesday December 18th, 2007; 2:38 PM CST
Dear Customer,
This E-Mail did in fact originate from your server. This assessment is based on the following information in the mail headers:
Received: from 209.62.57.163 (HELO bitcorp.net) by phonesexcoalition.com with esmtp (TBWJNXUQOE CCDQWA) id HMbClH-7iz5ZC-F7 for prez@phonesexcoalition.com; Fri, 14 Dec 2007 17:37:06 +0200 Message-ID: <114701c83e67$2e793c70$c0a80202@Marva>
This line was generated by the recipient mail server (in this case, phonesexcoalition.com [66.101.198.236]). As such, it is not possible that this information is spoofed, as it is not generated until after the message has left the originating server.
Please keep us updated with your progress in this matter.
Regards,
YYYYYY
Abuse Team
The Planet
abuse@theplanet.com
Sorry but I am not convinced. I see no outgoing connections on our server and the spam bounces are getting worse -indicating an on going spam effort.
I have exhausted all of my options and theplanet is obviously unwilling to help without charging me astronomical prices -in spite of the fact that I am the victim here.
What exactly do you propose we do?
By the way, I suspended the bitcorp.net domain along with numerous others, and the spam continued to pour in and increase. Doesn’t that tell you anything?
By the way, phonesexcoalition.com doesn’t strike me as being a 100% reputable domain. I don’t think I’m going to trust a header possibly forged by THAT domain over my gut feeling. My gut could be wrong, but no one can show me definitively that my server is sending spam by looking at server logs, exim queues, or anything else and I can tell from those same logs and queues that the server isn’t doing much besides accepting a million bounced emails.
Update: I got a response from them (12.19.2007)
ZZZZZZ - Wednesday December 19th, 2007; 6:02 PM CST
From the headers that were attached, there is indeed good evidence to support the possibility that your server, and specifically the domain “bitcorp.net”, has fallen victim of a joejob attack in which the headers have been forged to make the appearance that your server was responsible for being the original sender of the message.
Unfortunately, options to combat this type of abuse are somewhat limited. I would suggest having the administrator of bitcorp.net set up a strict SPF record to specify the server as the only authorized sender of email on behalf of the domain.
I would also change the option to “Send all unrouted e-mail for:” from “:blackhole:” to “Discard with error to sender (at SMTP time)”.
Our abuse team will be notified of my findings.
Please let us know if any additional assistance is needed. Thank you!
Update: Having this issue?
Read how to combat a Joe Job Spam Attack.
After that support incident chain; the spam bounces have stopped coming in. If this were spam originating from my server, it would not stop without some sort of intervention.
In any case, here’s a couple of things to note:
- Professional services they recommend are $150/hour
- ThePlanet is not as bad as this incident might make them seem
Generally speaking, I have been VERY pleased with the support from ThePlanet. They are quick to respond and very knowledgeable. In many cases, they work quickly and give detailed explanations of how they fixed things. It’s just these cases of “abuse” where the support seems to break down. To their defense, these spammers are very good at what they do, and it is understandable how -at first glance- these emails can point to my server being the culprit. However, I have a repeated history of proving my points and my opinions should carry a little more weight. That department is really quick to point a finger and let you resolve it. They should spend more time investigating before they jump the gun. That’s all.
p.s. At least they are WORLDS BEYOND Hostgator -where I originally started hosting all of my sites. Man, don’t get me started. I think that might have been some of the worst couple of years of my life, and I’ll spare you the miles of support tickets from them. Something like:
Me - “All of my sites are down!!”
Hostgator (2 hours later) - “Who are you? What is your account?”
Me - “Oh my God! My account number is in the original ticket”
Hostgator (6 hours later) - “Ok. I can reboot your server, but I need your password to verify your account”
That’s only a “slight” exaggeration of the kinds of support they gave me! I just don’t want to sift through my old emails to get the wording right. That’s basically how it went. Whew. I’m glad those days are over! lol.
Ah, the Woes of an Online Business
Well, I have ended up mostly wasting away another day because of spammers. Once every month or two, I have to spend a couple of days dealing with spammers or hackers. I’ve had to deal with them for so many years, that I’ve become efficient at it. However, the anger I feel towards these “faceless” persons and the money I lose grows each time I have to deal with them.
Today, I was dealing with someone (indirectly) who has been spamming millions of emails with various lewd offers and they spoofed (i.e. faked) that it came from one of my domains (bitcorp.net). However, I purposely disabled email (and routed the MX record to a non-existant domain) just to ensure that spamming off of that domain would never be possible. The domain itself is merely a placeholder with a static HTML page -no scripts and no formmail available. There is virtually no way to spam from that account.
Every couple of months, this particular spammer collects all of the servers his script has hacked and will send out a blast containing millions upon millions of email addresses. Naturally, all of the bounce messages, complaints, and legal threats come directly to me –since it appears that my domain sent them. How nice of the spammer, eh?
It is usually only a minor inconvenience because I’ve gotten good at keeping server load low by ignoring the bounces and I have been quick to prove that the emails were spoofed by showing that the IP address from which they originated was not my server.
Well, spammers aren’t stupid. They’re just like you and me and the professionals have a lot of experience, knowledge, and powerful tools at their disposal. Typically, these kind of spam rings are run by mafia-like organizations that have a lot of money –generated by the crap they peddle. This spammer has evolved his script a bit and now spoofs even the correct IP address in many of his emails. So, today, I had to spend 3 hours to do a tracelog and backtrack exactly where the email came from. It took me all that time just to prove I was innocent. Oh well, I guess it’s old hat by now. I’m used to proving my innocence (see my bio).
At any rate, this cost me a lot of time and money. I really despise these people and would love to get my hands on any one of them -even if only for a few minutes. It’s all I would need to make them think twice about sending more spam. I suppose the irony of the situation is that their profession is a constant battle and wrought with just as many headaches in their never-ending struggle to hack more servers to send their junk as it is for people like me to block it.
Anyway, the moral of the story -especially for beginners in online businesses- is to find a reputable hosting company and get a shared account or get a dedicated server that is managed, if you have the money. That way, you won’t have to deal with the headaches I have to deal with. If only I had the income I used to have, I would upgrade my servers to fully managed servers over at RackSpace (see review) instead of the lower-price but self-managed servers at ThePlanet (see review).
Off to a good start…
I will write a little bit today after all. We just checked the stats for Zakioo.com, our new online video game store, and even though it has not yet been launched; we are seeing more than 20 visitors per day from Google and a growing trickle from Yahoo and MSN.
This is a good sign. The META tags and content are doing their job -that’s with no backlinks (except from whois.sc -given automatically to every registered website) and no marketing. That’s not bad. We should launch in a few days and finally monetize that incoming traffic. Then, we’ll update on the success of that versus the affiliate traffic (probably a 1-month delay in reporting).
Whirlwind Workday
No new theories today and no time to check the latest search engine news.
I’ve been under-the-gun to get a problem resolved with the supplier of video games for our new online video game store, Zakioo.com (visit site). We cannot launch the store until that’s done, so it’s been a crazy, hectic day between that, a proposal I finished, and some web design work I’m working on.
For that store, though, I should mention that we are using affiliates alone to promote the store. That, plus the comprehensive selection and text should drive decent amounts of buying traffic. I will update here with results as they come in. We already launched one store in Hungary as a BETA test and it is working great -i.e. ten (10) sales per day average after only two (2) months operational. That’s pretty darn good with no other outside marketing efforts –content and affiliates only.
Alas, tomorrow is another day. 
Well, it’s taken me quite awhile to get to the point of being able to get this site up-and-running, and I have a LOT of ideas, theories, and notes to share. So, here’s to post #1… There’s lots more to come, and I will be making a lot of catch up posts soon!