12.18.07

Diplomacy was never my strong suit

Author: puravida

Although I can be very diplomatic and cordial on behalf of my clients in their problems, I tend to take spammers and hackers personally –especially when they are costing me money.

Here’s the transcript of my incident with ThePlanet today. They ended up wasting hours of my time today because of some stupid spammer (see related story). Maybe someone will get a kick out of this. ;)

*** Names have been changed to protect the innocent ***

XXXXXX - Tuesday December 18th, 2007; 12:09 AM CST

Dear Customer,

The Planet or its upstream providers has received a complaint related to a probable violation of the Acceptable Use Policy (AUP). We are forwarding the report, requesting that you take appropriate measures to address the issue.

It is very important that you take action on this matter and respond to this ticket within the deadline specified. Failures to investigate, address the issue, and update this ticket with the root cause and actions taken to resolve the problem may result in service interruption. Consider this your only notification. If there is no productive reply, or if the abuse does not cease, Policy Enforcement will be forced to interrupt and/or terminate your service to protect the integrity of the network.

For general reference regarding The Planet’s stance on abuse, refer to:
http://www.theplanet.com/about_us/legal.asp

Please direct any questions regarding this specific issue directly in this ticket, or open a new ticket if you are unable to update this ticket due to being logged in on a different user account.

The server in question has been identified as the origin of spam messages. This issue needs to be addressed quickly to prevent blacklisting, for which fees may be assessed for removal.

Please investigate, resolve the issue, and update this ticket with your actions, referencing the attached reports for details. Failure to perform these actions within 24 hours will result in service interruption. Thank you in advance for your time, efforts, and cooperation.

Regards,
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.

Me - Tuesday December 18th, 2007; 12:33 AM CST

Hi, This is an on-going issue with bitcorp.net. Please view the headers and see that the spam did not originate from our server or domain.

Thank you.

XXXXXX - Tuesday December 18th, 2007; 12:44 AM CST

Dear Customer,

Thank you for your response. Per the headers below, you will see that the emails are originating from 209.62.57.163. We do show that bitcorp.net is 209.62.57.170, which is also tied to your account. We have attached an additional sample complaint, for your convenience. Please keep us updated.

Received: from 209.62.57.163 (HELO bitcorp.net)
by 59diner.com with esmtp (MTPUVMIMNJY NWHDOX)

Regards,
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.

Me - Tuesday December 18th, 2007; 1:38 AM CST

Hmm… I was looking at the x-originating IP.

I do see it says it was received from our main shared IP after closer inspection.

I see a pop3 attack on the server and have banned the primary offending IP. However, I do not see how anyone would be sending spam off this machine –especially not from the bitcorp domain (i.e. no configured email or MX records).

I tried:
> tail /var/log/messages - nothing special

I see email attempts failing in the messages log, so it appears that someone may be spamming off the server and spoofing the addresses/IPs, but how can I tell who is the culprit?

I then tried:
> tail /var/log/maillog

I saw a lot of pop3 failed logins, so I added the offending IP to iptables to DROP

I then tried:
> tail /var/log/exim_mainlog

I see the following but I cannot determine who is sending the emails. Any help would be appreciated.

2007-12-18 02:34:31 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] Warning: Sender rate 281.2 / 1h
2007-12-18 02:34:31 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] Warning: Sender rate 282.2 / 1h
2007-12-18 02:34:31 H=webshieldin02.smed.net (webshieldin01.smed.net) [199.21.28.117] F=<> rejected RCPT <jraleigh@bitcorp.net>: webshieldin02.smed.net (webshieldin01.smed.net) [199.21.28.117] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] F=<> rejected RCPT <jraleigh@bitcorp.net>: aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=mail01.ikarisuper.co.jp (MAIL01) [219.122.47.218] F=<> rejected RCPT <jraleigh@bitcorp.net>: mail01.ikarisuper.co.jp (MAIL01) [219.122.47.218] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=mozart.eds-progical.fr (proxy.edspro.fr) [212.155.167.50] Warning: Sender rate 5.5 / 1h
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] Warning: Sender rate 283.1 / 1h
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] F=<> rejected RCPT <jraleigh@bitcorp.net>: aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:32 H=aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] F=<> rejected RCPT <jraleigh@bitcorp.net>: aaoforums.aao.org (AAOLYRIS.sf.aao.org) [206.14.233.230] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.
2007-12-18 02:34:33 H=mozart.eds-progical.fr (proxy.edspro.fr) [212.155.167.50] F=<> rejected RCPT <jraleigh@bitcorp.net>: mozart.eds-progical.fr (proxy.edspro.fr) [212.155.167.50] is currently not permitted to relay through this server. Perhaps you have not logged into the pop/imap server in the last 30 minutes or do not have SMTP Authentication turned on in your email client.

Me - Tuesday December 18th, 2007; 1:49 AM CST

After further inspection, the messages showing in exim_mainlog appear to be bounced emails trying to deliver to the bitcorp emails but are failing because there is no route -since MX is invalid.

Is it possible that the spoofed emails are bouncing back here and that is what we’re seeing? If not, I need a little more help in figuring out which account is sending the email.

Please keep in mind that this has been an off and on again issue for months. In each case, there was no evidence of any spam actually originating from our server.

-Me

XXXXXX - Tuesday December 18th, 2007; 1:50 AM CST

Dear Customer,

We have received your request for assistance in tracking down this abuse matter. We can suggest the following:
A) Please search the forums at theplanet.com for help with your abuse issue.
B) Consider searching google or an alternative search engine
C) Consider hiring a third party administrator

Please note that as the system administrator, it is expected that you investigate and resolve all abusive issues. Should you require further assistance, please contact our professional services department.


Regards
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.

Me - Tuesday December 18th, 2007; 1:56 AM CST

This is the last thing I need right now.

I haven’t used any administrative hours in quite a while. Why don’t you use my administrative hours and get professional services to look into this?

Believe me, I have searched the forum, requested help, and search all kinds of search engines. It seems that there are no qualified persons who know the answer to “How do I know which user account is sending spam?” It is the ultimate question apparently.

XXXXXX - Tuesday December 18th, 2007; 2:01 AM CST

Dear Customer,

Thank you for your response. If you would like professional services to investigate this issue with your administrative time, you will need to open a new trouble ticket requesting that they do so. Please keep us updated.

Regards,
XXXXXX
Abuse Department
The Planet
Creating a Better World for Your Business.

Me - Tuesday December 18th, 2007; 1:19 PM CST

This spammer has been spoofing emails with our domain for over a year and I am inclined to believe I am wasting my last two hours troubleshooting this problem -as I feel it is still the same issue.

I see no outgoing emails and I see in the logs only incoming emails and “refused: too many connections” from bounces trying to be delivered.

I am not 100% convinced that these emails are originating from our server. However, exim is having a lot of trouble dealing with the influx of spam bounces and I cannot fix that. Here is a related article that sounds exactly like what I’ve been experiencing and what I currently see in my exim logs:

http://www.webhostingtalk.com/archive/index.php/t-347496.html

Me - Tuesday December 18th, 2007; 1:49 PM CST

After nearly 3 hours of investigation, I am convinced the headers were spoofed and you have once again wasted my time.

Shall I send YOU an invoice for $150/hour?

I decided to begin accepting email bounces for jraleigh@bitcorp.net and see what the actual messages really were. I will paste it below, and you can see that the IP address from which the email was received (in all cases) by the bouncer was NOT my server IP address. The fact that the email trail shows my server IP only indicates the high probability that the IP was spoofed. This is further corroborated by the fact that my exim mail queues are clean and we have no outbound connections. I suggest you investigate server at IP: 66.101.198.234. Please do not open any new tickets about spam on this server unless you are able to prove it to me.

Return-Path: <jraleigh@bitcorp.net>
Received: from ns1.hostsedona.com ([66.101.198.234])
by 66-101-196-128.accesssedona.net (8.11.6/8.11.6) with ESMTP id lBEFbB829195
for <president@phonesexcoalition.org>; Fri, 14 Dec 2007 08:37:11 -0700
Received: from co.clark.nv.us (dsl88-247-12379.ttnet.net.tr [88.247.48.91] (may be forged))
by [66.101.198.236] (8.11.6/8.11.6) with SMTP id lBEFb6X27532
for <prez@phonesexcoalition.com>; Fri, 14 Dec 2007 08:37:08 -0700
Received: from 209.62.57.163 (HELO bitcorp.net)
by phonesexcoalition.com with esmtp (TBWJNXUQOE CCDQWA)
id HMbClH-7iz5ZC-F7
for prez@phonesexcoalition.com; Fri, 14 Dec 2007 17:37:06 +0200
Message-ID: <114701c83e67$2e793c70$c0a80202@Marva>
From: “Marva T. Blackman”<Marva@bitcorp.net>
To: “Jerry N. Denny”<prez@phonesexcoalition.com>
Subject: Promote your little soldier of love in a new year!
Date: Fri, 14 Dec 2007 17:37:06 +0200
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary=”—-=_NextPart_4421_11AF_01C83E77.F2020C70″
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1441
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1441

This is a multi-part message in MIME format.

YYYYYY - Tuesday December 18th, 2007; 2:38 PM CST

Dear Customer,

This E-Mail did in fact originate from your server. This assessment is based on the following information in the mail headers:

Received: from 209.62.57.163 (HELO bitcorp.net) by phonesexcoalition.com with esmtp (TBWJNXUQOE CCDQWA) id HMbClH-7iz5ZC-F7 for prez@phonesexcoalition.com; Fri, 14 Dec 2007 17:37:06 +0200 Message-ID: <114701c83e67$2e793c70$c0a80202@Marva>

This line was generated by the recipient mail server (in this case, phonesexcoalition.com [66.101.198.236]). As such, it is not possible that this information is spoofed, as it is not generated until after the message has left the originating server.

Please keep us updated with your progress in this matter.

Regards,
YYYYYY
Abuse Team
The Planet
abuse@theplanet.com

Me - Tuesday December 18th, 2007; 3:27 PM CST

Sorry but I am not convinced. I see no outgoing connections on our server and the spam bounces are getting worse -indicating an on going spam effort.

I have exhausted all of my options and theplanet is obviously unwilling to help without charging me astronomical prices -in spite of the fact that I am the victim here.

What exactly do you propose we do?

Me - Tuesday December 18th, 2007; 3:28 PM CST

By the way, I suspended the bitcorp.net domain along with numerous others, and the spam continued to pour in and increase. Doesn’t that tell you anything?

Me - Tuesday December 18th, 2007; 3:37 PM CST

By the way, phonesexcoalition.com doesn’t strike me as being a 100% reputable domain. I don’t think I’m going to trust a header possibly forged by THAT domain over my gut feeling. My gut could be wrong, but no one can show me definitively that my server is sending spam by looking at server logs, exim queues, or anything else and I can tell from those same logs and queues that the server isn’t doing much besides accepting a million bounced emails.

Update: I got a response from them (12.19.2007)

ZZZZZZ - Wednesday December 19th, 2007; 6:02 PM CST

From the headers that were attached, there is indeed good evidence to support the possibility that your server, and specifically the domain “bitcorp.net”, has fallen victim of a joejob attack in which the headers have been forged to make the appearance that your server was responsible for being the original sender of the message.

Unfortunately, options to combat this type of abuse are somewhat limited. I would suggest having the administrator of bitcorp.net set up a strict SPF record to specify the server as the only authorized sender of email on behalf of the domain.

I would also change the option to “Send all unrouted e-mail for:” from “:blackhole:” to “Discard with error to sender (at SMTP time)”.

Our abuse team will be notified of my findings.

Please let us know if any additional assistance is needed. Thank you!

Update: Having this issue?

Read how to combat a Joe Job Spam Attack.

After that support incident chain; the spam bounces have stopped coming in. If this were spam originating from my server, it would not stop without some sort of intervention.

In any case, here’s a couple of things to note:

  • Professional services they recommend are $150/hour
  • ThePlanet is not as bad as this incident might make them seem

Generally speaking, I have been VERY pleased with the support from ThePlanet. They are quick to respond and very knowledgeable. In many cases, they work quickly and give detailed explanations of how they fixed things. It’s just these cases of “abuse” where the support seems to break down. To their defense, these spammers are very good at what they do, and it is understandable how -at first glance- these emails can point to my server being the culprit. However, I have a repeated history of proving my points and my opinions should carry a little more weight. That department is really quick to point a finger and let you resolve it. They should spend more time investigating before they jump the gun. That’s all.

p.s. At least they are WORLDS BEYOND Hostgator -where I originally started hosting all of my sites. Man, don’t get me started. I think that might have been some of the worst couple of years of my life, and I’ll spare you the miles of support tickets from them. Something like:

Me - “All of my sites are down!!”

Hostgator (2 hours later) - “Who are you? What is your account?”

Me - “Oh my God! My account number is in the original ticket”

Hostgator (6 hours later) - “Ok. I can reboot your server, but I need your password to verify your account”

That’s only a “slight” exaggeration of the kinds of support they gave me! I just don’t want to sift through my old emails to get the wording right. That’s basically how it went. Whew. I’m glad those days are over! lol.


Share This! Social Bookmarking and Sharing

Bookmark to del.icio.us:Diplomacy was never my strong suit  Digg It:Diplomacy was never my strong suit  Share with furl:Diplomacy was never my strong suit  Share with reddit:Diplomacy was never my strong suit  Add to blogmarks:Diplomacy was never my strong suit  Add to Y!:Diplomacy was never my strong suit

Search Engine Conspiracy

↑ Grab this Headline Animator


Leave a Reply