lots of dropped TCP packets on local IP

[127.0.0.1]

hi all:

in my logwatch logs each day i am seeing hundreds and sometimes thousands of tcp packets being dropped on many of the (local) ip-based sites i am running on the server (centos 5 + interworx panel 3.0.3). the entries look like:

Dropped 2432 packets on interface eth1
From 24.213.78.182 - 1 packet to udp(1026)
From 58.10.68.163 - 2 packets to tcp(4899)
From 58.241.178.213 - 8 packets to tcp(2967)
From 58.246.73.74 - 4 packets to tcp(10000)
From 58.253.235.40 - 2 packets to tcp(10000)
From 62.87.178.204 - 2 packets to tcp(8080)
From 12.34.56.78 - 1712 packets to tcp(30,134,140,390,525,526,652,781,798,908,1030,10 34,1035,1043,1046,1070,1089,1104,1106,1107,1109,11 15,1119,1121,1138,1140,1151,1155,1159,1168,1172,11 75,1177,1179,1180,1193,1196,1197,1200,1210,1212,12 18,1232,1236,1238,1240,1241,1243,1248,1266,1271,12 76,1279,1281,1299,1311,1313,1316,1325,1343,1360,13 76,1392,1393,1394,1407,1423,1468,1504,1506,1509,15 10,1511,1513,1514,1563,1573,1627,1634,1645,1651,16 55,1667,1671,1678,1683,1728,1729,1730,1732,1733,17 39,1742,1744,1748,1753,1754,1755,1756,1765,1767,17 70,1771,1773,1775,1776,1778,1786,1788,1793,1794,17 97,1798,1799,1806,1813,1829,1831,1837,1842,1849,18 83,1885,1887,1890,1899,1934,1935,2007,2008,2014,20 37,2038,2039,2040,2043,2088,2093,2102,2108,2110,21 11,2125,2130,2147,2155,2162,2165,2175,2177,2182,21 86,2188,2189,2217,2220,2225,2234,2288,2318,2344,23 57,2359,2371,2404,2420,2434,2461,2464,2534,2535,25 37,2562,2578,2607,2621,2625,2643,2645,2647,2649,26 51,2668,2688,2699,2736,2751,2752,2788,2794,2798,28 03,2848,2881,2890,2897,2902,2924,2982,2991,3012,30 13,3026,3052,3066,3106,3113,3119,3123,3170,3182,32 23,3244,3246,3262,3267,3310,3353,3384,3429,3432,34 38,3468,3470,3491,3492,3493,3494,3495,3571,3685,37 01,3714,3732,3742,3749,3752,3753,3755,3761,3762,37 63,3770,3786,3799,3815,3823,3825,3826,3827,3855,38 87,3906,3907,3939,3962,3996,4116,4125,4225,4241,42 42,4403,4412,4413,4417,4421,4449,4534,4577,4585,46 10,4634,4656,4695,4759,4771,4850,4851,4852,4854,48 55,4930,4951,4952,4953,4954,4958,4959,4969,4983,49 91,5000,5177,5372,5693,6401,7149,7210,7281,8828,92 39,9658,9736,9788,9897,10183,10185,11314,11316,115 53,11779,11797,11810,11914,11929,11944,11983,12439 ,12682,12685,12729,12741,13031,13060,13568,14274,1 4336,14966,15225,15350,15576,15879,15880,16164,165 65,16834,18573,19077,19262,20184,20763,20789,20866 ,20884,20932,21346,21796,22111,22158,22881,22882,2 3707,24575,25469,26505,26591,27367,28168,30277,302 91,30292,30294,30498,30728,32440,32575,32778,32799 ,32961,33000,33001,33002,33159,33186,33201,33216,3 3274,33388,33405,33437,33458,33476,33515,33581,336 44,33684,34134,34490,35169,35336,35469,35928,35952 ,36014,36016,36050,37046,37226,37852,37889,37917,3 7933,37956,38103,38105,39164,39172,39186,39474,401 50,40287,40656,40660,40694,41752,42073,42090,42130 ,42497,42677,43653,44232,44655,44934,46441,47379,4 9027,49161,49183,49199,49200,49236,49237,49284,492 91,49299,49304,49310,49374,49379,49400,49401,49417 ,49419,49449,49454,49455,49471,49473,49474,49478,4 9482,49491,49500,49518,49519,49523,49547,49557,495 58,49560,49719,49721,49734,49736,49772,49833,49852 ,49959,49977,49983,50006,50007,50073,50205,50250,5 0253,50257,50286,50345,50360,50364,50369,50414,505 13,50519,50571,50573,50590,50592,50596,50636,50642 ,50709,50716,50719,50739,50795,50816,50840,50894,5 0908,50910,50914,50915,50918,50921,50957,50968,509 87,51019,51021,51068,51128,51136,51146,51154,51159 ,51174,51184,51186,51187,51223,51277,51360,51375,5 1415,51420,51425,51445,51448,51602,51641,51644,516 47,51662,51671,51676,51687,51688,51710,51745,51809 ,51828,51883,51928,51929,52013,52014,52033,52072,5 2073,52074,52195,52244,52272,52396,52414,52617,530 47,53064,53082,53125,53140,53161,53230,53342,53367 ,53397,53505,53506,53518,53547,53680,53749,54142,5 4193,54299,54303,54576,54599,54619,54645,54663,546 64,54665,54666,54667,54765,54818,54999,55039,55275 ,55317,55322,55624,55688,56097,56189,56249,56442,5 6444,56462,56464,56473,56491,56500,56502,56512,565 25,56526,56535,56537,56538,56539,56544,56563,56566 ,56567,56568,56569,56571,56583,56588,56589,56593,5 6594,56596,56597,56600,56602,56604,56608,56612,566 14,56615,56618,56619,56620,56621,56623,56624,56660 ,56661,56664,56666,56667,56690,56698,56700,56705,5 6706,56709,56710,56713,56714,56724,56733,56737,567 39,56742,56746,56747,56749,56755,56757,56759,56761 ,56765,56768,56771,56773,56777,56779,56786,56787,5 6790,56795,56799,56801,56804,56809,56811,56812,568 13,56818,56819,56821,56822,56824,56827,56828,56831 ,56858,56861,56867,56868,56872,56873,56874,56875,5 6877,56878,56883,56888,56894,56898,56904,56912,569 18,56919,56922,56923,56938,56946,56951,56954,56965 ,56981,56982,56983,56984,56985,56989,57061,57069,5 7070,57079,57080,57081,57082,57085,57100,57105,571 06,57107,57108,57109,57111,57112,57113,57344,57489 ,57648,57682,57786,57880,57972,57984,58005,58014,5 8113,58167,58289,58316,58335,58336,58337,58538,585 49,58701,58802,59119,59271,59405,59509,59783,59797 ,59829,59832,59946,60043,60095,60112,60146,60312,6 0600,60651,60914,60915,60994,61004,61005,61089,611 24,61129,61335,61394,61406,61471,61475,61518,61562 ,61582,61586,61593,61692,61719,61775,61999,62005,6 2009,62011,62080,62095,62119,62161,62203,62217,624 66,62549,62558,62697,62871,62965,62997,63004,63047 ,63048,63074,63094,63119,63339,63344,63523,63534,6 3737,63755,63768,63795,63796,63797,63800,63861,638 99,64017,64025,64032,64117,64120,64121,64122,64165 ,64280,64421,64475,64594,64620,64923,64925,65165,6 5244,65283)

etc.... etc.... etc...

anyone have any ideas as to why i am seeing these? should i have a concern about it. 12.34.56.78 above is subst. for a real ip on the box.

[127.0.0.1]

hello?...anyone here?

[IWorx-Paul]

It's hard to say for sure just based on that if there's cause for concern. One interpretation could be an attempted port scan from the server itself, perhaps through a php script exploit, or a user with a shell account. There might be a less nefarious explanation as well, but I'd take it as an opportunity to look for anything else out of the ordinary on the server.

Wish I could be more helpful :(

Paul

[127.0.0.1]

thanks paul. 'preciate the advice.

j

[127.0.0.1]

along with dropped packets on the server's own IP's, the server is now randomly crashing. hmmm.... decided to send in a TT. checked dmesg and there does not appear to be a panic. looks to be software related. do you have any other boxes reported with random crashes?

[127.0.0.1]

and another crash....this morning. what can i look at to see what is causing this condition?

[IWorx-Paul]

Sorry to hear that :(

Checking the console before the server is rebooted would be one place to look.

Paul

[127.0.0.1]

please explain. server is remote and obviously i have no access to anything when it goes down. is there a specific log file(s) i can look at to try to identify where the crashes are coming from? /var/log/messages just shows a high number of denied TCP packets to local IP's on the box. i am thinking iptables or apf.....possible?

thx again,
j

[IWorx-Paul]

is there a specific log file(s) i can look at to try to identify where the crashes are coming from?
/var/log/messages would usually be the place. Are you 100% sure the box is crashing? Or do you just know that it goes "offline" and you have to remote reboot it?

please explain. server is remote and obviously i have no access to anything when it goes down

I mean someone that does have physical access to the server would have to go plug in a monitor and keyboard and see what the status of the server is when it's down. It may just show a blank screen, which wouldn't be helpful, but maybe it has a meaningful message, or maybe it's not even actually crashed, but just blocked from the network for some reason. Either way it would give more information about the problem, of which there seems to be very little after the reboot.

Paul

[127.0.0.1]

i checked /var/log/messages prior to the time of the crash and it appears to be showing normal activity and then server goes off line. data center has to physically reboot each time. there is no remote access when this happens. i have confirmed with colo provider that there are no network probs on their end.

[127.0.0.1]

and down again, as of 12:50...

update: back up w/ a reboot. console now on box to monitor. see TT. could this be related to fively.pex adjustment to accommodate axfrdns + cron updates? dunno....

j

[dazza]

What was your solution? Seems I'm having the same issue.