2136 server STILL LEAKS memory! ack!

[Bandit_Z]

I applied the latest patches for Linux and Win32.
On my Linux server, I am still seeing what appears to be a memory leak.

Unfortunately, I lost my paste buffer that had the process info. Will post again to this thread when server starts to eat more ram.

At it's worst, the THREE processes for ucc-bin where each taking 247M of physical ram EACH. 3x247M sounds like a leak, however I didnt let it get any worse before killing the processes.

Previously, it would consume all the ram in the server until other processes started to die...

Anyone else seeing this?

I have clans that want to be hosted, but I can't stand to run more than 2 ports on a 2 gig server...unacceptable!

[PapaDoom]

I'm observing the same behavior here, watching the ucc-bin processes in 'top'. I'm running Mandrake 8.2 on a system with 512 MB of RAM.

I've had 2136 running for about 24 hours. When I checked first thing this morning, each of the three processes was sitting at 83 MB of used physical memory ("RSS" in 'top'). I've watched five or six games play through, and each process is now (about two hours later) at 89 MB. My system is not particularly active- I'd be guessing 20 or so games a day run, usually just with two or three players.

Here's what it looks like at the moment:


[code:1:634fa959bd]3 processes: 3 sleeping, 0 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 0.1% system, 0.0% nice, 99.8% idle
Mem: 513800K av, 506236K used, 7564K free, 0K shrd, 76584K buff
Swap: 248968K av, 1532K used, 247436K free 249552K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
4288 root 18 0 92060 89M 6136 S 0.0 17.9 7:18 ucc-bin
4294 root 8 0 92060 89M 6136 S 0.0 17.9 0:00 ucc-bin
4295 root 9 0 92060 89M 6136 S 0.0 17.9 0:00 ucc-bin[/code:1:634fa959bd]



This could be perfectly normal. Possibly UT2003 will keep grabbing available memory until it runs out, and then do some garbage cleaning. But it seems a little odd.

Any other observations?

[obiwan]

this sucks

8:13pm up 1 day, 6:48, 1 user, load average: 0.00, 0.00, 0.00
43 processes: 40 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 0.1% user, 0.3% system, 0.0% nice, 99.4% idle
Mem: 376600K av, 367212K used, 9388K free, 0K shrd, 23204K buff
Swap: 457844K av, 0K used, 457844K free 237672K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1417 obiwan 15 0 90880 88M 6260 S 0.0 24.1 27:09 ucc-bin
1423 obiwan 15 0 90880 88M 6260 S 0.0 24.1 0:00 ucc-bin
1424 obiwan 15 0 90880 88M 6260 R 0.0 24.1 0:04 ucc-bin

I bought extra ram yesterday now got 384mb in a box that that dont really need it. after a few hours something sucks all memory and eventually the box crashes out, I can't even say its the 3 x ucc processes, because when i reboot and check top I have something like 256mb free, which if you do 3 x 45=135 take that from my 384 gives 249 left without considering other processes!!! but top show at least 256 available grrrrrrrrrrrrr, whats going on here, is each of the three threads consuming the same amount of ram?? or is it just 88mb for all three threads?? something consumes large amount all at once. for example after running for say 12 hours I had 120mb free (server in use on and off) 4-5hrs later theres nothing left, server may have only been used by 2-3 players.....Im confused, anyone including epic got a clue???

[obiwan]

this sucks

okay killing the process our stopping ucc.init (if you use it) didnt release all the ram, I had to reboot to get every thing back!!! okay before i launch ucc.init

9:06pm up 3 min, 1 user, load average: 0.14, 0.08, 0.03
38 processes: 36 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: 0.1% user, 0.1% system, 0.0% nice, 99.6% idle
Mem: 376600K av, 42756K used, 333844K free, 0K shrd, 7372K buff
Swap: 457844K av, 0K used, 457844K free 20956K cached

when the game is first launched

9:10pm up 7 min, 2 users, load average: 0.16, 0.08, 0.02
43 processes: 42 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.1% user, 0.3% system, 0.0% nice, 99.4% idle
Mem: 376600K av, 126044K used, 250556K free, 0K shrd, 8136K buff
Swap: 457844K av, 0K used, 457844K free 60456K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
924 obiwan 15 0 46388 45M 5552 S 0.0 12.3 0:08 ucc-bin
930 obiwan 15 0 46388 45M 5552 S 0.0 12.3 0:00 ucc-bin
931 obiwan 15 0 46388 45M 5552 S 0.0 12.3 0:00 ucc-bin

Ill post in 12hours and see where we're at......watch this thread, as for not releasing all the ram on kill apparently something to do with malloc, but I havent a clue....

[Bandit_Z]

13767 unreal 19 19 183M 183M 6076 S N 7.7 9.1 127:16 ucc-bin
13773 unreal 18 19 183M 183M 6076 S N 0.0 9.1 0:00 ucc-bin
13774 unreal 19 19 183M 183M 6076 S N 0.0 9.1 0:32 ucc-bin
13756 unreal 19 19 180M 180M 6136 S N 3.5 8.9 128:48 ucc-bin
13762 unreal 19 19 180M 180M 6136 S N 0.0 8.9 0:00 ucc-bin
13763 unreal 19 19 180M 180M 6136 S N 0.0 8.9 0:32 ucc-bin

I will let it continue to grow, last time with 2136 release I was seeing as much as 250M per process. Not as bad as before 2136, but something is still broken.

I dont know about the rest of you guys, but I have been hosting servers for a long time. I have several servers racked and have a lot of money invested, even so I have to make better use of my resources and Unreal has always been a problem for me.

In the original UT release, there was a terrible bug that caused CPU usage to skyrocket to 100% and pegged the processors while everyone's ping climbed into the 2000 range. This was unacceptable, I have other ports that I host. I can't have an entire CPU committed. But at least it didnt eat all the ram. Epic didnt fix this bug for AGES, only a long time later did they actually find it...

Now the UT2k3 CPU utilization is great!, but it is destroying my server with ram leaks.

I can't keep baby sitting my servers, and I dont like the idea of having to forcibly kill the processes when they consume too much. I can change my monitoring script to do this, but I dont like this solution...

If something isn't done soon, I will have to abandon any efforts to host this game until they have fixed the problem...

[obiwan]

okay nearly 12 hours later, and it been used for approximately 2 hours in total

8:08am up 11:06, 2 users, load average: 0.06, 0.02, 0.00
44 processes: 41 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 0.1% user, 0.7% system, 0.0% nice, 99.0% idle
Mem: 376600K av, 330920K used, 45680K free, 0K shrd, 54328K buff
Swap: 457844K av, 0K used, 457844K free 176812K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
924 obiwan 15 0 85744 83M 6260 S 0.0 22.7 17:40 ucc-bin
930 obiwan 15 0 85744 83M 6260 S 0.0 22.7 0:00 ucc-bin
931 obiwan 15 0 85744 83M 6260 R 0.0 22.7 0:02 ucc-bin

strange thing happened while i was watching it last nite, as I left the game it showed 52mb free, the game continued until the bots finished, then as the process went into idle my free ram dropped instantly to 36mb!!! wtf's all that about?? somrtime over nite some got releasd because its back to 45mb odd now.......so is this just the free dedicated server or the retail server too?

[obiwan]

hey hum, okay last nite had four player and within minutes available free ram was down to 5mb ish,

script must have restarted ucc.init at some point but i still have missing memory becuase the figures dont add up....

9:07am up 1 day, 12:04, 1 user, load average: 0.00, 0.00, 0.00
42 processes: 38 sleeping, 4 running, 0 zombie, 0 stopped
CPU states: 0.1% user, 0.3% system, 0.0% nice, 99.4% idle
Mem: 376600K av, 326592K used, 50008K free, 0K shrd, 43560K buff
Swap: 457844K av, 0K used, 457844K free 185208K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1709 obiwan 15 0 85136 83M 6288 R 0.0 22.6 23:00 ucc-bin
1715 obiwan 15 0 85136 83M 6288 S 0.0 22.6 0:00 ucc-bin
1716 obiwan 15 0 85136 83M 6288 R 0.0 22.6 0:03 ucc-bin

the only way I can get all my ram back is a reboot, why wont kill ot stopping ucc.init release the allocated memory???

[Bandit_Z]

Hate to say it, but thats how your system is supposed to work.
Notice how much is going to "cached" ?

That memory will be freed upon demand for more pages of memory when needed/requested.

Swap: 0K av, 0K used, 0K free 1420900K cached

I don't use any swap space, but I have 2.0gig ram. !.5g is being used to cache filesystem objects, and currently I am seeing my UCC processes sitting around 150Mx3 per port. Has anyone posted any info on what each process is doing? Is it possible to get rid of the extra forked processes?

[obiwan]

as i said I havent a clue with linux (still learning) so everything is okay then becuase mine doesnt grow much higher than 88mb :(

thanx for the knowledge Bandit_Z, Ill wonder off now and look into my other problems.......

[Smurphy]

as i said I havent a clue with linux (still learning) so everything is okay then becuase mine doesnt grow much higher than 88mb :(

thanx for the knowledge Bandit_Z, Ill wonder off now and look into my other problems.......
Well Folks,

guess you don know Linux. Wvwn if you Folks think the UT2003-Server does not release the memory - it false. The LInux system, in contratrary to Windows Systems - uses every memory available to cache system processes. Means - even if you have 0MBytes left, check the Buffers.
You'l see many is left.
Regarding the Memory Leak - I do agree this is the case. However- the time the System required to fill up the Server's memory depends on the number of active map in the rotation list. I did put only 6 Maps - and now it takes about half a day to have the ucc-bin executable eat up to 300MBytes of Ram:

9 0 323M 183M 3536 S 16.3 73.6 788:44 ucc-bin

788Minutes ... 6 Maps. If I put in 12 Maps into the rotation - it takes about 8 Hours ...
Note - the Server is a Dual-450MHz Pentium II with 256 MBytes of Ram - and a 100MBit Internet connection :)

[Scabies]

I was working on a named problem and found this in the named FAQ...maybe it means something in this case.......

Q: Why do I see 5 (or more) copies of named on Linux?

A: Linux threads each show up as a process under ps. The approximate
number of threads running is n+2, where n is the number of CPUs. Note that the amount of memory used is not cumulative; if each process is using 10M of memory, only a total of 10M is used.

[b0x]

Before patching to 2166 i had a lovely 768mb of ram being used after about 24 hours :dontgetit: . Patched to 2166 and its gone from 46mb of ram to 112, where its hovering about.

Will see if it holds :(

jon