Serious concurrency problems on fast systems
in [Java Programming]
|
Prev: DOWNLOAD FREE PALTALK LIVE VIDEO AND VOICE CHAT SOFTWARE
Next: Should -Xmx be a multiple of -Xms?
From: Arne Vajhøj on 2 Jun 2010 20:03 On 02-06-2010 09:52, Mike Amling wrote: > Arne Vajh�j wrote: >> -XXdonotkickthreadoffcpuifitisabadtimeforperformance does not exist. > > No, but someday there could be an option to let a Thread synchronized on > a monitor for which another Thread is waiting run a little longer, in > hope that it will desynchronize. Maybe. But I think it would be difficult to implement, because the JVM knows about the locks but the OS manage the threads. But it is possible. If the OS API has: - StartBadTimeToKickMeOff - EndBadTimeToKickMeOff then the JVM could call it at enter and exit. Arne
From: Mike Schilling on 3 Jun 2010 01:37 "Arne Vajh�j" <arne(a)vajhoej.dk> wrote in message news:4c06f03d$0$274$14726298(a)news.sunsite.dk... > On 02-06-2010 00:57, Mike Schilling wrote: >> >> >> "Arne Vajh�j" <arne(a)vajhoej.dk> wrote in message >> news:4c059872$0$272$14726298(a)news.sunsite.dk... >>> On 01-06-2010 00:21, Kevin McMurtrie wrote: >>>> I've been assisting in load testing some new high performance servers >>>> running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is >>>> suspending threads for time-slicing in very unfortunate locations. >>> >>> That should not come as a surprise. >>> >>> The thread scheduler does not examine the code for convenience. >>> >>> Correct code must work no matter when the in and out of >>> CPU happens. >>> >>> High performance code must work efficiently no matter when the >>> in and out of CPU happens. >>> >>> > For >>>> example, a thread might suspend in Hashtable.get(Object) after a call >>>> to >>>> getProperty(String) on the system properties. It's a synchronized >>>> global so a few hundred threads might pile up until the lock holder >>>> resumes. Odds are that those hundreds of threads won't finish before >>>> another one stops to time slice again. The performance hit has a ton of >>>> hysteresis so the server doesn't recover until it has a lower load than >>>> before the backlog started. >>>> >>>> The brute force fix is of course to eliminate calls to shared >>>> synchronized objects. All of the easy stuff has been done. Some >>>> operations aren't well suited to simple CAS. Bottlenecks that are part >>>> of well established Java APIs are time consuming to fix/avoid. >>> >>> High performance code need to be designed not to synchronize >>> extensively. >>> >>> If the code does and there is a performance problem, then fix >>> the code. >>> >>> There are no miracles. >> >> Though giving a thread higher priority while it holds a shared lock >> isn't exactly rocket science; VMS did it back in the early 80s. JVMs >> could do a really nice job of this, noticing which monitors cause >> contention and how long they tend to be held. A shame they don't. > > A higher priority could reduce the problem, but would not > eliminate it. > > Arne > > PS: I thougth DECThreads came with VMS 5.5 in 1991- VMS actually gave a boost to *processes* that held locks. Close enough to the same thing, methinks.
From: Arne Vajhøj on 3 Jun 2010 22:00 On 03-06-2010 01:37, Mike Schilling wrote: > "Arne Vajh�j" <arne(a)vajhoej.dk> wrote in message .... >> PS: I thougth DECThreads came with VMS 5.5 in 1991- > > VMS actually gave a boost to *processes* that held locks. Close enough > to the same thing, methinks. True. Arne PS: A quick glance in IDS indicates that it is locking on mutexes not regular $ENQ/$DEQ that raises priority.
From: Mike Schilling on 3 Jun 2010 23:09 "Arne Vajh�j" <arne(a)vajhoej.dk> wrote in message news:4c085e13$0$280$14726298(a)news.sunsite.dk... > On 03-06-2010 01:37, Mike Schilling wrote: >> "Arne Vajh�j" <arne(a)vajhoej.dk> wrote in message > ... >>> PS: I thougth DECThreads came with VMS 5.5 in 1991- >> >> VMS actually gave a boost to *processes* that held locks. Close enough >> to the same thing, methinks. > > True. > > Arne > > PS: A quick glance in IDS indicates that it is locking on > mutexes not regular $ENQ/$DEQ that raises priority. That's a great book, but I must have given away my copy at least fifteen yeas ago.
From: Arne Vajhøj on 6 Jun 2010 22:21
On 02-06-2010 01:45, Kevin McMurtrie wrote: > In article<4c048acd$0$22090$742ec2ed(a)news.sonic.net>, > Kevin McMurtrie<mcmurtrie(a)pixelmemory.us> wrote: >> I've been assisting in load testing some new high performance servers >> running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is >> suspending threads for time-slicing in very unfortunate locations. For >> example, a thread might suspend in Hashtable.get(Object) after a call to >> getProperty(String) on the system properties. It's a synchronized >> global so a few hundred threads might pile up until the lock holder >> resumes. Odds are that those hundreds of threads won't finish before >> another one stops to time slice again. The performance hit has a ton of >> hysteresis so the server doesn't recover until it has a lower load than >> before the backlog started. >> >> The brute force fix is of course to eliminate calls to shared >> synchronized objects. All of the easy stuff has been done. Some >> operations aren't well suited to simple CAS. Bottlenecks that are part >> of well established Java APIs are time consuming to fix/avoid. >> >> Is there JVM or Linux tuning that will change the behavior of thread >> time slicing or preemption? I checked the JDK 6 options page but didn't >> find anything that appears to be applicable. > > To clarify a bit, this isn't hammering a shared resource. I'm talking > about 100 to 800 synchronizations on a shared object per second for a > duration of 10 to 1000 nanoseconds. Yes, nanoseconds. That shouldn't > cause a complete collapse of concurrency. But either it does or your entire problem analysis is wrong. > My older 4 core Mac Xenon can have 64 threads call getProperty(String) > on a shared Property instance 2 million times each in only 21 real > seconds. That's one call every 164 ns. It's not as good as > ConcurrentHashMap (one per 0.30 ns) but it's no collapse. That is a call per clock cycle. ?!?! > Many of the basic Sun Java classes are synchronized. Practically only old ones that you should not be using anymore anyway. Arne |