Serious concurrency problems on fast systems
in [Java Programming]
|
Prev: DOWNLOAD FREE PALTALK LIVE VIDEO AND VOICE CHAT SOFTWARE
Next: Should -Xmx be a multiple of -Xms?
From: Robert Klemme on 1 Jun 2010 12:55 On 01.06.2010 13:39, Lew wrote: > Kevin McMurtrie wrote: >> I've been assisting in load testing some new high performance servers >> running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is >> suspending threads for time-slicing in very unfortunate locations. For >> example, a thread might suspend in Hashtable.get(Object) after a call to >> getProperty(String) on the system properties. Just out of curiosity: How did you find out? >> It's a synchronized >> global so a few hundred threads might pile up until the lock holder >> resumes. Odds are that those hundreds of threads won't finish before >> another one stops to time slice again. The performance hit has a ton of >> hysteresis so the server doesn't recover until it has a lower load than >> before the backlog started. >> >> The brute force fix is of course to eliminate calls to shared >> synchronized objects. All of the easy stuff has been done. Some > > You call that "brute force" as if it weren't the actual, correct answer. > >> operations aren't well suited to simple CAS. Bottlenecks that are part >> of well established Java APIs are time consuming to fix/avoid. > > But necessary. Having repeated calls on system properties that require > synchronization is just plain stupid. System properties are the ones > that don't change during a program run, so they should be (and should > have been) written once into an immutable structure at class-load time > and read thence thereafter. End of synchronization woes for that one. > >> Is there JVM or Linux tuning that will change the behavior of thread >> time slicing or preemption? I checked the JDK 6 options page but didn't >> find anything that appears to be applicable. > > The jmap/jhat dump utilities have some of that, IIRC. Otherwise you > break into the add-on diagnostic tools. > > But really it sounds like your code needs refactoring in that it did not > handle concurrency correctly from jump. I couldn't agree more to what Lew wrote. If all your threads hammer on single global resources you've got a design level issue: that application is simply not built with scalability in mind - even if the effect did not show up yet with other hardware / different load. This is nothing you can blame the JVM or hardware for. Kind regards robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/
From: Arne Vajhøj on 1 Jun 2010 19:32 On 01-06-2010 00:21, Kevin McMurtrie wrote: > I've been assisting in load testing some new high performance servers > running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is > suspending threads for time-slicing in very unfortunate locations. That should not come as a surprise. The thread scheduler does not examine the code for convenience. Correct code must work no matter when the in and out of CPU happens. High performance code must work efficiently no matter when the in and out of CPU happens. > For > example, a thread might suspend in Hashtable.get(Object) after a call to > getProperty(String) on the system properties. It's a synchronized > global so a few hundred threads might pile up until the lock holder > resumes. Odds are that those hundreds of threads won't finish before > another one stops to time slice again. The performance hit has a ton of > hysteresis so the server doesn't recover until it has a lower load than > before the backlog started. > > The brute force fix is of course to eliminate calls to shared > synchronized objects. All of the easy stuff has been done. Some > operations aren't well suited to simple CAS. Bottlenecks that are part > of well established Java APIs are time consuming to fix/avoid. High performance code need to be designed not to synchronize extensively. If the code does and there is a performance problem, then fix the code. There are no miracles. > Is there JVM or Linux tuning that will change the behavior of thread > time slicing or preemption? I checked the JDK 6 options page but didn't > find anything that appears to be applicable. -XXdonotkickthreadoffcpuifitisabadtimeforperformance does not exist. Arne
From: Lew on 1 Jun 2010 19:54 Arne Vajhøj wrote: > High performance code need to be designed not to synchronize > extensively. > > If the code does and there is a performance problem, then fix > the code. This comes up again and again and again in our profession. People almost never want to hear it. They will go broke spending on "quick-fix" solutions that aren't quick, fix nothing and solve nothing. The answer is always "fix the code". This always causes controversy. It shouldn't. Fixing the code is cheaper and works. Not fixing the code is more expensive and doesn't work. > There are no miracles. This is both true and false. It's true because the kind of miracle people are hoping for is one that lets them not admit there is a problem. Doomed. It's false because there is a miracle. Doing it right turns out to be cheaper, easier, quicker and more satisfying, starting very early in the process, and gets results. It seems miraculous that doing things the right way produces correct results astonishingly soon. -- Lew
From: Arne Vajhøj on 1 Jun 2010 20:21 On 01-06-2010 19:54, Lew wrote: > Arne Vajhøj wrote: >> High performance code need to be designed not to synchronize >> extensively. >> >> If the code does and there is a performance problem, then fix >> the code. > > This comes up again and again and again in our profession. People almost > never want to hear it. They will go broke spending on "quick-fix" > solutions that aren't quick, fix nothing and solve nothing. > > The answer is always "fix the code". This always causes controversy. It > shouldn't. > > Fixing the code is cheaper and works. Not fixing the code is more > expensive and doesn't work. > >> There are no miracles. > > This is both true and false. > > It's true because the kind of miracle people are hoping for is one that > lets them not admit there is a problem. > > Doomed. > > It's false because there is a miracle. Doing it right turns out to be > cheaper, easier, quicker and more satisfying, starting very early in the > process, and gets results. It seems miraculous that doing things the > right way produces correct results astonishingly soon. Even though doing it right is likely to be less costly in the long run, then it is still hard work initially, so I will not call it a miracle. Arne
From: Mike Schilling on 2 Jun 2010 00:57
"Arne Vajh�j" <arne(a)vajhoej.dk> wrote in message news:4c059872$0$272$14726298(a)news.sunsite.dk... > On 01-06-2010 00:21, Kevin McMurtrie wrote: >> I've been assisting in load testing some new high performance servers >> running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is >> suspending threads for time-slicing in very unfortunate locations. > > That should not come as a surprise. > > The thread scheduler does not examine the code for convenience. > > Correct code must work no matter when the in and out of > CPU happens. > > High performance code must work efficiently no matter when the > in and out of CPU happens. > > > For >> example, a thread might suspend in Hashtable.get(Object) after a call to >> getProperty(String) on the system properties. It's a synchronized >> global so a few hundred threads might pile up until the lock holder >> resumes. Odds are that those hundreds of threads won't finish before >> another one stops to time slice again. The performance hit has a ton of >> hysteresis so the server doesn't recover until it has a lower load than >> before the backlog started. >> >> The brute force fix is of course to eliminate calls to shared >> synchronized objects. All of the easy stuff has been done. Some >> operations aren't well suited to simple CAS. Bottlenecks that are part >> of well established Java APIs are time consuming to fix/avoid. > > High performance code need to be designed not to synchronize > extensively. > > If the code does and there is a performance problem, then fix > the code. > > There are no miracles. Though giving a thread higher priority while it holds a shared lock isn't exactly rocket science; VMS did it back in the early 80s. JVMs could do a really nice job of this, noticing which monitors cause contention and how long they tend to be held. A shame they don't. |