Serious concurrency problems on fast systems
in [Java Programming]
|
Prev: DOWNLOAD FREE PALTALK LIVE VIDEO AND VOICE CHAT SOFTWARE
Next: Should -Xmx be a multiple of -Xms?
From: Kevin McMurtrie on 1 Jun 2010 00:21 I've been assisting in load testing some new high performance servers running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is suspending threads for time-slicing in very unfortunate locations. For example, a thread might suspend in Hashtable.get(Object) after a call to getProperty(String) on the system properties. It's a synchronized global so a few hundred threads might pile up until the lock holder resumes. Odds are that those hundreds of threads won't finish before another one stops to time slice again. The performance hit has a ton of hysteresis so the server doesn't recover until it has a lower load than before the backlog started. The brute force fix is of course to eliminate calls to shared synchronized objects. All of the easy stuff has been done. Some operations aren't well suited to simple CAS. Bottlenecks that are part of well established Java APIs are time consuming to fix/avoid. Is there JVM or Linux tuning that will change the behavior of thread time slicing or preemption? I checked the JDK 6 options page but didn't find anything that appears to be applicable. -- I won't see Google Groups replies because I must filter them as spam
From: Lew on 1 Jun 2010 07:39 Kevin McMurtrie wrote: > I've been assisting in load testing some new high performance servers > running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is > suspending threads for time-slicing in very unfortunate locations. For > example, a thread might suspend in Hashtable.get(Object) after a call to > getProperty(String) on the system properties. It's a synchronized > global so a few hundred threads might pile up until the lock holder > resumes. Odds are that those hundreds of threads won't finish before > another one stops to time slice again. The performance hit has a ton of > hysteresis so the server doesn't recover until it has a lower load than > before the backlog started. > > The brute force fix is of course to eliminate calls to shared > synchronized objects. All of the easy stuff has been done. Some You call that "brute force" as if it weren't the actual, correct answer. > operations aren't well suited to simple CAS. Bottlenecks that are part > of well established Java APIs are time consuming to fix/avoid. But necessary. Having repeated calls on system properties that require synchronization is just plain stupid. System properties are the ones that don't change during a program run, so they should be (and should have been) written once into an immutable structure at class-load time and read thence thereafter. End of synchronization woes for that one. > Is there JVM or Linux tuning that will change the behavior of thread > time slicing or preemption? I checked the JDK 6 options page but didn't > find anything that appears to be applicable. The jmap/jhat dump utilities have some of that, IIRC. Otherwise you break into the add-on diagnostic tools. But really it sounds like your code needs refactoring in that it did not handle concurrency correctly from jump. -- Lew
From: Tom Anderson on 1 Jun 2010 08:08 On Mon, 31 May 2010, Kevin McMurtrie wrote: > I've been assisting in load testing some new high performance servers > running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is > suspending threads for time-slicing in very unfortunate locations. For > example, a thread might suspend in Hashtable.get(Object) after a call to > getProperty(String) on the system properties. It's a synchronized > global so a few hundred threads might pile up until the lock holder > resumes. Odds are that those hundreds of threads won't finish before > another one stops to time slice again. The performance hit has a ton of > hysteresis so the server doesn't recover until it has a lower load than > before the backlog started. > > The brute force fix is of course to eliminate calls to shared > synchronized objects. Well, yeah. > All of the easy stuff has been done. Some operations aren't well suited > to simple CAS. Bottlenecks that are part of well established Java APIs > are time consuming to fix/avoid. Sadly true. > Is there JVM or Linux tuning that will change the behavior of thread > time slicing or preemption? I checked the JDK 6 options page but didn't > find anything that appears to be applicable. I'm not aware of anything. Somewhere out there on the web is a complete list of the -XX options - see if you can find that and have a read through it. tom -- Yulava? Niob Yam!
From: Eric Sosman on 1 Jun 2010 08:33 On 6/1/2010 12:21 AM, Kevin McMurtrie wrote: > I've been assisting in load testing some new high performance servers > running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is > suspending threads for time-slicing in very unfortunate locations. For > example, a thread might suspend in Hashtable.get(Object) after a call to > getProperty(String) on the system properties. It's a synchronized > global so a few hundred threads might pile up until the lock holder > resumes. Odds are that those hundreds of threads won't finish before > another one stops to time slice again. The performance hit has a ton of > hysteresis so the server doesn't recover until it has a lower load than > before the backlog started. You've got "a few hundred threads" all calling System.getProperty over and over again so rapidly that they get in each other's way? Why? Seems a very odd situation, but ... Okay: As a first cut, write your own PropertySurrogate class. Have it slosh all the System properties into a HashMap instead of System's Hashtable, and change the System.getProperty calls to PropertySurrogate.getProperty instead. Since HashMap isn't synchronized, threads calling PropertySurrogate.getProperty won't block each other. If this over-and-over reading and re-reading of the System properties is because you suspect somebody's changing them, you could have one thread that just keeps iterating over the System properties, watching for changes. When it sees a change, it would update the PropertySurrogate HashMap accordingly. This veriant needs synchronization, but you could use a ReadWriteLock to allow multiple read-only threads to access the map simultaneously; they'd be locked out (briefly) only when the monitoring thread was in the middle of an update. Still, it seems to me that what you should be looking at is the reason for all these repetitive getProperty calls. It sounds like reading the clock every microsecond to see if it's 2014 yet. -- Eric Sosman esosman(a)ieee-dot-org.invalid
From: Patricia Shanahan on 1 Jun 2010 09:06
Kevin McMurtrie wrote: > I've been assisting in load testing some new high performance servers > running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is > suspending threads for time-slicing in very unfortunate locations. For > example, a thread might suspend in Hashtable.get(Object) after a call to > getProperty(String) on the system properties. It's a synchronized > global so a few hundred threads might pile up until the lock holder > resumes. Odds are that those hundreds of threads won't finish before > another one stops to time slice again. The performance hit has a ton of > hysteresis so the server doesn't recover until it has a lower load than > before the backlog started. > > The brute force fix is of course to eliminate calls to shared > synchronized objects. All of the easy stuff has been done. Some > operations aren't well suited to simple CAS. Bottlenecks that are part > of well established Java APIs are time consuming to fix/avoid. > > Is there JVM or Linux tuning that will change the behavior of thread > time slicing or preemption? I checked the JDK 6 options page but didn't > find anything that appears to be applicable. I can't help with the substantive question, but I can provide a couple of search terms that may be useful, "priority inversion" and "priority inheritance". Patricia |