Serious concurrency problems on fast systems
in [Java Programming]
|
Prev: DOWNLOAD FREE PALTALK LIVE VIDEO AND VOICE CHAT SOFTWARE
Next: Should -Xmx be a multiple of -Xms?
From: ClassCastException on 8 Jun 2010 02:28 On Mon, 07 Jun 2010 21:40:40 -0400, Lew wrote: > Aside from the fact that the JVM optimizes lock acquisition in the > uncontended case, once a thread blocks on a monitor, all the other > threads also trying to acquire that monitor also block. As soon as one > finally gobbles the lock, the rest mill about waiting their turn while > still more threads pile up on the monitor. > Sure, the critical section might only require a few hundred clock > cycles, > but the threads can wait seconds, even minutes, as they jostle about the > revolving door trying to enter. It reminds me of city gridlock. As rush hour commences, traffic begins to ramp up smoothly. It also keeps flowing smoothly right up to a certain point, and then whamo! It's like it hits a brick wall. Suddenly every brake light in the city comes on seemingly at the same time and huge traffic jams develop. The reason? Once the traffic gets tight enough, people slowing to turn, or merging in onto the highway, or whatever will trigger jams that just keep growing and growing. If the traffic is sparse enough one person slowing briefly doesn't slow anyone else down much of the time. If they do, it's often for a shorter time. If it gets dense enough, one person slowing down slows one or more others down for as long or longer. What eventually ends it is enough people getting where they're going and removing themselves from the roads, reducing the traffic density again. The jams dissolve then as the cars at the front speed up and then the ones behind them, and so on, and they all spread themselves out. Of course there are things that can make it worse. A good concurrency design with lock striping is like a good cloverleaf intersection; there's some merging but no stop signs or lights. A really awful concurrency design with a global lock that's highly contended is like having a large city's roads all radiate from a single hub, which has a stop sign. (Nah. I guess worse is with the hub, but without the stop sign. Screeeech! Crash! Crunch. Race condition.) If you can swing it, functional programming idioms with immutable data structures are like making your whole road network 3D, so traffic flows rarely intersect at all, only where truly necessary.
From: Martin Gregorie on 8 Jun 2010 07:30 On Mon, 07 Jun 2010 20:39:19 -0700, Kevin McMurtrie wrote: > The properties aren't immutable. > But its unlikely that you'll want to change them frequently, regardless of whether 'frequently' is defined in human or electronic terms. > The best feature of properties rather > than hard-coded values is being able to update them in an emergency > without server restarts. > So, why not design in the same sort of mechanism that is used by high performance servers such as bind or Samba? IOW, store properties in read- only objects and provide a method that reloads them on demand. It could be an explicit command that's issued when a change is made or simply a thread that scans for changes every minute or so and only makes changes when a modification is detected. -- martin@ | Martin Gregorie gregorie. | Essex, UK org |
From: Lew on 8 Jun 2010 07:46 Kevin McMurtrie wrote: >> The properties aren't immutable. Martin Gregorie wrote: > But its unlikely that you'll want to change them frequently, regardless > of whether 'frequently' is defined in human or electronic terms. > >> The best feature of properties rather >> than hard-coded values is being able to update them in an emergency >> without server restarts. >> > So, why not design in the same sort of mechanism that is used by high > performance servers such as bind or Samba? IOW, store properties in read- > only objects and provide a method that reloads them on demand. It could > be an explicit command that's issued when a change is made or simply a > thread that scans for changes every minute or so and only makes changes > when a modification is detected. Why are people so deucedly afraid of "new ImmutableThingie()"? Kevin, you asked for advice. You've gotten good advice - make shared objects immutable. Even non-shared objects should mostly be immutable, unless they really, really, really, really should be mutable by their inherent nature. Most of the time, as in your properties case, you can get by with creation of a new immutable object. In in-between scenarios, like if you want slowly-changing properties, you create one immutable shared object from a synchronized factory (that in this case copies System.properties) at the top of a long operation. System.properties is such the wrong place to put properties that change during the life of the program. As you point out it's not designed for good concurrency. Put those mutable properties somewhere else, like in a ConcurrentHashMap as I've already suggested. Sheesh. -- Lew
From: Robert Klemme on 8 Jun 2010 17:27 On 08.06.2010 05:39, Kevin McMurtrie wrote: > In article<874m08Fib7U1(a)mid.individual.net>, > Robert Klemme<shortcutter(a)googlemail.com> wrote: > >> On 07.06.2010 08:25, Kevin McMurtrie wrote: >>> In article<4c0c57a7$0$282$14726298(a)news.sunsite.dk>, >>> Arne Vajhøj<arne(a)vajhoej.dk> wrote: >>> >>>> On 02-06-2010 01:45, Kevin McMurtrie wrote: >>>>> In article<4c048acd$0$22090$742ec2ed(a)news.sonic.net>, >>>>> Kevin McMurtrie<mcmurtrie(a)pixelmemory.us> wrote: >>>>>> I've been assisting in load testing some new high performance servers >>>>>> running Tomcat 6 and Java 1.6.0_20. It appears that the JVM or Linux is >>>>>> suspending threads for time-slicing in very unfortunate locations. For >>>>>> example, a thread might suspend in Hashtable.get(Object) after a call to >>>>>> getProperty(String) on the system properties. It's a synchronized >>>>>> global so a few hundred threads might pile up until the lock holder >>>>>> resumes. Odds are that those hundreds of threads won't finish before >>>>>> another one stops to time slice again. The performance hit has a ton of >>>>>> hysteresis so the server doesn't recover until it has a lower load than >>>>>> before the backlog started. >>>>>> >>>>>> The brute force fix is of course to eliminate calls to shared >>>>>> synchronized objects. All of the easy stuff has been done. Some >>>>>> operations aren't well suited to simple CAS. Bottlenecks that are part >>>>>> of well established Java APIs are time consuming to fix/avoid. >>>>>> >>>>>> Is there JVM or Linux tuning that will change the behavior of thread >>>>>> time slicing or preemption? I checked the JDK 6 options page but didn't >>>>>> find anything that appears to be applicable. >>>>> >>>>> To clarify a bit, this isn't hammering a shared resource. I'm talking >>>>> about 100 to 800 synchronizations on a shared object per second for a >>>>> duration of 10 to 1000 nanoseconds. Yes, nanoseconds. That shouldn't >>>>> cause a complete collapse of concurrency. >>>> >>>> But either it does or your entire problem analysis is wrong. >>>> >>>>> My older 4 core Mac Xenon can have 64 threads call getProperty(String) >>>>> on a shared Property instance 2 million times each in only 21 real >>>>> seconds. That's one call every 164 ns. It's not as good as >>>>> ConcurrentHashMap (one per 0.30 ns) but it's no collapse. >>>> >>>> That is a call per clock cycle. >>> >>> HotSpot has some (benchmark-driven?) optimizations for this case. It's >>> hard to not hit them when using simple tests on String and >>> ConcurrentHashMap. >> >> What exactly do you mean by that? I can't seem to get rid of the >> impression that you are doing the second step (micro optimization with >> JVM internals in mind) before the first (proper design and implementation). >> >>>> ?!?! >>>> >>>>> Many of the basic Sun Java classes are synchronized. >>>> >>>> Practically only old ones that you should not be using anymore >>>> anyway. >>> >>> Properties is a biggie. A brute-force replacement of Properties caused >>> the system throughput to collapse to almost nothing in Spring's >>> ResourceBundleMessageSource. There's definitely a JVM/OS problem. The >>> next test is to disable hyperthreading. >> >> As someone else (Lew?) pointed out it's a bad idea to always go to >> System.properties. You should rather be evaluating them on startup and >> initialize some other data structure - if only to not always repeat >> checking of input values over and over again. > The properties aren't immutable. The best feature of properties rather > than hard-coded values is being able to update them in an emergency > without server restarts. Anyways, that was fixed by overriding every > method in Properties with a high-concurrency implementation. Too bad > Properties isn't an interface. Well, then use an immutable Hash map as Lew suggested and store it via AtomicReference. You get fast concurrent access and can update it at any time. > Fixing every single shared synchronized method in every 3rd party > library could take a very, very long time. I have no idea where you take that from. Nobody suggested fixing third party libraries - if anything the suggestion was to use them properly. Note that you can even use a Vector without too much performance degradation if it is accessed by a single thread only (of course one would use ArrayList then). It's not the libs - it's the way you use them. Cheers robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/
From: Mike Schilling on 8 Jun 2010 23:09
"Robert Klemme" <shortcutter(a)googlemail.com> wrote in message news:877quaFr6gU1(a)mid.individual.net... > On 08.06.2010 05:39, Kevin McMurtrie wrote: > >> Fixing every single shared synchronized method in every 3rd party >> library could take a very, very long time. > > I have no idea where you take that from. Nobody suggested fixing third > party libraries - if anything the suggestion was to use them properly. What if they use system properties promiscuously? Hypothetically: 1. My application receives XML messages. 2. I use a third-party library to deserialize the XML into Java objects. 3. The third-party library uses JAXP to find an XML parser. 4. JAXP always checks for a system property that points to the parser's class name. Even if the details are off (I don't know whether current versions of JAXP cache the class name), you get the idea. |