Larrabee delayed: anyone know what's happening?
|
Prev: PEEEEEEP
Next: Texture units as a general function
From: Kai Harrekilde-Petersen on 7 Jan 2010 13:00 Stephen Fuld <SFuld(a)alumni.cmu.edu.invalid> writes: > Del Cecchi` wrote: >> Anne & Lynn Wheeler wrote: >>> Stephen Fuld <SFuld(a)alumni.cmu.edu.invalid> writes: >>> >>>> Do you want to know the history of Infiniband or some details of what >>>> it was designed to do (and mostly does)? >>> >>> >>> minor reference to SCI (being implementable subset of FutureBus) >>> http://en.wikipedia.org/wiki/Scalable_Coherent_Interface >>> >>> eventually morphing into current InfiniBand >>> http://en.wikipedia.org/wiki/InfiniBand >>> >> >> I don't recall any morphing at all from SCI to IB. And I was >> involved in both. For openers SCI was source synchronous parallel >> and ib is byte serial. SCI is coherent, IB is not. > > I agree with Del here. I don't remember SCI as being a part of it, as > SCI was clearly intended as a coherent bus among multiple processors, > whereas IB was aimed at things like clusters, but as much as that, an > I/O interface to disks, etc. SCI was never targeted at that market. SCI started out as the grand be-all-end-all cache-coherent super-thing, but it could also do non-coherent transfers, and at Dolphin ICS, we sure had more success in attracting customers to use the noncoherent transfers for clustering (or as an IO extension bus) than doing the cache coherence stuff. On the cc-SCI side, we only had DG as a customer, with their Numaliine series. Kai -- Kai Harrekilde-Petersen <khp(at)harrekilde(dot)dk>
From: Stephen Fuld on 8 Jan 2010 02:14 Andy "Krazy" Glew wrote: > > Del Cecchi` wrote: > >> ... if you were astute enough to use InfiniBand interconnect. :-) > >> > >> you can lead a horse to water but you can't make him > >>give up ethernet. > > > Andy "Krazy" Glew wrote: > > > > What's the story on Infiniband? > > Do you want to know the history of Infiniband or some details of what it > was designed to do (and mostly does)? > >>>> Stephen Fuld <SFuld(a)alumni.cmu.edu.invalid> writes: >>>> >>>>> Do you want to know the history of Infiniband or some details of what >>>>> it was designed to do (and mostly does)? > ... >> IB was, as the Wikipedia article states, a "combination" of Intel's >> NGIO and IBM/Compaq's Future I/O. There is a fairly long story here, >> that I have related before, but let's just say that internal (within >> groups in Intel) politics and different visions of the requirements >> between the two groups led to the situation we have today - that is, a >> niche solution(fairly expensive), instead of a ubiquitous (and low >> cost)one. > > > This last is mainly what I was asking with "What's the story on > Infiniband?" > > First, my impression is that Infiniband is basically a niche solution. > Mainly used in HPC, expensive systems. Right? Yes. > And the trend is not good. It doesn't look like it will become > ubiquitous. Right? Right. > In fact, I guess I am wondering when IB will die out. Any guesses? I think it will stay around as a high speed cluster interconnect until something better comes around. I don't see anything one the horizon, but I am out of that world now. > I'm somewhat interested in why this happened. In the late 1990s, the PCI bus was running out of steam. It had quadrupled in throughput from its original implementation by doubling the bus width (to 64 bits), and doubling the clock rate (to 66 MHz). But particularly servers wanted more. They wanted to attach more disks and more network connections to their servers than the then current PCI bus would allow. No one wanted to go to a 128 bit bus, and the plans to double the clock speed to 132 MHz (called PCI-X) were having problems. PCI-X was so limiting that you could only have one card per bus. There was a kludge to allow two cards per bus, but that required lowering the clock rate to IIRC 100 MHz). The other solution was the extensive use of PCI/PCI bridges, etc. but that had its own set of problems. Since PCI was memory mapped, using bridges made some memory references go out over cables to external PCI cards to be satisfied. This slowed things down considerably. The Intel server group in Portland responded to this with a project code named IIRC Wekiva, later known as NGIO. It was a complete PCI replacement. The idea was an "I/O network" of high speed serial links and switches. It addressed the PCI shortcomings by several means. The hardware was to be in the chip set (then the North Bridge), to eliminate the memory latency problems. It was a "channel type" system. You set up a command packet in memory then loaded a register with its address, and the NGIO hardware took it from there. It was designed to reduce OS overhead by allowing queuing of requests by the hardware and the optional deferral of interrupts. It had the capabilities described in the Wikipedia article, RDMA, scatter/gather, multicast, etc. It combined the typical I/O capabilities, to storage, NICs, etc. and inter computer communication (i.e. cluster interconnect) in the same hardware. It was, let me call it, a medium sized system. NGIO supported a maximum of 64K nodes (16 bit address field). It had a fairly robust RAS capability, about what you might expect from a high end PC server implementation. It wasn't designed for extension to the whole enterprise within a single network. The idea was that, since it would be in the chip set and thus inexpensive, and it would replace the widely used PCI, it would become ubiquitous (i.e. on every desktop) and that a whole infrastructure would grow up around it. Intel set up a consortium that included Microsoft and others to develop and push the technology. Microsoft would insure the software drivers, etc. would be there, and presumably relatively standardized. There were various other members and interested parties who would provide NGIO network cards and disk adapters. An 8 port switch would fit on a single chip, so expansion would be inexpensive. At about the same time another group, led by IBM, Compaq and HP, was developing a competing technology, called Future I/O. It was similar to NGIO but had a much more expansive, enterprise type view of what was required. For example, Future I/O supported full IPV6 addressing. It had a much more robust RAS capability, It included support for optical, as well as electrical interconnect over much larger distances than NGIO supported. Furthermore, though the two were in some ways similar, they were fundamentally incompatible with each other. I wasn't involved in the development of Future I/O, so I know less about the sociology than I do about NGIO. It was clear to all involved that having two similar but incompatible systems was a really bad idea for the industry. So talks began between the leaders of the two groups to see if they could somehow merge their respective technologies. This led to a "freezing" of development. The talks took a couple of months to say that they would combine, but a lot longer to come up with a single spec with inputs from both groups. The "combined" result was Infiniband. However, in the meantime, there were problems. The combination of the delay and the fact that it was clear that Infiniband was much more complex than NGIO and was a vast overkill for desktops led Intel's desktop division to get off the bandwagon and go its own way. They addressed the PCI problems by developing "serial PCI", which we now know as PCI-E. This effectively killed the ubiquity of Infiniband. It would not be included in the CPU chip set and thus would be an add on card. This hurt its latency as well. And the lack of ubiquity led many in the industry to relegate IB development to at best a secondary product. The idea of using it to connect to storage and NICs fell victim to the new ubiquity of PCI-E, and so it was relegated only to its major use today, as a cluster interconnect. In a subsequent discussion I had with Intel's program manager for the project, he said he regretted that he hadn't done a "badge on the table" moment with Intel upper management to force the desktop division to implement as least a subset of Infiniband, but it isn't clear it would have helped. The die had been cast. > From where I sit, Infiniband looks a lot like IPI, the Intelligent > Peripheral Interface, which succumbed to SCSI. Some f the same > features, too. Sort of. IPI was really primarily a storage interface and didn't have the flexible switching, etc. of IB. > It's been a while since I looked at Infiniband, but... while it does > have some neat features - scatter/gather per Del, the following from > wikipedia > > * a direct memory access read from or, write to, a remote node (RDMA) > * a channel send or receive > * a transaction-based operation (that can be reversed) > * a multicast transmission. > * an atomic operation > > My complaint is also said by wikipedia: InfiniBand has no standard > programming interface." See above. With NGIO, there would have been at least a defacto standard. With all the different groups involved with IB, on so many different systems, it was a problem to agree on a single standard. > In particular, many of the above good idea features make more sense if > they were some reasonably low latency way of accessing them from user > space. Which there is... in some expensive controllers. IMHO they need > to be bound into the instruction set, if necessary by microcode > accessing said controllers. Along with implementations that can work on > mass market systems without Infiniband. Yup. And it could have happened, but I believe the time window for that is closed. I hope this helps. -- - Stephen Fuld (e-mail address disguised to prevent spam)
From: nmm1 on 8 Jan 2010 05:51 In article <hi6m1b$88u$1(a)news.eternal-september.org>, Stephen Fuld <SFuld(a)Alumni.cmu.edu.invalid> wrote: >Andy "Krazy" Glew wrote: > >> It's been a while since I looked at Infiniband, but... while it does >> have some neat features - scatter/gather per Del, the following from >> wikipedia >> >> * a direct memory access read from or, write to, a remote node (RDMA) >> * a channel send or receive >> * a transaction-based operation (that can be reversed) >> * a multicast transmission. >> * an atomic operation >> >> My complaint is also said by wikipedia: InfiniBand has no standard >> programming interface." > >See above. With NGIO, there would have been at least a defacto >standard. With all the different groups involved with IB, on so many >different systems, it was a problem to agree on a single standard. I haven't checked up on all of those, but I believe the majority are not generally available - e.g. they may not be in OpenIB. That issue affects programming language standards, too, because many of them are specifying features that can be implemented efficiently on clusters only if those features are available. And their proponents often claim that they are available, because InfiniBand supports them .... >> In particular, many of the above good idea features make more sense if >> they were some reasonably low latency way of accessing them from user >> space. Which there is... in some expensive controllers. IMHO they need >> to be bound into the instruction set, if necessary by microcode >> accessing said controllers. Along with implementations that can work on >> mass market systems without Infiniband. > >Yup. And it could have happened, but I believe the time window for that >is closed. Um, yes and no. Whether it could ever have happened is unclear, because many of those features need integrating into both the cache design and interconnect. Even with the specialist systems, there are a LOT of restrictions on how you may use those facilities. I don't think that it ever was feasible to deliver some of those in a usable fashion for the mass market, at least while that does not mean a single CPU design. [ To anyone who can't believe that: try writing a PRECISE specification of RDMA, covering race conditions, ECC recovery and interaction with the more arcane aspects of the CPU's memory subsystem. It's hairy, and almost impossible to do without also specifying how the CPU behaves. ] And, while I agree that the InfiniBand boat foundered because it was overloaded, that doesn't stop something else making the same trip. What it would start from is unclear, but guessing which of several possible technologies will actually take off is always uncertain. The inertia nowadays is huge, but that merely means that the pressure has to build up further before a change occurs. Regards, Nick Maclaren.
From: Del Cecchi on 8 Jan 2010 11:19 "Andy "Krazy" Glew" <ag-news(a)patten-glew.net> wrote in message news:4B454AA4.4020103(a)patten-glew.net... > > Del Cecchi` wrote: > >> ... if you were astute enough to use InfiniBand interconnect. > >> :-) > >> > >> you can lead a horse to water but you can't make him > >>give up ethernet. > > > Andy "Krazy" Glew wrote: > > > > What's the story on Infiniband? > > Do you want to know the history of Infiniband or some details of > what it was designed to do (and mostly does)? > >>>> Stephen Fuld <SFuld(a)alumni.cmu.edu.invalid> writes: >>>> >>>>> Do you want to know the history of Infiniband or some details of >>>>> what >>>>> it was designed to do (and mostly does)? > ... >> IB was, as the Wikipedia article states, a "combination" of Intel's >> NGIO and IBM/Compaq's Future I/O. There is a fairly long story >> here, that I have related before, but let's just say that internal >> (within groups in Intel) politics and different visions of the >> requirements between the two groups led to the situation we have >> today - that is, a niche solution(fairly expensive), instead of a >> ubiquitous (and low cost)one. > > > This last is mainly what I was asking with "What's the story on > Infiniband?" > > First, my impression is that Infiniband is basically a niche > solution. Mainly used in HPC, expensive systems. Right? > > And the trend is not good. It doesn't look like it will become > ubiquitous. Right? > > In fact, I guess I am wondering when IB will die out. Any guesses? > > I'm somewhat interested in why this happened. > > -- > > From where I sit, Infiniband looks a lot like IPI, the Intelligent > Peripheral Interface, which succumbed to SCSI. Some f the same > features, too. > > -- > > It's been a while since I looked at Infiniband, but... while it > does have some neat features - scatter/gather per Del, the > following from wikipedia > > * a direct memory access read from or, write to, a remote node > (RDMA) > * a channel send or receive > * a transaction-based operation (that can be reversed) > * a multicast transmission. > * an atomic operation > > My complaint is also said by wikipedia: InfiniBand has no standard > programming interface." > > In particular, many of the above good idea features make more sense > if they were some reasonably low latency way of accessing them from > user space. Which there is... in some expensive controllers. IMHO > they need to be bound into the instruction set, if necessary by > microcode accessing said controllers. Along with implementations > that can work on mass market systems without Infiniband. > > So long as these ideas remained (1) Infiniband only, (2) only > certain systems within Infiniband, (3) accessible only through OS > device drivers or (4) if usr accessible, decidely idiosyncratic, > often only working for pinned memory or processes - so long as these > problems persist, Infiniband was predestined for a niche in embedded > systems. Remember: I call HPC supercomputers embedded systems. In > fact, I consider anything that requires pinning memory to be > embedded. Pinning memory is the kiss of death for general purpose > I/O, not because it's a bad idea, but because it is unreasonable to > pin every process, and if you can't use it from every process, it > isn't ubiquitous. > > Hmm. "Pinning is the kiss of death". This has just been added to > my wiki page on Bad, Good, and Middling Ideas in Computer > Architecture. Although maybe I also need I page of mottoes. > > http://semipublic.comp-arch.net/wiki/index.php?title=Bad,_Good,_and_Middling_Ideas_in_Computer_Architecture > http://semipublic.comp-arch.net/wiki/index.php?title=Pinning_is_the_Kiss_of_Death > http://semipublic.comp-arch.net/wiki/index.php?title=Mottoes_and_Slogans_in_Computer_Architecture > > (Not much on my wiki yet, but I'm starting.) My perspective is that IB got caught in a war/battle between the server guys at intel and the desktop guys. Looks like desktop won since pci-express is basically the same physical layer as IB without all the good stuff that makes IB complicated. PCI express was, to my view, the desktop guys throwing the server guys under the bus, so to speak. As for sayings, you might like one I used periodically "(I don't care what the simulations say) You can't bullshit an electron. " I don't recall if I made that up or stole it from someone. Probably the latter. del
From: Del Cecchi on 8 Jan 2010 11:22
"Stephen Fuld" <SFuld(a)alumni.cmu.edu.invalid> wrote in message news:hi6m1b$88u$1(a)news.eternal-september.org... > Andy "Krazy" Glew wrote: >> > Del Cecchi` wrote: >> >> ... if you were astute enough to use InfiniBand interconnect. >> :-) >> >> >> >> you can lead a horse to water but you can't make him >> >>give up ethernet. >> >> > Andy "Krazy" Glew wrote: >> > >> > What's the story on Infiniband? >> >> Do you want to know the history of Infiniband or some details of >> what it was designed to do (and mostly does)? >> >>>>> Stephen Fuld <SFuld(a)alumni.cmu.edu.invalid> writes: >>>>> >>>>>> Do you want to know the history of Infiniband or some details >>>>>> of what >>>>>> it was designed to do (and mostly does)? >> ... >>> IB was, as the Wikipedia article states, a "combination" of >>> Intel's NGIO and IBM/Compaq's Future I/O. There is a fairly long >>> story here, that I have related before, but let's just say that >>> internal (within groups in Intel) politics and different visions >>> of the requirements between the two groups led to the situation we >>> have today - that is, a niche solution(fairly expensive), instead >>> of a ubiquitous (and low cost)one. >> >> >> This last is mainly what I was asking with "What's the story on >> Infiniband?" >> >> First, my impression is that Infiniband is basically a niche >> solution. Mainly used in HPC, expensive systems. Right? > > Yes. > >> And the trend is not good. It doesn't look like it will become >> ubiquitous. Right? > > Right. > >> In fact, I guess I am wondering when IB will die out. Any guesses? > > I think it will stay around as a high speed cluster interconnect > until something better comes around. I don't see anything one the > horizon, but I am out of that world now. > >> I'm somewhat interested in why this happened. > > In the late 1990s, the PCI bus was running out of steam. It had > quadrupled in throughput from its original implementation by > doubling the bus width (to 64 bits), and doubling the clock rate (to > 66 MHz). But particularly servers wanted more. They wanted to > attach more disks and more network connections to their servers than > the then current PCI bus would allow. No one wanted to go to a 128 > bit bus, and the plans to double the clock speed to 132 MHz (called > PCI-X) were having problems. PCI-X was so limiting that you could > only have one card per bus. There was a kludge to allow two cards > per bus, but that required lowering the clock rate to IIRC 100 MHz). > The other solution was the extensive use of PCI/PCI bridges, etc. > but that had its own set of problems. Since PCI was memory mapped, > using bridges made some memory references go out over cables to > external PCI cards to be satisfied. This slowed things down > considerably. > > The Intel server group in Portland responded to this with a project > code named IIRC Wekiva, later known as NGIO. It was a complete PCI > replacement. The idea was an "I/O network" of high speed serial > links and switches. It addressed the PCI shortcomings by several > means. The hardware was to be in the chip set (then the North > Bridge), to eliminate the memory latency problems. It was a > "channel type" system. You set up a command packet in memory then > loaded a register with its address, and the NGIO hardware took it > from there. It was designed to reduce OS overhead by allowing > queuing of requests by the hardware and the optional deferral of > interrupts. It had the capabilities described in the Wikipedia > article, RDMA, scatter/gather, multicast, etc. It combined the > typical I/O capabilities, to storage, NICs, etc. and inter computer > communication (i.e. cluster interconnect) in the same hardware. > > It was, let me call it, a medium sized system. NGIO supported a > maximum of 64K nodes (16 bit address field). It had a fairly robust > RAS capability, about what you might expect from a high end PC > server implementation. It wasn't designed for extension to the > whole enterprise within a single network. > > The idea was that, since it would be in the chip set and thus > inexpensive, and it would replace the widely used PCI, it would > become ubiquitous (i.e. on every desktop) and that a whole > infrastructure would grow up around it. Intel set up a consortium > that included Microsoft and others to develop and push the > technology. Microsoft would insure the software drivers, etc. would > be there, and presumably relatively standardized. There were > various other members and interested parties who would provide NGIO > network cards and disk adapters. An 8 port switch would fit on a > single chip, so expansion would be inexpensive. > > At about the same time another group, led by IBM, Compaq and HP, was > developing a competing technology, called Future I/O. It was > similar to NGIO but had a much more expansive, enterprise type view > of what was required. For example, Future I/O supported full IPV6 > addressing. It had a much more robust RAS capability, It included > support for optical, as well as electrical interconnect over much > larger distances than NGIO supported. Furthermore, though the two > were in some ways similar, they were fundamentally incompatible with > each other. I wasn't involved in the development of Future I/O, so > I know less about the sociology than I do about NGIO. > > It was clear to all involved that having two similar but > incompatible systems was a really bad idea for the industry. So > talks began between the leaders of the two groups to see if they > could somehow merge their respective technologies. This led to a > "freezing" of development. The talks took a couple of months to say > that they would combine, but a lot longer to come up with a single > spec with inputs from both groups. The "combined" result was > Infiniband. > > However, in the meantime, there were problems. The combination of > the delay and the fact that it was clear that Infiniband was much > more complex than NGIO and was a vast overkill for desktops led > Intel's desktop division to get off the bandwagon and go its own > way. They addressed the PCI problems by developing "serial PCI", > which we now know as PCI-E. This effectively killed the ubiquity of > Infiniband. It would not be included in the CPU chip set and thus > would be an add on card. This hurt its latency as well. And the > lack of ubiquity led many in the industry to relegate IB development > to at best a secondary product. The idea of using it to connect to > storage and NICs fell victim to the new ubiquity of PCI-E, and so it > was relegated only to its major use today, as a cluster > interconnect. > > In a subsequent discussion I had with Intel's program manager for > the project, he said he regretted that he hadn't done a "badge on > the table" moment with Intel upper management to force the desktop > division to implement as least a subset of Infiniband, but it isn't > clear it would have helped. The die had been cast. > >> From where I sit, Infiniband looks a lot like IPI, the Intelligent >> Peripheral Interface, which succumbed to SCSI. Some f the same >> features, too. > > Sort of. IPI was really primarily a storage interface and didn't > have the flexible switching, etc. of IB. > >> It's been a while since I looked at Infiniband, but... while it >> does have some neat features - scatter/gather per Del, the >> following from wikipedia >> >> * a direct memory access read from or, write to, a remote node >> (RDMA) >> * a channel send or receive >> * a transaction-based operation (that can be reversed) >> * a multicast transmission. >> * an atomic operation >> >> My complaint is also said by wikipedia: InfiniBand has no standard >> programming interface." > > See above. With NGIO, there would have been at least a defacto > standard. With all the different groups involved with IB, on so > many different systems, it was a problem to agree on a single > standard. > >> In particular, many of the above good idea features make more sense >> if they were some reasonably low latency way of accessing them from >> user space. Which there is... in some expensive controllers. IMHO >> they need to be bound into the instruction set, if necessary by >> microcode accessing said controllers. Along with implementations >> that can work on mass market systems without Infiniband. > > Yup. And it could have happened, but I believe the time window for > that is closed. > > I hope this helps. > > > -- > - Stephen Fuld > (e-mail address disguised to prevent spam) Interesting. One additional data item, as I understand it PCI-X was an IBM/Compaq/whoever that got foisted on Intel. So perhaps there was a little "in your face" thing there also. del |