Christopher Gruin, director of technology for the Democratic National Convention, sat down with eWEEK News Editor Scot Petersen at the DNCs Boston headquarters earlier this month to discuss preparations for next weeks convention.
What do you do when youre not setting up conventions for the DNC?
I spent 10 years in the Air Force. I was exposed to a lot of systems development and research development. After that I went to Washington and was an intern at the White House, and worked on their systems there. Then I got involved in the event world, applying information technology to political events and political operations. I did the Inauguration , then went to the Senate, did a project overseas.
The Inauguration was really the kickoff of [technology when] people expected a computer on every desk. We put an e-mail system up; it was the first event I did where we had an Internet connection. It was kind of a big deal, to get that up quickly. Back then, T-1 circuits took forever to get in. Broadband was not everywhere. Then in 2000 they asked me to come out for the Los Angeles convention. And that was a kind of culmination of large events—Id done other large political events.
The convention environment is a whole new world, with compressed development schedules, timelines, a lot of vendors. After that I went into the private sector in Los Angeles. I went to work for a company called Digital Planet, which was a streaming media broadcast studio, and they like many other companies back then started off great and I think in October 2000 they went under. A dot-com bust, went through a lot of money very quickly. Then I went to a production company in Hollywood, Imaginary Forces, as director of operations. … Then last summer I got a call from Rod OConnor, our CEO [of the DNCC] who asked if I wanted to do this from scratch this time.
Last time I came in more towards the end and [had to] move some mountains, but this time it was a really good opportunity to come in early. We were early enough this time that it made a big difference in what we were able to achieve.
So you had some experience in 2000…
In 2000 we actually ended up rolling everything back and starting from scratch with about six weeks to go, so it had its own challenges. I was involved but not on-site except towards the end.
What was your first order of business this time?
The approach was to get an office environment up. The office where we are in now, 53 State [St.], had no infrastructure, no cabling that we could use, no wiring, no connectivity, no infrastructure at all. So the first thing was to get computers on desks, get some connectivity. Then the next step of building the enterprise network, which was putting the [Microsoft] Exchange server in place, putting the server farm into place.
Thats the infrastructure that the convention itself is using?
Partially. Were getting the services up here, building a really robust network, so that they [staff] can do the heavy lifting and planning for the convention itself. We knew the staff would grow very quickly, so we had to build lots of scalability into the system. We had to line up our official providers. We had to line up our equipment, our hardware, software and then build it with a migration down to the Fleet Center in mind.
We started from the beginning to integrate everything here with the production as a whole, when in the past they were seen as two separate operations: the enterprise side and the production technology side. We tried to integrate those early, and we also made plans for some remote offices and made sure everyone had all the service they need on a daily basis available to them wherever they go.
The providers and the
Who are the primary vendors you are working with?
Microsoft is our official software provider, Cisco is our official internetworking provider, Hewlett-Packard is our official mobile solutions provider, but they also provide us everything from iPaqs, Tablets, printers, multifunctions, laptops, servers—kind of the whole shebang. IBM, while not an official provider, provided a $2 million grant to the Committee and the City of Boston. They provide us with the computers we use on a daily basis. Verizon is the telecommunications provider; Nextel is our cell phone provider.
Are these companies donating the equipment and software?
In legalese an official provider can donate goods and services in exchange for promotional considerations. Thats what we have to give, naming them official providers, allowing them to use that title.
Any open-source software?
No. Without hesitation we made a commitment to Microsoft and were going to use Microsoft. Were not going to use any open source. We had a lot of offers in various forms, firewalls, and IBM had some efforts in that area. But were pretty satisfied. We have good solutions from Microsoft for this event and what were trying to do here.
How are you connecting to the Internet?
Were using what is called a BLEC, which is a building local exchange carrier, Thats Cyprus Communications. Thats what we use to just get the office up and the connectivity in. That carried us through now. Now were switching over to a Verizon circuit, thats an OC-3 and a DS3 backup.
How much of a budget do you have to put it all together?
My budget is outside that circle. I have a budget for staff and a budget for utility hardware and software, but part of the trick is managing this through the official providers. And we took a different approach to that this time as well.
In the past we just looked for the hardware and software, and this time we made a concerted effort to work with their engineers and their support people, and what thats allowed us to do is to put a Microsoft person on-site. We have three Cisco engineers—actually five engineers—here today. So the Microsoft Exchange environment, server environment, was built by Microsoft. The IBM servers were put together by IBM. Cisco is programming all of their own stuff. They were doing the routing configuration and the switches, and the firewall, and thats really in their best interest, supporting their products.
So now theres a line of people coming to work here now because the word has gotten out through the various consulting and integration groups about what were doing. Theyre all really excited, which makes us feel really good here. They want to be here. Thats allowed my upper-level engineering to be done by the experts. So we have a midlevel staff that does our help desk support and in-house support, and we have a large group of volunteers, from tech academies to universities. Its a combination of engineers, staff and consultants from official providers.
How many are working on it all together?
About 50. Thats a combination of staff, permanent staff, volunteers and engineers from providers.
Give me some statistics. How many servers, miles of cable?
Were up to about 32 servers in various forms—a lot of thats redundancy and clustering. Miles of cable I dont know off the top of my head. Were going to deploy over 100 access points. They wont all be doing the same thing; well be managing them. Several hundred voice-over-IP phones, tying that into four PRIs [Primary Rate Interface], we can have land-line phone service through the voice over IP as well as phone-to-phone and system-system.
We have terabytes of storage, if we need it. We have lots of storage, both deployed and not deployed. We have a lot of storage, and were using it too. Were pushing out a lot of documents, supporting close to 800 e-mail accounts, lots of attachments. We have SharePoint installed, an extranet—were moving a lot of data around. Were creating video clips each night from the local news stories and send those around. So our video store is pretty large right now. Were doing a lot of encoding for use on the Web.
Who are the actual users going to be, the delegates?
We support three primary groups. Our staff and volunteers, which fluctuates between 800 and 1,000 people. Were going to deploy over 800 computers [to them]. We provide services to all the delegates, roughly 5,600 to 6,000 delegates.
The services come in the form of the voting system, bloggers, tablets, laptops, as well as to provide services off the floor, like e-mail. In addition were setting up a Polycom videoconferencing system. So we can do a one-to-many broadcast of a delegate breakfast in the morning. Were making sure they have access to the support they need at the hotels.
The media is the third client. The numbers are about 15,000 members, and that includes their support staff, producers, directors, roadies, as well as their on-air talent, reporters and such. We have two general press filing areas. So we provide Internet connectivity, phones, wireless; we dont provide equipment.
We also help the larger organization coordinate with Verizon and get all their drops installed and work on the wiring. [We have] two parallel networks installed: one network to support our network and our operations, and the second parallel network that Verizons doing to support the media, because the media has to contract for those services.
Theres no crossover between the networks?
No. They are two completely separate networks. We share closets, but theres no logical connection, no VLAN. They are two separate networks. We are providing a small amount of Wi-Fi connectivity for the photographers, so that they can have some mobility and file photos from the floor. And then as we ramp up and gauge our capacity well try to increase that and create some real targeted hot spots around the floor. Its not for the general press, not for anyone who walks through the door. Itll be a very targeted, very strategic area.
So they go to a certain spot.
Yes, they [the photographers] live at a certain spot near the stage, behind the media towers. Photographers know where theyre going to be and what well do is try to light those areas for them.
How do you keep the access restricted?
Theres a couple layers of what we can do. With the tri-band access points that were using from Cisco we can do a couple different things. We can do surveys with the antennas. Each band, [802.11] a, b and g can be VLANed, so if you hit the access point with a card, youll be sent to our authenticated PEAP [Protected Extensible Authentication Protocol] system through our firewall, through our ISA server and into our network.
Our staff is going to be using an a network. B and g users will be VLANed though a different access point out to the Internet. And then layered on top of that well use the standard MAC address and some other things we can do to keep people off. There will be access points in each of the voting stations. Each of the delegations has a podium. And were going to bury our access points in there. Well be able to turn them on and off as we need to through a centralized management system. We can also monitor for rogue activity.
So delegations can get wireless access when you give it to them?
If we are going to be giving it to the delegates. At this point its such a difficult radio environment. Theres wireless, TV cameras, wireless microphones. Were going to be surrounded by microwave trucks. Were two miles from Logan [airport] and a mile from a hospital that uses wireless from end to end. Its just going to be, to say the least, a very rich radio environment. And so we keep that in mind as we try to do this. The members of the media are aware of this as well. They know theres limitations. They have wired drops to file their stories. Its mission critical for them, so nobody is relying on wireless. For the delegates well roll that out if we can.
Everyone knows this is a very difficult environment. That being said, if we control the number of people that hit our access points, if they are very focused on what they are doing, were moderately confident we can achieve what we want to, which is provide Wi-Fi access to people who really need it to work in and outside of the convention. Then if we have capacity we can extend that to public spaces and for our staff.
So if you expand it to public spaces, would it be relatively open?
Its a credentialed event so we know everyone whos going through the door. We are in contact with everybody such as the delegations and what they need, what the medias requesting and what our staff needs. Its really providing a service thats requested of us. We knew from the beginning that we wouldnt be able to supply Wi-Fi to anyone who walks through the door. So its more of a corporate setting. Were providing it for people who need it, not just to have it out there to say were doing it.
Are there any technologies that are new or unique that havent been tried before in this kind of environment?
From what Ive been able to ascertain [yes], the Wi-Fi, the application of Wi-Fi in this environment, in a closed space, large number of media organizations, security concerns, also meeting the expectations. Everybodys so accustomed to Wi-Fi wherever they go. Weve been talking with some of the senior technical people at the different news organizations and everybody recognizes the challenge. So the strategy that weve taken to provide limited service in limited ways under very tight controls and circumstances is not just to say we did it but to provide a service where it needs to be. So thats one thing.
The VOIP is giving us a level of flexibility that we did not think wed have, saving us money, because we have to pay for all of our drops. So we can deploy voice over IP that helps us. It also adds some communications that we didnt have that we use. We can use instant messaging on the voice over IP. So those two applications, both of which were considered in 2000, both of which existed in 2000, but we just didnt think the technology was mature enough [then] in 2000, but we do at this point.
In 2000 we did a lot of technology just for technologys sake and this time were a little more focused. Were using technology as tools to support what were doing, support our communications, support the event.
What kind of security do you have in place?
Were working through three levels at the point. With Verizon, our connectivity, they have at their data center and at their POPs, layers of security, and were then running everything through our Cisco firewalls, then after that they hit an ISA Server 2004 box, which is Microsofts stateful firewall and proxy server. So between those three … all the companies [vendors] are invested in the event and the security of this event.
Nobody wants to be responsible for a security breach. Theyve assigned top people to this and were also working with law enforcement agencies to monitor what happens, what to look for, how to differentiate between a threat and a harassment, and we have meetings with them on a weekly basis.
What kind of security is in place for the staffers?
We use Active Directory authentication, we do the standard enterprise password changes and complex passwords. We work very closely with human resources, who actually build the accounts when someone comes in, does the account management, so they match up names. Theres no communication gap between when the person started or left and we didnt know about it. Weve given human resources the tools to manage the user accounts so we keep a very good eye on that.
We do audits all the time and theyre all layered behind our firewalls and our proxy servers. On a more practical level, we limit whos allowed to send e-mail, in and out, attachments—not just size limits, but attachments—without being too intrusive.
We dont want to kind of be big brother watching everything that people do. Everybodys really focused on what they are doing, and that really helps in monitoring that situation.
Did you borrow any best practices from other large events outside of the 2000 convention?
Yes, from some of the other events I worked in the past, especially with the radio issue. I worked at the 50th anniversary of NATO, which was at the time one of the largest gatherings of world leaders. It was in Washington, D.C., a three-day event, with really, really, really tight security. Although Wi-Fi wasnt deployed there I saw what worked and what didnt as far as radio frequencies, cell phones, two-way pagers, so we knew for this event that if we were going to try Wi-Fi and some of these other technologies, wed have to coordinate that with the security people, the FCC frequency people. So that was one thing, early coordination.
Also, speaking with some of the senior technology people with some of the news organizations, and early coordination with the city and early coordination with the venue. We worked very closely with the VP of technology and CTO at the Fleet Center about what their infrastructure is, can we use any of this, if we had to engineer any new pieces. It was done in concert with them, and I think thats really helped us a lot. So there were no surprises for either party, and we really moved forward in a partnership.
What is left to be done?
Well start laying the final extension of the network … theyre starting to put the raised floor down, were still building out our MDFs and IDFs, racking equipment, so well be connecting the dots over the next couple of days. The next big thing after that is the migration of a large number of employees and staff members from this office down to the Fleet Center and the back of house offices that were building there.
Were also bringing some remote offices up, our credentials office, our housing office, headquarters, and tying them all together though point-to-point circuits to the Fleet Center. Those are some of the things that need to be done, and then just the deploying of 500 computers to be used there, at the headquarters hotels, the train tracks. I mean, we had people in showers last time.
When is this going to be switched on?
Were actually on now. Weve connected our credentials office back to here, were connected down to the Fleet Center, with our point-to-point DS3 now. The OC-3 will be up, if not today then in the next day or two. So that will be switched on and well start working.
Were moving our first offices in there Saturday morning, so were up, for all intents and purposes, and again, from the beginning, weve known were going to have to migrate, weve known were going to have to tie it all together, so weve made it all as seamless as possible. Its never seamless, but weve tried.
What happens to all the equipment once the convention is over?
The IBM equipment is scheduled to be donated to the Boston Public School System. The Cisco equipment similarly weve donated to the Boston Public School System, and integrated with digital divide programs, technology programs, technology at home program, so were going to go to the city and school system.
The HP equipment will similarly go to the city. HP has three major consumers of their equipment in the city: the police department, the mayors office and the school district, and we work with all the local reps on these projects; the person who helps with all this is our point of contact with Hewlett-Packard. Everybodys kind of on the same page, and well be meeting with the schools to see if theres anything else that they need.
What weve ordered and what weve used for our event—all the equipment that Ive ordered Ive tried to keep in mind what the school district could use and what they need, if it could be repurposed. We always knew that it would remain here in the Boston community and it would be used by the schools.
What is the total value of all the equipment?
I think were right around $7 million. But we are actually in the process of our final accounting, because we also have to take into account the services and support that theyve … provided. The IBM equipment is a $2 million grant.
How much extra work did you have to do to the Fleet Center? How ready were they for this event?
There were actually two areas that we had to build: the video infrastructure and the network itself. They have a very good network that extends into their rooms, but what we did is we had to extend it into the floor, extend it to the stage that has 40 or 50 workstations.
We have areas around the perimeter, so the biggest challenge was extending and going beyond the normal infrastructure. But as far as what we had to lay in—very, very minimal. I think were laying some vertical risers because were laying some fiber. And then back in the locker rooms, in some of the back offices, were trying to take advantage of what they have.
Whos in the locker rooms?
One locker room is podium operations, I think the Bruins locker room, and Im not sure what the Celtics is being used for.
But all the space is being used?
All the space is being used, and were building new space were walling off. The press filing center, were building a wall. Whats called the boiler room, for the campaign, were building a wall—14-foot walls—and enclose them. Were turning a janitors closet into our help desk area. I wasnt joking, last time we had people in the shower, with banquet tables and ran wires into it. We really use every inch.
We built a very large corporation—$60 million dollar plus operation—in a matter of months, in an enclosed space. Thats our challenge, and here in technology we try and support that from an enterprise perspective.
Did the Kerry-Edwards campaign ask for anything in particular?
No, because the people that come in, a lot of them have done this before, and so they have a very clear idea. Ive worked with them before, so theres no surprises.
Much of what we do is not glamorous, its providing computers, connectivity, e-mail, support, printing support—thats always hard at events like this because people have to be able to sit down; it has to be obvious to them where their printers are, how they print to them, so its about making it simple, but on a larger scale.
What keeps you awake now and what will keep you awake during the convention?
Everything, everything. Just trying to keep our arms around everything, from printing, to making sure that services are in place for them when they hit the ground. Thats always a large challenge, and then to make sure on a high level everything stays up, that weve built enough redundancy into the system, make sure were monitoring the system, keeping our security in place, not taking any shortcuts. Its both on the desktop level and the system level, and the short answer is we dont get any sleep.
Has there been any discussion with the Republicans, any sharing of information about what theyre doing, their setup vs. you guys?
No, not from our point of view, and weve also made it very clear with our providers. Ciscos working the New York [Republican National] convention, Microsofts working the New York convention, IBM is providing a similar grant of $2 million, and both the providers and the DNCC made it very clear from the beginning that it should be separate business units, that they should only touch at a very high level, so therell be no copying of information.
Maybe as an after-action, Id love to see how they do things and what they did, but we take two very different approaches to how we do events, to how we do advance, to how we do gatherings, and I think that will be shown in the end. Im curious, but Im worried about us right now. The companies have been very good about maintaining the separation, which could be hard. All our partners have been really great. The only thing I do know is that a guy from Boston went down from Microsoft to work the Republican [convention], and most of our consultants are coming from New York, from Albany.