For those of us who like to scramble over complex systems, spending hours or days getting hardware and software to work together, working at places like NASA or CERN seems like a dream job. Imagine having the opportunity to turn a wrench on the Space Shuttle or the Large Hadron Collider (LHC) – not only do you get to spend some quality time with some of the most advanced machines ever produced, you can be safe in the knowledge that your work will advance humanity’s scientific understanding of the universe around us.
Or at least, that’s what we assume it must feel like outsiders. But what about someone who has actually lived it? What does an actual employee, someone who has had to wake up in the middle of the night because some obscure system has broken and stopped a machine that cost taxpayers $4.75 billion to build, think about working at the European Organization for Nuclear Research?
Luckily for us, Daniel Valuch stopped by Hack Chat this week to answer that very question. Sharing everything from the grueling nine-hour entrance interview to the first days of the LHC, where errors occurred every few minutes, he did not gloss over the situation. That said, after spending two hours sharing his fascinating stories with members of the community, the answer seems pretty clear – Daniel loves his job as much as we thought he would.
Early in the chat, he says that joining the LHC team while it was still being built reminded him of movies and documentaries he had seen about NASA’s Apollo Moon program. There was a palpable sense that what they were working on was of immense scientific and historical importance, and he couldn’t help but be in awe of the brilliant scientists and engineers he rubbed elbows with. He may have been the new guy in the lab, but thanks to his RF experience, Daniel soon found himself working with high-speed ADCs, DACs and FPGAs responsible for controlling the beam of accelerated particles.
To the home hacker, it may not be immediately obvious how RF skills apply to particle accelerators. But of course this is RF on a scale few of us can imagine. As Daniel explains, accelerating and stabilizing the beam requires several megawatt-class amplifiers, and a good understanding of RF theory is a must. In his own words, “there is no physics without RF and also no RF without physics.”
Even with all the equipment and know-how in the world, working with a cutting-edge particle accelerator is proving to be exceptionally challenging. When dealing with events that happen at the speed of light, nothing is really simple. Even the fastest electronic systems can’t react fast enough to control the beam in real time, so in a circular accelerator like the LHC you have to predict what adjustments you need to make for the next pass. Still, particles only take a few microseconds to complete the 27-kilometer (16.7-mile) round through the gas ring before they’re back in front of you—so there’s no time to waste.
Of course, as interesting as these technical details are, the topic of conversation for this Hack Chat was specifically “Life at CERN”, so of course people wanted to know what an average day for Daniel looks like. Working in what is called an equipment group, Daniel and his team are responsible for developing and building hardware for the accelerator. But since it runs 24/7, there are only occasional windows to actually get in and work on their new equipment. Consequently, an incredible amount of testing and troubleshooting is done before it’s actually time to install the hardware. When every hour of downtime costs a quarter of a million dollars, due diligence is the name of the game.
When the accelerator is running, and it is not possible to work directly with the systems, the teams have time to pursue new designs and experiments. But should something go wrong with the gas pedal, it will immediately get top priority. Although there is an emergency response team that can be on site within 45 minutes, they are only able to deal with a small subset of potential problems. For everything else, system-specific experts must intervene. As such, Daniel says you must be ready to answer a call from the “Big Lady” at any time, day or night.
Solving problems on a machine with hundreds of thousands of systems requires not only an encyclopedic knowledge of how it all fits together, but also the ability to quickly pick up on the tiniest clues. He describes it as “very high caliber hacking” – an apt term if ever we heard one.
But he also notes, almost wistfully, that the LHC has performed with remarkable stability recently. When first brought online in 2008, he says the mean time between failures (MTBF) of the LHC was just five minutes. Now he goes months without hearing the siren call of the Big Lady, which gives his team more opportunities to work on smaller projects and machines. It may not be as exciting, but it is certainly productive.
We would like to thank Daniel Valuch for providing such an incredible insight into what it is like to work at CERN, and specifically on a machine as incredible as the Large Hadron Collider. We’re glad to hear that working in such an environment is as rewarding as we hoped, and wouldn’t be surprised if readers saw the sciences in a different light after reading his first-hand accounts. As always, the full transcript of this Hack Chat is available on Hackaday.io, which we’d highly recommend checking out if you’re even remotely interested in high-energy physics or the nuts and bolts of the LHC.
Hack Chat is a weekly chat session hosted by leading experts from all corners of the hardware hacking universe. It’s a great way for hackers to connect in a fun and casual way, but if you can’t make it live, these overview posts as well as the transcripts posted to Hackaday.io make sure you don’t miss a thing.
Featured image: CERN, CC BY 4.0, via Wikimedia Commons
