Asheville beefs up cybersecurity plans

One late summer day in 2012, Jonathan Feldman, chief information officer for the city of Asheville, was helping his son move into his first college dorm when he got an unexpected call from then-City Manager Gary Jackson. Technological pandemonium was afoot in City Hall, and Jackson was not happy.

“It’s hilarious in retrospect,” Feldman recalls. He had contracted an outside vendor to conduct the city’s first cybersecurity drill, which involved sending fake emails to city employees to see if they would click unknown links and inadvertently download dangerous software. The vendor had chosen to copy a popular scam involving “cancellation warnings” from Blue Cross Blue Shield insurance — just as the city was switching to BCBS of North Carolina as its insurance provider.

“Not only did people get fooled at that time, but people got mad,” says Feldman. Nonetheless, he maintains that making the city uncomfortable was the right thing to do. Employees who first get tricked by a drill are less likely to trust real scammers, he says, and that skepticism is a critical piece of the cybersecurity puzzle.

“It doesn’t matter if you have all the guns and perimeter fences and security cameras around Fort Knox if someone goes, ‘Knock knock, I have free pizza for everyone!’” Feldman offers as an analogy. “If the guard says OK, then that’s all out the window.”

Asheville’s situation has drastically improved since 2012. The city’s IT Services department now runs regular cybersecurity drills and has developed an education program for employees, centered on the slogan “Sec_rity: It’s nothing without you.”

But as an increasing number of city services get tied into computer networks, Feldman believes it’s time to schedule another upgrade. The Asheville city budget for fiscal year 2018-19 includes money to hire two new staffers, a security coordinator and technical support technician who will “work to minimize threats to the city’s cybersecurity.”

No time like the present

Feldman laid out the case for adding new capacity in an IT Services business plan distributed to budget staff and the city manager’s office early this year. The department, he says, finds itself squeezed between increasing internal demand for digital work and “mushrooming” external security threats.

Within the city, IT Services is managing a wider array of operations and equipment than ever before. Feldman cites web-available parking deck data, security cameras and building access control as three newer areas of concern, but work in general has increased due to bond-funded capital projects and overall growth in city staff.

That growth caused the department’s monthly work order backlog to remain at or over 500 for all of 2017. When staff members are constantly stressed with more requests than they can handle, Feldman explains, they might put off work on longer-term security initiatives. “The worst thing that could happen is for us to be so task-saturated with immediate and urgent that we stop doing the advisable, proactive stuff,” he says.

And in the modern cybersecurity environment, Feldman continues, the city can’t afford to delay those efforts. “I think that we clearly have looked at places like Mecklenburg [County, which contains Charlotte]; we’ve looked at the Atlantas of the world. And you’d be foolish to say it can’t happen here,” he says, citing two recent high-profile cyberattacks on Southern governments.

Last December, hackers forced Mecklenburg officials to decline electronic tax payments, process jail admissions by hand and rely on paper records for development services after locking data on nearly a quarter of the county’s servers. The county refused to pay the requested $23,000 for the hackers to lift this ransomware attack and spent six weeks restoring the servers from backup copies. A similar March attack in Atlanta could cost the city up to $17 million in security services, software upgrades and new hardware.

Gap analysis

Asheville’s government finds itself in stiff competition with other organizations as it hires its two new employees. Brian Drawert, an assistant professor of computer science at UNC Asheville, says the city’s planned budget allocation of $131,609 would be on the low end — for a single position.

“The going rate is probably 20 percent higher than that for any single position in cybersecurity. They’re highly sought after,” Drawert says. Industry researcher Cybersecurity Ventures estimates a global shortage of 3.5 million cybersecurity professionals by 2021, with roughly 350,000 job openings currently unfilled in the U.S. alone.

Drawert says UNCA is doing its part to address this gap by beginning regular cybersecurity courses in the spring semester and considering a new cybersecurity concentration in the university’s computer sciences major. However, he notes that fresh graduates often lack the experience needed in real-life security environments.

“You can learn about all of the pieces you need, but without having had to secure systems, to have gone through being hacked and recovering from the hack, it’s hard to know what to do and how to handle that sort of emergency situation,” Drawert says. “[Asheville] probably could get younger college graduates, but that means they would be ill-prepared were something to actually happen.”

To this concern, Feldman counters that one of the new hires is a lower-level support technician, whose job will focus on freeing up capacity for senior staff to handle security issues. The new security coordinator, he says, will act more like a “project manager for a security program” than a more expensive chief information security officer.

“We can start with a coordination program and make sure everyone’s talking to one another. It doesn’t matter if the infrastructure people are secure if the application people aren’t secure,” Feldman explains. “I’m not looking to hire somebody that we can’t afford.”

The city has already hired a technical support technician at an annual salary of $37,614.98, but Feldman says his department is still hashing out the exact description for the security coordinator with the advice of an outside professional, a process expected to cost $8,800. While he wants the new coordinator “to have an educational or work experience background in security,” specific qualifications for the role have yet to be determined.

“We have been educated that when we insist on degrees versus experience, that has a disparate impact on potential applicants,” Feldman notes in reference to the city’s equity program. “So I’m trying to be careful about what I’m saying.”

Worst-case scenario

According to John Bumgarner, community outreach coordinator for the Bsides Asheville cybersecurity conference and chief technology officer for the U.S. Cyber Consequences Unit think tank, whomever Asheville picks will be thrust into a high-stakes situation.

“Any site that’s publicly available to the internet — some hacker, attacker or cybercriminal is scanning those websites or IP addresses every day, and they’re looking for potential vulnerabilities,” Bumgarner says. “Any city can be a target of ransomware or a cyberattack because they have to expose services out to the world.”

Those services include Asheville’s development and permitting portal, utilities payment system and open data websites. Any infrastructure with internet-connected equipment, such as traffic lights or the city’s water system, could also be at risk. Bumgarner points to recent cyberattacks on Ukraine’s electrical grid and water control technology as examples of how the city could become a target.

Leslie Carreiro, the city’s division manager for water production and water quality, says her department “takes security as seriously as we do our work in the treatment and delivery of water.” She notes that all city water treatment plants are staffed around the clock and that multiple layers of oversight are present for any anomalies in the system.

Based on recent history, however, Asheville is more likely to be hit with a ransomware attack like those on Mecklenburg County and Atlanta. Feldman says the city doesn’t have a policy for its response to such a hack, and he declined to give his own preference about paying or refusing a ransom demand.

“I think a policy is for something that happens a bunch: sick leave policy, vacation policy,” Feldman says. “If [a ransomware attack] happened, I assure you that we would have the attention of the policymakers and we would make a decision.”

Wait and see

The challenge of cybersecurity, Feldman admits, is ever-present. “This stuff is superhard because we have to be right all the time. The bad guys have to be right how many times?” he asks, holding up a single finger.

But Feldman notes that Asheville has support from other layers of government. The N.C. Department of Information Technology, he says, keeps an eye on the “dark web” — parts of the internet intentionally hidden from search engines — for stolen employee login credentials and other information. The FBI also conducts occasional audits of Asheville’s systems and offers help evaluating the results.

Working within the walls of City Hall, Feldman says the key to security isn’t the latest software package or restrictive policies toward new technology. Instead, it’s a social matter, an attitude of continuous improvement.

“The right approach to this stuff is a learning approach,” Feldman says. “If it works out, we do more of it; if it doesn’t work out, we do something different. Because the bad guys? That’s exactly what they’re doing.”