Information Technology Services activities, winter 2020

Information Technology Services’ (ITS) support during the protracted COVID-19 pandemic continued with staff constructively working on a variety of pandemic-related initiatives along with other numerous projects. COVID support continued with new initiatives requested to further automate and streamline processes at the campus COVID test site. Those requests, along with typical business requests and in concert with existing project requests ensured that ITS remained very busy.

In early November, ITS responded to a security incident that affected multiple IT services. The staff worked diligently to maintain all student systems with minimal disruption of service; however several business systems saw a disruption or degradation of service performance. Working with an outside specialist, ITS simultaneously endeavored to perform a forensic analysis and restoration of disrupted services. Additionally, ITS seized the opportunity to implement additional secure safeguards to mitigate the current situation and reduce the opportunity for future similar incidents.

Several IT changes to existing IT services or process modifications were to utilize Google Drive File System for data backup, install Carbon Black software on all devices, reset passwords for every account holder, implement multi-factor authentication on additional systems and access venues. A Frequently Asked Questions section on the top of the ITS homepage enabled students, faculty and staff to easily view relevant information such as how to download and install Google Authenticator. The page also included recent updates and was supported by regular Dateline and B-Line messaging.

Upgrades to technical systems and infrastructure

The Enterprise Systems and Applications (ESA) Team updated major systems such as Banner, Shibboleth, Infosilm and the Central Authentication Service (CAS). The team put into production a new “Genesee” server and a new Exagrid appliances, installed the Extron system (for media control in classrooms) and worked with IBM Spectrum and Microsoft engineering to implement a failover mode on the network attached storage (NAS) cluster, culminating four months of coordinating complicated processes. Binghamton University is now the first location to have a truly ‘productional’ failover mode for a NAS cluster. The ESA team assisted Terra Dotta and the abroad program with updating their SSL Certificate, decommissioned older SQL servers, provisioned winter 2021 courses into Blackboard, archived and deleted the 2019-2020 U Drives and created new views for Operational Data Store (ODS).

Security Infrastructure and Support (SIS)

Physical installations at the Decker building (48 Corliss Ave., Johnson City) included 61 cameras, 52 mass notification speakers, 34 door readers, two Morse Watchman Key Boxes, an intercom and one blue light; all were installed in advance of the building being occupied. The increased need to support exterior work necessitated that SIS procure a van with a lift bucket. The van is now in service and is used to facilitate exterior camera, speaker and Wi-Fi installations. With the number of devices SIS supports, the off-hour calls have increased and SIS hired another technician to help alleviate the strain on existing staff. In response to the recent security incident, SIS rebuilt 15 critical servers and all databases with no degradation of service or negative impact to the campus.

Network Administration and Operations

The Networking and Operations Team was busy this fall supporting final “pre-opening” network needs at 48 Corliss Ave./Decker building, network access for COVID testing at the Events Center and then in the Old Union Hall, additional installation/network support in the Engineering Building, and Science 2. Preplanning network needs and security (door access and cameras) for the Hinman Dining Hall was greatly expanded with the significant increase in requested proximity doors (an almost 400% increase from the original plans).

Several significant initiatives were implemented this fall; the Virtual Private Network (VPN) access process implemented two-factor authentication, BingView was moved behind the internal firewall as was faculty and staff VPN access, improved firewall rules blocked unauthorized source/destination IP addresses, customization of internal processes increased network and access efficiencies, and ISE device management installation was completed. The Network Team spent a major portion of its efforts in November and December supporting the Information Security, Enterprise Systems and Application, and the Technology Support Services teams as ITS worked through the recent security incident. The team assisted the Help Desk with VPN/password incidents and Carbon Black installations, and moved Wi-Fi authentication and VPN authentication into a new Domain. During this same timeframe, the team worked closely with Wm. H. Lane Construction as the Data Center remodel project in the Technology Hub commenced in earnest. The remodel necessitated significant restructuring and relocation of existing network equipment; all work is carefully done to avoid any network service interruptions to the University.

The Telecommunications Team updated and revised the online Campus Directory; rebuilt the Mitel phone system; completed all data, voice, panic button and fiber optic installations in the Decker building (48 Corliss Ave.); and reinstalled the Cellular DAS (Distributed Antena System) in the University Union basement once renovations were completed. The Telecommunications team worked with a multitude of contractors and utility companies in the last 12 months to secure permission and ‘rights’ to utilize their infrastructure to install a redundant, campus-owned fiber optic line from the Innovative Technologies Complex to the main campus along the Vestal Parkway. This is a critical component of ITS’s continued disaster plan, specifically to have redundant fiber between all campus facilities. New fiber lines were installed in the West Gym, Academic Building and the Nelson Rockefeller Center. These installations expand the redundant fiber connectivity of the campus; potentially minimizing a service interruption due to disruption of a fiber pathway.

Information Security

The Information Security Team continued communicating anti-phishing and “job scam” messages through social media, Dateline and B-Line, the ITS Phish Tank webpage and blog articles. The messaging and communication venues successfully reached our target audiences as anti-phishing scams reached an all-time low based on victim reports. The latter portion of the fall saw the team heavily involved in mitigating the recent security incident. One result of the incident was the campus was able to implement a full password reset of every active account; impacting over 25,000 users over multiple systems. The password reset necessitated all areas of ITS to closely coordinate and execute an extremely complicated process in an extremely short period of time.

The security team also researched and procured additional software (Spycloud) and support services (Beazley Breach Response Services) focused on recovering from data breach incidents. The team worked closely with the Enterprise Systems and Applications, Networking and Operations and Technology Support Services teams to remove malicious users from our computing systems. The installation of Endpoint, Detection, Response and Carbon Black facilitated mitigation and recovery efforts. The recent decision to create an IT Task Force aimed at setting overall campus security policy and processes enables ITS to proactively plan and implement future preventive data/access risk measures.

ITS support to the University

ITS staff completed 25 COVID-19 related projects and 55 unrelated projects in the past three months. The current fiscal year saw 137 projects closed or completed with ITS actively working on 73 projects. Twenty-four additional projects need additional resources (such as staff, equipment or vendor support) and there are 24 project requests on hold. Demand for ITS support continues with another 21 new project requests pending review and approval.

The Innovations and the Emerging Technology Studio (ETS) team developed and implemented an automated front-end (registration, check-in and parking pass validation) for the COVID surveillance testing site. This reduced the number of volunteer workers and more importantly, made the registration process virtually touch-free. All of this was instrumental in the continued success and smooth operation of the site as the site increased testing by two-fold. The ETS team also completed the Self-Guided Campus Tour, the Graduate Recommendation for Award Workflow and rewrote the Center for Civic Engagement Service Listing database.

The ITS Help Desk had an increased call volume related to issues with the security incident, password resets, two-factor authentication, connecting to the VPN, reporting of phishing scams, myCourses help, and assistance with University-owned computers and installation of Carbon Black. The Help Desk resolved more than half of the calls (58.9%) with the remainder of calls being deferred to second-level support. Of the 5,400 incidents via ServiceNow, 3,149 were assigned to the Help Desk with the remaining incidents being assigned to second-level support. Student printing for the fall semester was just over 297,945 B&W pages and 19,585 color pages printed; this was a major increase from the summer but significantly less than the fall of 2019, when students printed almost 3 million B&W pages and over 103,000 color pages.

The Technology Support team implemented new and innovative methods to keep all users of public computing resources safe. One such solution was a social distancing/virtually touchless printing process. Students are now able to literally print to a public printer without touching any public device except when picking up their output. The positive feedback from the students is gratifying as they see ITS continually looking at ways to keep everyone distanced and safe during the COVID-19 pandemic.

The Help Desk and Technology Support Services continued with their appointment-only strategy and offered virtual help through Zoom sessions and continued telephonic support, all of which minimized in-person interactions while still serving the campus and maintaining health and safety parameters. ITS supported and participated in the Middle States Accreditation for Binghamton University, developed a rapid re-deployment image for compromised University-owned computers, secured additional support for the Help Desk during periods of high user calls and staffed the Walk-In Carbon Black Clinic at the Events Center. These efforts were instrumental in resolving issues related to the security incident and certainly helped the campus stay on track with the technical demands for the fall semester.

ITS staff accolades

Kurt Knox received the “Harpur Hero Award,” a peer-to-peer Harpur Heroes recognition program for outstanding service to the University. President Harvey Stenger recently extended his appreciation to the following ITS staff: Denise Hawkins, Duane Theleman, Lauri Arnold, Ali Tanrvierdi, Jim Conroy, Kurt Knox, Debbie Cavallucci, Debbie Slattery and other members of the ITS staff members for volunteering and helping at the COVID test site.