Frequently Asked Questions/Timeline
Early in November, computer servers at Binghamton University experienced malicious activity that rendered some of its servers inaccessible. The University’s Information Technology Services (ITS) staff immediately took the affected servers offline, took proactive measures and notified law enforcement.
Below you will find answers to many of the common questions surrounding this situation as well as a timeline of events and actions that has been developed from the the twice-daily reports from Associate Vice President and Chief Information Office Niyazi Bodur to President Harvey Stenger.
FAQs updated at 8 p.m. Tuesday, Dec. 22
How does this situation impact the campus systems?
Campus systems that relied on University servers were not accessible for a period of time, including H drives and U drives. The list of systems that remain unavailable can be found on the ITS system status page.
How do I know if my computer is affected?
If your computer was affected, a message appeared on your screen and you were unable to use it.
Can I turn my computer on?
Yes, you may turn your computer on, but you should not connect to the University’s network, whether on campus or at home, until you have installed Carbon Block or had ITS install it for you.
What should I do to avoid infecting my computer?
Leave your computer off whenever possible or if you do not need it to work. Ensure that you have installed the most recent operating system updates for your device and that you have updated antivirus software.
The University has installed a security tool called Carbon Black on University-owned computers that will help protect the computers. ITS went building by building on campus installing Carbon Black on computers in offices. Do not connect office computers to the campus network until you or ITS have installed Carbon Black and installed updates.
If your computer is at home, you will need to install Carbon Black before connecting to the campus network once VPN access is restored.
Installers for Carbon Black are available online at Carbon Black Installers. (Note: You must be logged in with your Binghamton Google Account to access these installers.) If you need assistance installing Carbon Black, contact the ITS Help Desk for assistance at firstname.lastname@example.org.
I need Carbon Black installed to protect my computer. Can I install it myself?
(Updated Nov. 12, 2020)
Carbon Black is an IT security tool that protects the computer it is installed on while providing insight into the campus network to reduce the likelihood of future attacks. The tool protects the computer in the background without any user interaction.
Individuals can install Carbon Black rather than waiting for ITS to reach out. Installers for Carbon Black are available online at Carbon Black Installers. (Note: You must be logged in with your Binghamton Google Account to access these installers.) If you need assistance installing Carbon Black, contact the ITS Help Desk for assistance at email@example.com.
Will I see a confirmation that I have correctly installed Carbon Block?
(New Nov. 10, 2020)
Once you have completed the Carbon Black setup and click finish, you have successfully installed it. It is a silent completion and the software runs completely in the background on your device. You will not see an icon or any changes in your settings, but you are now able to connect to the University's network.
Where should I back up a copy of my files?
(New Nov. 14, 2020)
If you are in need of a place to store files, use your Binghamton University Google Drive.
A better option is to have your Google Drive files available as a drive on your computer by downloading and installing Drive File Stream.
To download Drive File Stream:
On your computer, open:
- DriveFileStream.exe on Windows
- DriveFileStream.dmg on Mac
Follow the on-screen instructions and when prompted to login, use your Binghamton University email and password.
Can I access the campus network from a remote location through the SSL VPN (Pulse
(New Nov. 12, 2020)
Access has been restored to the SSL VPN, but now requires two-factor authentication. To set up your computer to use the SSL VPN, follow the directions found online.
Find complete information on the Pulse Secure VPN online.
What systems should I avoid?
(Updated Nov. 13, 2020)
Individuals should not log into any service on the BGM Domain until Carbon Black has been installed on their computer. Once they have installed Carbon Black, they can log onto the BGM Domain, though some systems may remain unavailable. Systems that are hosted by vendors outside of the BGM Domain, including BMail, Zoom, MyCourses, MyBinghamton, Banner, Panopto, Qualtrics, Starfish and OmniUpdate remain available,
How do I access BingView?
(New Nov. 14, 2020)
To access BingView, students and users off-campus are required to connect to the Pulse Secure VPN before connecting to BingView.
Below are instructions for connecting:
- Step 1. Setup Two Factor Authentication for VPN (if not already done)
- Step 2. Install Pulse Secure
- Step 3. Connect to BingView
Contact the Help Desk at firstname.lastname@example.org with questions.
Can I safely use the CAS to sign on?
Yes. It is safe to sign on through the Central Authentication Service (CAS) using your PODS credentials.
Is it safe to use a personal laptop or USB to teach in a classroom?
(Updated Dec. 22, 2020)
Faculty should not use their University-owned computer on the campus network until after they have installed Carbon Black and any critical operating system and antivirus updates. It is safe to use a personal laptop that is current with operating system and antivirus updates. Some classrooms have computers installed and those are also safe.
Can I safely use Eduroam to connect to Wi-Fi?
Yes. Both Wi-Fi through Eduroam and wired ethernet are fine for personal devices.
University-owned computers should have the Carbon Black client installed before connecting to Eduroam or the wired ethernet.
How can I access library services?
(New Nov. 17, 2020)
All Libraries online services are now available, including Interlibrary loan, course reserves and off-campus access to articles, e-books and other e-resources via the Libraries website.
Is my mobile device (iPhone, iPad, Android, etc.) safe on Eduroam?
At this point there is no indication that mobile device platforms (iOS, Android) were targeted. It is important to remember to regularly install security updates on mobile devices.
Is any of my personal information compromised?
At this time, we do not have any reason to believe that personal information has been compromised.
When systems are restored, will all of my files be intact?
At this time, ITS has found backups intact and we do not have any reason to believe that files will not be restored.
How long will it be before the campus systems are back to normal?
It is not yet known when all systems will return to normal. ITS continues to work to safeguard the systems so they will be operational and the campus will be updated as systems are brought back online.
Are system sensors working?
Sensors on special equipment such as ultra-low freezers, oxygen sensors, etc. and Sodexo equipment are functioning, but they are not reporting automatically through our building management system. If you have such equipment, we recommend you check to see if it is functioning.
The campus fire alarm system is functioning.
Is building/card access affected?
Building/card access is generally functioning as it should.
Timeline (updated Jan. 2, 2021)
Saturday, Nov. 7, 2020
ITS became aware of suspicious activity at approximately 4:30 a.m. and immediately shut down servers, checked backup systems and notified law enforcement. Additionally, ITS notified New York state as per the New York State Information Security Policy, as well as SUNY according to the SUNY Cyber-Incident Reporting Procedure and also activated Breach Response Services under our Beazley Cyber Insurance Policy. Campus leadership was notified and a B-Alert text was sent to the entire campus at 9:57 a.m. to notify everyone that many systems and files were not available. The incident effectively affected all Windows servers and a number of Windows desktops. A sample of affected services were StarRes, Starfish, SAS, Hyperion, Infosilm, Medicat, Listserv, and Maximo. Because Linux and UNIX systems were not affected, Banner and any outside-hosted system was working fine, including Blackboard, Slate, BMail, Zoom, Panopto, etc.
An update was sent to the entire campus via B-ALERT text and email at 5:29 p.m. to explain more about the extent of the situation, requesting that staff work remotely if they were able, but to not use the VPN or access a campus domain, and that staff working on campus not turn on their campus-based network-connected computers. It was noted that students’ computers were not likely affected because they do not use the campus domain. Questions were referred to the Help Desk.
Assessment began, including gathering information through the Help Desk of desktops and laptops affected, and by the end of the day, the University had contracted with industry leader Kroll to contain and begin a forensics investigation into the malicious activity.
Sunday, Nov. 8, 2020
A check of auxiliary services confirmed that dining services, the bookstore and Visions Federal Credit Union were not affected and were fully operational. In addition, the campus surveillance testing center would continue to operate.
The installation of Carbon Black Endpoint Detection and Response software began. Carbon Black will allow Kroll and ITS to identify bad actors and eject them from the network. Carbon Black will allow Kroll and ITS to identify and mitigate attacks as we move forward.
Monday, Nov. 9, 2020
ITS reported that the following are safe and fully operational:
- The campus network is safe and fully operational.
- Phones are operational on all campuses, with the exception of soft phones used to remotely attach to campus numbers.
- The Help Desk call center, though not all other call centers are, and those that are not are being worked on.
In addition, on a preliminary basis, PODS/classroom computers seem to be clean, safe and operational, though more analysis is needed
Students, faculty and staff do not yet have access to their shared files, but G Drive is fully accessible.
Our consultant preliminarily identified the malware, which is an important step in developing remediation strategies. Meanwhile, ITS continued working on re-building several servers, some of them critical as they are life safety related, and some to improve communications:
- Surveillance cameras
- Card access, which is working in a local mode
At 4:54 p.m. a Dateline/B-line Addition utilizing the B-Engaged platform because listservs were not yet operational. The message explained that server outages continued and ITS was working to resolve them. Users were referred to the ITS system status page at https://itstatus.binghamton.edu:8443/ to monitor affected systems. The message noted that the investigation was in the preliminary stages of analysis and mitigation, and if any individuals’ data was affected, the University would provide updates to those individuals.
Those with affected computers were asked to contact the Help Desk at email@example.com.
Tuesday, Nov. 10, 2020
At 5:09 p.m. a Dateline/B-Line Addition was sent to the entire campus, again using the B-Engaged platform, noting that the University continued to make progress to repair system outages affecting its business information systems, and was bringing systems back online as quickly as possible. Safety and academic functions were the top priority, including the ability for students and faculty to access all of their files to enable them to complete assignments and finish the semester.
Faculty, staff and students were encouraged to visit the Frequently Asked Questions page that had been posted that day to find answers to the most commonly asked questions.
Carbon Black, a tool that helps safeguard devices and also assist with the forensics investigation was obtained and faculty and staff were encouraged to install it on their computers with the assistance of ITS or on their own through instructions in the FAQ.
Once Carbon Black was installed on a computer, the user was able to log onto the University’s network and operate as normal, though might not have access to all systems.
Access was restored to the following systems:
- Password server
Wednesday, Nov. 11, 2020
(began utilizing the regular Dateline and B-Line listservs)
Carbon Black installations continued, with the client installed on 223 servers and 2,355 PCs to date.
The SSL VPN was upgraded, hardened, and two-factor authentication will now be required. Instructions and a FAQ page will be disseminated tomorrow morning.
The focus was on Admissions, Student Accounts, Student Records, the Foundation, DDEI, Athletics, Research, Legal, Business Office, Student Affairs, Operations, your office, the Provost’s office, the Health Sciences Campus, Watson, CCPA and the Libraries. A number of servers were brought online, including:
- Lenel servers that University Police monitor for card access
- Genetec servers that University Police monitor for surveillance cameras
- Evisions and Transcript services
The consultant continued to make significant progress with its forensic analysis.
ITS continued to work on a process for rebuilding desktop systems.
Listservs were restored. At 4:55 p.m. a Dateline Addition was sent requesting all faculty and staff to install Carbon Black on their University-owned computers. With assistance and coordination by the University’s Incident Management Team (IMT) technicians were made available at a socially distanced location at the Events Center from Nov. 12 through Nov. 17, to install Carbon Black for individuals if they chose. Once Carbon Black was installed, users could work on campus as they normally would without concern, though some systems were not yet available.
Once the majority of users installed Carbon Black as part of the University’s mitigation efforts, a remediation phase began and ITS began collecting information about the affected computers so they could be evaluated. This process was prioritized with assistance from division and department heads.
Access was restored to the following systems:
- Kronos: The Physical Facilities time-tracking/payroll system.
- Mitel Phone System: Phones had been working and now all ECC/ACD Call Center access and soft phones are enabled. Telecom staff is working with individual Call Center users to add them to the appropriate queues.
- Library databases
The Frequently Asked Questions page continued to be updated as information changed.
Thursday, Nov. 12, 2020
Installations of Carbon Black continued and we will work with Kroll to increase the number of licenses available to users. We have already exceeded 3,000 installations on devices.
At 4:46 p.m. a Dateline Addition was sent reminding users to install or have Carbon Black installed for them. The SSL VPN (Pulse Secure) became accessible again, but with two-factor authentication necessary to access it. Directions to set up computers with the new authentication process were made available at https://www.binghamton.edu/its/about/organization/operations-and-infrastructure/networking/2fa-pulse-secure-ssl-vpn.html.
To ease the burden on the Help Desk, those with infected University-owned computers that required remediation were asked to send a message to firstname.lastname@example.org and include the following information:
- Is the computer infected (Yes, No, Unknown)?
- Computer operating system (PC or Mac)
- Location of the computer (Building and room #)
- Typical days/times you can be available with your computer
The information was compiled with the help of the University’s IMT and sent to the Help Desk to assist in prioritizing the rebuilding process of infected computers
The focus was on infrastructure, but access was also restored to the following systems:
- The SSL VPN with two-factor authentication.
- The TSM operations management center, which is internal to ITS, but is needed to be able to start the larger restores.
- Infosilm Scheduling: This is for B There scheduling.
- Interlibrary Loan service
The Beechwood print server that serves print jobs for department shared printers
The applications pool of BingView, which now requires logging into the VPN for access from the Internet.
The H and S drives will take some more time in part because there are more than 30,000,000 files on the storage structure. The Network Attached Storage (NAS) infrastructure that houses the storage was brought online and it will take several days to restore the files.
Friday, Nov. 13, 2020
File restore continued with hopes that it would be completed by early next week.
A total of 48 people brought their computers to the Events Center over the past two days for assistance. In addition, 36 individuals with a total of 43 computers are on the list the IMT is consolidating for ITS to help us when we are able to rebuild desktop systems.
ITS continued to actively work with our consultants in identifying the forensics, which still requires active due diligence.
At 5 p.m. a Dateline Addition sent explaining that ITS was working to restore access to the H, U and S drives, but it wasn’t known when restorations would be complete and the campus would be notified when it would be.
Access was restored to the following key systems:
- The Marathon server is back up, which provides licensure for virtual desktops and should allow the virtual desktop infrastructure environment to come back online.
- All ECC/ACD phone queues were enabled and all agents were back in.
- Deerpark is back online and Timetabler and Exam scheduling are up and running.
ITS continues to work on a process for rebuilding desktop systems that will be efficient. Anyone with an infected computer was reminded to send their information to email@example.com and ITS will notify these individuals when they are able to assist with their computers.
Saturday, Nov. 14, 2020
One rather small drive in the infrastructure cluster finished restoring and ITS started restoring a larger one, which is still continuing.
As of today, Carbon Black installations are going well, with the client installed on 223 servers and 2,355 PCs.
We have not yet made it widely available, but a Dashboard was built at https://my.binghamton.edu/app/campus_life/sys_recovery_status. As the numbers kept steady after the second week of December, on Dec. 23, the site was taken down and kept only for archival purposes.
Today, we focused on Admissions, Student Accounts, Student Records, the Foundation, DDEI, Athletics, Research, Legal, Business Office, Student Affairs, Operations, your office, the Provost’s office, the Health Sciences Campus, Watson, CCPA and the Libraries.
Sunday, Nov. 15, 2020
ITS made its first pass on the recovery of the following shared drives:
- U drive - select courses that use local network storage for student course materials was restored. It is a relatively small number because the majority of courses use Google Drive for student course materials.
- H drive - personal/individual network shares
These drives were not available to their owners yet. Two things must happen before access to the owners can be opened:
- We need to quickly verify the contents are complete
- We need to identify a strategy/mechanism where the files in these drives are protected from unauthorized access. We are in the process of developing some options.
The following shared drive is in the process of recovery:
- S drive - departmental network shares, where different users from the department share files
Once we complete these recoveries, we will start recovering departmental small systems that were affected.
ITS continued to actively work with our consultants in identifying the forensics and removal of the suspect materials. This will be an on-going activity for the near future.
Monday, Nov. 16, 2020
The server for Library course reserves was restored. The Library technical staff began the process of confirming that all the relevant documents were available.
ITS was finalizing the announcement to campus regarding password changes and the availability of personal and departmental file shares. The password change for the BMail, PODS domain, and BU domain is automated and synchronized and pretty straightforward. Plans for how to accomplish this for the BGM domain users, which is a legacy domain that is not straightforward were being finalized.
Tuesday, Nov. 17, 2020
ITS focused on the restoration of shared files, the S drive. The general, Administration, Watson, and Library restorations are completed.
Regarding the password reset, the BGM domain password reset will be included on the web-based password page with approximately 80 users who will need to be handled on a case-by-case basis for the password change by the Help Desk. An announcement has been prepared for tomorrow’s Dateline.
Since Sunday, ITS has focused on re-building the MS-SQL database infrastructure, which is a clustered and high-availability infrastructure that houses a number of departmental database systems.
Access to the following services was restored:
- Sodexo servers and services
- All transcripts
Wednesday, Nov. 18, 2020
The re-building of the MS-SQL database infrastructure, which is a clustered and high-availability infrastructure that houses a number of departmental database systems was completed and ITS readied for the password change tomorrow.
Thursday, Nov. 19, 2020
The password reset did not go smoothly in the morning, causing significant disruption to University business. The plan had been tested on 20,000 users in a test environment prior to its implementation, which did not translate well to the production environment. ITS Systems staff was able to rectify the situation by mid-day and users were able to start resetting their passwords. By 4:15 p.m., 13,000 users had changed their passwords. By late afternoon, the demand/load had diminished such that there were no delays in the user experience. The decision to set the password change was on the strong recommendation of our consultant, which had advocated for the change to happen earlier in the week. The date was set for today to avoid any potential disruption to the University’s Middle States reaccreditation visit.
Accomplishments for the day include:
- Students can now print in PODS
- The Reports server is online
- U drives for students are accessible
- Binghamton Foundation servers and systems are up and running
Friday, Nov. 20, 2020
Most of the H and S drives are now available in read-only mode.
Saturday, Nov. 21, 2020
The campus was notified of the availability of the H ad S drives.
There is progress in xTender service, which is a document repository for a number of departments, including Admissions. The server is restored, the service is up and ITS is checking the integrity of the documents and finalizing access.
Sunday, Nov. 22, 2020
ITS continued recovery and restoration.
Monday, Nov. 23, 2020
ITS continued recovery and restoration.
Tuesday, Nov. 24, 2020
Today is the first regular weekday after the availability of the H and S drives was announced to campus. ITS needed to verify restored files and access for a few departments and individuals, and was successful in assisting in all cases, though it took most of the day for the Systems Team to accomplish this.
- ITS continued file restoration with the remaining departments.
- The Help Desk made further progress clearing the backlog of emails.
- The Project Management Office (PMO) and Systems Team started to work with the specific departments and owners to develop a priority list for restoration of departmental systems.
Wednesday, Nov. 25, 2020
Restoration of xTender, the University’s document management system, was completed with minimal limitations. It is used by a number of departments and is critical for Admissions and Legal. The full status is:
- Users can now log into Xtender as normal.
- Users with permission to create/edit documents previously should still be able to do so.
- The last documents loaded were on or around Nov. 6.
- Automatic loading of Admissions documents is not currently running and will not begin again until after Thanksgiving, pending testing and verification of the load process.
- Reports Manager document processing and loading has not been confirmed or tested as of noon today.
Some faculty are having difficulty accessing their shared files. There are two possible reasons for that:
- Hard-coded links or mappings to their files will need to be changed. This is being handled by Help Desk, Desktop Support and local IT support if a department has such support.
- In rare cases, the first pass does not restore all of the files. In such a case, ITS recommends calling the Help Desk, such that we have a central point for all case reports. In response, ITS will restore that group of files again and has found that the second restore solves the issue.
Thursday, Nov. 26, 2020
ITS distributed a message to the deans that they could re-distribute to their college/school lists regarding any faculty having difficulty accessing their shared drives and missing files.
The Systems Team re-restored H drive files for about 45 faculty and staff. Three faculty confirmed that the re-restore provided them with their missing files. We did not yet hear from others.
- The Pharmacy and the Pharmaceutical Sciences departmental shares were confirmed as fully restored.
- A first pass of recovery of shared files for Physical Facilities was completed.
- The Systems Team will continue with the re-restore of the H drive files throughout the next four days.
- The interlibrary loan system is back online.
Friday, Nov. 27, 2020
The Systems Team continued with the re-restore of the H drive files.
Saturday, Nov. 28, 2020
ITS completed restoration of the Synapsis system for International Student and Scholar Services (ISSS). ISSS will verify and validate on Monday.
Departmental shared files for the University Center for Training and Development and Communications and Marketing are being restored.
The Systems Team continued with the re-restore of the H drive files.
Sunday, Nov. 29, 2020
The Systems Team continued with the re-restore of the H drive files.
Monday, Nov. 30, 2020
ITS had been concerned about potential malicious activity over the holiday period but none was found.
- Since Nov. 18, 28,744 users changed their passwords.
- The Systems Team continued with the re-restore of the H drive and S drive files.
- The Systems Team made the final preparations to share the S drive files with the appropriate users.
Tuesday, Dec. 1, 2020
The Systems Team continued with the re-restore of the H drive and S drive files, which will continue throughout the week at the very least.
With the password reset effectively completed, ITS slowly started adding full read/write capability for select users.
In addition to the general re-restore, the Systems Team re-restored H drive files for 50+ users who were missing files after the first restore, who reported getting their files after the re-restore.
The Systems Team started making departmental shares (S drive files) available for the appropriate users. In addition to the several back-office departments, Physical Facilities S drive has become available.
The Reports server was back online last week with data files as of Nov. 6. As of today, new reports and data from different systems will be added to the reports server and will include all new reports and data starting Nov. 7.
The Systems Team began working on re-starting routine backups.
Wednesday, Dec. 2, 2020
The Systems Team continued with the re-restore of the H drive and S drive files.
The SAS client depot, which is used to distribute various SAS software, has been rebuilt and is operational.
ITS continued to develop additional plans for further hardening and segmenting access to the SSL VPN to present to SOG for review and possible approval. One of the additional security hardening options is to use Endpoint Detection and Response (EDR) similar to the Carbon Black agent and deploy Managed Detection and Response (MDR) similar to the Kroll Responder service on a permanent basis. ITS will continue to research this to present to the IT Task Force.
Thursday, Dec. 3, 2020
The Systems Team is continuing with the re-restore of the H drive and S drive files, which will continue throughout this week at the very least.
The StarRez system, which is the housing system for Res Life, is restored and is online.
The Internal Res Life database server, which contains more than a dozen databases/systems for Student Affairs and Res Life, is restored and is online.
Friday, Dec. 4, 2020
The Systems Team completed the re-restore of the H drives and continued with the re-restore of the S drives, which is coming to completion. Almost every department has department shares, and some have several. Almost all S drives are available to departments and ITS is doing the re-restore to recover the skipped files from the first restore. For some departments the second restore is finished and they have access to all their shared files. The remaining departments have access to their files with some exceptions. Physical Facilities has access to its energy management files (Z drive).
Saturday, Dec. 5, 2020
The Systems Team is continuing with the re-restore of the S drives.
Sunday, Dec. 6, 2020
No specific tasks completed.
Monday, Dec. 7, 2020
The Systems Team completed restoring the S drives. Going forward, ITS will re-restore only if any user has a particular need.
Several additional databases/systems' file restores have been completed. They still need to be re-configured with the new security and connection parameters so are not yet available.
There are about 333 departmental shared drives.
Carbon Black statistics:
- Carbon Black is installed on 483 servers, including Windows and Unix servers
- At its peak,124 Windows servers were infected
- As of the end of last week, that number was down to 72
- Carbon Black is installed on 4,649 devices
- At its peak, 334 devices were infected
- As of the end of last week, that number was down to 112
- Rebuilt devices 162
- Unknown status 60
Tuesday, Dec. 8, 2020
The Information Technology Task Force, co-chaired by Vice President for Research Bahgat Sammakia and Vice President for Operations JoAnn Navarro, convened to receive its charge. The broad-based group will guide the University as it develops a proactive and adaptive security system that will minimize security exposures and provide a secure backup system to protect critical information moving forward. It will hold its first full meeting Monday, Dec. 14.
The KMS licensing server for Microsoft Windows, Office 2016 and Office 2019 was restored and in operation.
The Embark system for the Art Museum was restored and in operation.
Wednesday, Dec. 9, 2020
A Dateline message informed the campus that if any users need to renew or refresh their Microsoft product licenses, the option is available to them and they were directed to where/how to renew.
Thursday, Dec. 10, 2020
The Cumulus system, which is the photo repository for Communications and Marketing, was restored.
Friday, Dec. 11, 2020
Real time Banner ILP, which provides a real-time data interchange between Banner and Blackboard, was online.
The ID Card system went online, including all the photos prior to May 2020. The photos after May 2020 may need to be taken again.
The Lutron system, used by Physical Facilities and others to control high-end lighting in specific venues, was restored.
Saturday, Dec. 12, 2020
ITS continued with background preparation for the remaining systems.
The Systems Team, which has been working 12+ hours a day for 36 straight days, was given a day off tomorrow.
Monday, Dec. 14, 2020
The Information Technology Task Force held its first full meeting and heard a summary of the malware event and where the campus stands today. The group discussed the role played by consulting firm Kroll, how Carbon Black has been installed and will continue to aid the campus and also discussed bringing in an independent consultant to work directly with the task force.
Tuesday, Dec. 15, 2020
The Systems Team continued building new servers to replace the domain controllers.
The Whatsup Gold monitoring system that is used to monitor availability of servers and services was restored and is up and running.
Wednesday, Dec. 16, 2020
The Systems Team continued building new servers to replace the domain controllers. As part of that work some legacy configurations, which were acceptable a decade ago but undesirable by today's state-of-the-art standards, were changed.
The Domain Name Service (DNS), which was bound to the old domain controllers, was moved to its dedicated servers. As a result, a majority of the servers required new SSL certificates, which were generated and installed.
Thursday, Dec. 17, 2020
Due to the unexpected, excessive snowfall, most ITS staff were not able to make it to campus so were unable to complete a restoration they were planning to wrap up. However, they started working on another one, which they were able to work on remotely. It’s expected that most of the remaining non-ITS systems restorations will be completed by the end of the month.
Additionally, the IT Task Force met and heard from Lodestone Security, a firm recommended by our cyber insurance firm. Lodestone will work directly with the task force to help establish, manage and monitor a cybersecurity risk management program tailored to University needs. Action items were also discussed.
Friday, Dec. 18, 2020
- The server and the database installation for the Campus Mail system were prepared. Pitney Bowes will install the new version of their software Monday, Dec. 21.
- ITS made progress on recovering photos in the ID Card system pictures that were taken after May 2020. Most of the first-year class pictures (3,895) have been recovered.
- The parking office uses two systems and preparations for the servers were completed.
- ITS completed building the new PODS domain controllers and promoted them. They are in production.
- Files were restored for a few departments and end users.
Monday, Dec. 21, 2020
Pitney Bowes started installing the Campus Mail software, which will take several days.
The website for older/archived bulletins is online.
Tuesday, Dec. 22, 2020
The IT Task Force met and heard a presentation about the University at Buffalo that the University can learn from, and discussed a number of issues, including recommendations for moving forward with additional security measures.
The Lock Shop key control system is back online and functioning.
Wednesday, Dec. 23, 2020
ITS reached out to the system owners of internal, customer-facing systems and obtained their consent as to the priority and timeline for restoration as noted:
- Campus Mail - in progress, expected completion end of December
- Parking Systems - expected completion late January (It was rather cumbersome and long when originally set it up.)
- International Tax Treaty System - expected completion end of December
- Physical Facilities Licensing Server - expected completion mid-January
- Fine Arts Collection System - expected completion early-February. Fine Arts uses this only for archival purposes, not for daily production.
For ITS internal systems, all production systems are up and running, though there are some redundant and test systems left to restore.
Thursday, Dec. 24, 2020
The Mass Spectrometer for the School of Pharmacy and Pharmaceutical Sciences is controlled by a server that was compromised during the continuing cyber attack. Kroll was able to stop the malware execution and isolate the machine, but was not able to remediate. The server software was vulnerable and out of date, but today, ITS completed rebuilding it from bare metal and turned over to SOPPS to finalize the control software for the mass spectrometer.
A week earlier than the original schedule, the Campus Mail system has been fully rebuilt and upgraded to the most recent version of the Pitney Bowes software. The system is up and running and the only task left is to confirm proper data exchange with all the other systems it communicates with.
Wednesday, Dec. 30, 2020
The Campus Mail system has been interfaced with Banner and Res Life system, completing all the necessary interfaces.
The Systems Team is continuing to automate the backup jobs. The H and S drives are still being backed up manually, but backup for all other systems and data has been automated.
Thursday, Dec. 31, 2020
The Campus mail system (Pitney Bowes Send Suite) is fully online. The six workstations have been updated and the new version installed. The seven scanning units have new firmware installed and the software and database have been updated to the latest version while moving to server 2019 and SQL server 2019. Their batch email scripts have been updated to reflect the new database structure.
Monday, Jan. 4, 2021
The WhatsUP services are now all restored and the database and server backups are back online
Tuesday, Jan. 5, 2021
All of the older Domain controllers are now offline and the new domain controllers will be scanned to verify they are secure.
Wednesday, Jan. 6, 2021
The maximo update server that runs the IBM software update facility is back online.
Tuesday, Jan. 12, 2021
The StarRez test system is online and running. This will allow testing of the billing process for residence hall living in the spring semester to proceed. In addition, the Fine Arts visual resource collection is online, moved to a commercial system and the file repository has been shared with appropriate staff who maintain it.
This leaves only two systems left facing University departments: the parking system and international tax treaty system. Both are being worked on and are expected to come online in the near future.
A service (GRADSVDB) that had been used by the Graduate School admissions team is no longer needed. The server will be taken offline permanently and be recycled.
Wednesday, Jan. 13, 2021
The International Tax Treaty software system, used for employees who are not U.S. citizens, is online for Human Resources use.
Thursday, Jan. 14, 2021
The Parking Services HUB system that controls the paid parking lot and parking ramp gates along with the parking payment systems is now back online. The gates will be lowered to normal status in these locations starting Friday, Jan. 15.
Outside of Medicat, which is in the process of being migrated to a hosted solution, this service marks the last of the campus-facing production systems that needed to be restored. There is still a long way to go with back-end interfaces, test systems, backups, permissions, new security requirements and other cleanup, but for the most part, the front-facing campus services are now fully functional.